@@ -174,10 +174,10 @@ Build status
- Travis
- Run tests on Linux and check code-style
- 
+ Github Actions
+ Run tests on Linux and Mac, plus check code-style
+ 
diff --git a/docs/src/apiref.rst b/docs/src/apiref.rst
index eef4684ac8..583e4528a9 100644
--- a/docs/src/apiref.rst
+++ b/docs/src/apiref.rst
@@ -20,6 +20,7 @@ Modules:
corpora/lowcorpus
corpora/malletcorpus
corpora/mmcorpus
+ corpora/opinosiscorpus
corpora/sharded_corpus
corpora/svmlightcorpus
corpora/textcorpus
@@ -27,6 +28,7 @@ Modules:
corpora/wikicorpus
models/ldamodel
models/ldamulticore
+ models/ensemblelda
models/nmf
models/lsimodel
models/ldaseqmodel
diff --git a/docs/src/auto_examples/core/images/sphx_glr_run_corpora_and_vector_spaces_001.png b/docs/src/auto_examples/core/images/sphx_glr_run_corpora_and_vector_spaces_001.png
index 807a84c4de..5c86a24471 100644
Binary files a/docs/src/auto_examples/core/images/sphx_glr_run_corpora_and_vector_spaces_001.png and b/docs/src/auto_examples/core/images/sphx_glr_run_corpora_and_vector_spaces_001.png differ
diff --git a/docs/src/auto_examples/core/images/thumb/sphx_glr_run_corpora_and_vector_spaces_thumb.png b/docs/src/auto_examples/core/images/thumb/sphx_glr_run_corpora_and_vector_spaces_thumb.png
index 5a8e564326..bab9aec4a4 100644
Binary files a/docs/src/auto_examples/core/images/thumb/sphx_glr_run_corpora_and_vector_spaces_thumb.png and b/docs/src/auto_examples/core/images/thumb/sphx_glr_run_corpora_and_vector_spaces_thumb.png differ
diff --git a/docs/src/auto_examples/core/run_corpora_and_vector_spaces.ipynb b/docs/src/auto_examples/core/run_corpora_and_vector_spaces.ipynb
index 40a3324206..875db7b507 100644
--- a/docs/src/auto_examples/core/run_corpora_and_vector_spaces.ipynb
+++ b/docs/src/auto_examples/core/run_corpora_and_vector_spaces.ipynb
@@ -15,7 +15,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "\nCorpora and Vector Spaces\n=========================\n\nDemonstrates transforming text into a vector space representation.\n\nAlso introduces corpus streaming and persistence to disk in various formats.\n\n"
+ "\n# Corpora and Vector Spaces\n\nDemonstrates transforming text into a vector space representation.\n\nAlso introduces corpus streaming and persistence to disk in various formats.\n"
]
},
{
@@ -33,7 +33,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "First, let\u2019s create a small corpus of nine short documents [1]_:\n\n\nFrom Strings to Vectors\n------------------------\n\nThis time, let's start from documents represented as strings:\n\n\n"
+ "First, let\u2019s create a small corpus of nine short documents [1]_:\n\n\n## From Strings to Vectors\n\nThis time, let's start from documents represented as strings:\n\n\n"
]
},
{
@@ -141,7 +141,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "By now it should be clear that the vector feature with ``id=10`` stands for the question \"How many\ntimes does the word `graph` appear in the document?\" and that the answer is \"zero\" for\nthe first six documents and \"one\" for the remaining three.\n\n\nCorpus Streaming -- One Document at a Time\n-------------------------------------------\n\nNote that `corpus` above resides fully in memory, as a plain Python list.\nIn this simple example, it doesn't matter much, but just to make things clear,\nlet's assume there are millions of documents in the corpus. Storing all of them in RAM won't do.\nInstead, let's assume the documents are stored in a file on disk, one document per line. Gensim\nonly requires that a corpus must be able to return one document vector at a time:\n\n\n"
+ "By now it should be clear that the vector feature with ``id=10`` stands for the question \"How many\ntimes does the word `graph` appear in the document?\" and that the answer is \"zero\" for\nthe first six documents and \"one\" for the remaining three.\n\n\n## Corpus Streaming -- One Document at a Time\n\nNote that `corpus` above resides fully in memory, as a plain Python list.\nIn this simple example, it doesn't matter much, but just to make things clear,\nlet's assume there are millions of documents in the corpus. Storing all of them in RAM won't do.\nInstead, let's assume the documents are stored in a file on disk, one document per line. Gensim\nonly requires that a corpus must be able to return one document vector at a time:\n\n\n"
]
},
{
@@ -152,7 +152,7 @@
},
"outputs": [],
"source": [
- "from smart_open import open # for transparently opening remote files\n\n\nclass MyCorpus:\n def __iter__(self):\n for line in open('https://radimrehurek.com/gensim/mycorpus.txt'):\n # assume there's one document per line, tokens separated by whitespace\n yield dictionary.doc2bow(line.lower().split())"
+ "from smart_open import open # for transparently opening remote files\n\n\nclass MyCorpus:\n def __iter__(self):\n for line in open('https://radimrehurek.com/mycorpus.txt'):\n # assume there's one document per line, tokens separated by whitespace\n yield dictionary.doc2bow(line.lower().split())"
]
},
{
@@ -177,7 +177,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Download the sample `mycorpus.txt file here <./mycorpus.txt>`_. The assumption that\neach document occupies one line in a single file is not important; you can mold\nthe `__iter__` function to fit your input format, whatever it is.\nWalking directories, parsing XML, accessing the network...\nJust parse your input to retrieve a clean list of tokens in each document,\nthen convert the tokens via a dictionary to their ids and yield the resulting sparse vector inside `__iter__`.\n\n"
+ "Download the sample `mycorpus.txt file here `_. The assumption that\neach document occupies one line in a single file is not important; you can mold\nthe `__iter__` function to fit your input format, whatever it is.\nWalking directories, parsing XML, accessing the network...\nJust parse your input to retrieve a clean list of tokens in each document,\nthen convert the tokens via a dictionary to their ids and yield the resulting sparse vector inside `__iter__`.\n\n"
]
},
{
@@ -224,14 +224,14 @@
},
"outputs": [],
"source": [
- "# collect statistics about all tokens\ndictionary = corpora.Dictionary(line.lower().split() for line in open('https://radimrehurek.com/gensim/mycorpus.txt'))\n# remove stop words and words that appear only once\nstop_ids = [\n dictionary.token2id[stopword]\n for stopword in stoplist\n if stopword in dictionary.token2id\n]\nonce_ids = [tokenid for tokenid, docfreq in dictionary.dfs.items() if docfreq == 1]\ndictionary.filter_tokens(stop_ids + once_ids) # remove stop words and words that appear only once\ndictionary.compactify() # remove gaps in id sequence after words that were removed\nprint(dictionary)"
+ "# collect statistics about all tokens\ndictionary = corpora.Dictionary(line.lower().split() for line in open('https://radimrehurek.com/mycorpus.txt'))\n# remove stop words and words that appear only once\nstop_ids = [\n dictionary.token2id[stopword]\n for stopword in stoplist\n if stopword in dictionary.token2id\n]\nonce_ids = [tokenid for tokenid, docfreq in dictionary.dfs.items() if docfreq == 1]\ndictionary.filter_tokens(stop_ids + once_ids) # remove stop words and words that appear only once\ndictionary.compactify() # remove gaps in id sequence after words that were removed\nprint(dictionary)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "And that is all there is to it! At least as far as bag-of-words representation is concerned.\nOf course, what we do with such a corpus is another question; it is not at all clear\nhow counting the frequency of distinct words could be useful. As it turns out, it isn't, and\nwe will need to apply a transformation on this simple representation first, before\nwe can use it to compute any meaningful document vs. document similarities.\nTransformations are covered in the next tutorial\n(`sphx_glr_auto_examples_core_run_topics_and_transformations.py`),\nbut before that, let's briefly turn our attention to *corpus persistency*.\n\n\nCorpus Formats\n---------------\n\nThere exist several file formats for serializing a Vector Space corpus (~sequence of vectors) to disk.\n`Gensim` implements them via the *streaming corpus interface* mentioned earlier:\ndocuments are read from (resp. stored to) disk in a lazy fashion, one document at\na time, without the whole corpus being read into main memory at once.\n\nOne of the more notable file formats is the `Market Matrix format `_.\nTo save a corpus in the Matrix Market format:\n\ncreate a toy corpus of 2 documents, as a plain Python list\n\n"
+ "And that is all there is to it! At least as far as bag-of-words representation is concerned.\nOf course, what we do with such a corpus is another question; it is not at all clear\nhow counting the frequency of distinct words could be useful. As it turns out, it isn't, and\nwe will need to apply a transformation on this simple representation first, before\nwe can use it to compute any meaningful document vs. document similarities.\nTransformations are covered in the next tutorial\n(`sphx_glr_auto_examples_core_run_topics_and_transformations.py`),\nbut before that, let's briefly turn our attention to *corpus persistency*.\n\n\n## Corpus Formats\n\nThere exist several file formats for serializing a Vector Space corpus (~sequence of vectors) to disk.\n`Gensim` implements them via the *streaming corpus interface* mentioned earlier:\ndocuments are read from (resp. stored to) disk in a lazy fashion, one document at\na time, without the whole corpus being read into main memory at once.\n\nOne of the more notable file formats is the `Market Matrix format `_.\nTo save a corpus in the Matrix Market format:\n\ncreate a toy corpus of 2 documents, as a plain Python list\n\n"
]
},
{
@@ -357,7 +357,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "In this way, `gensim` can also be used as a memory-efficient **I/O format conversion tool**:\njust load a document stream using one format and immediately save it in another format.\nAdding new formats is dead easy, check out the `code for the SVMlight corpus\n`_ for an example.\n\nCompatibility with NumPy and SciPy\n----------------------------------\n\nGensim also contains `efficient utility functions `_\nto help converting from/to numpy matrices\n\n"
+ "In this way, `gensim` can also be used as a memory-efficient **I/O format conversion tool**:\njust load a document stream using one format and immediately save it in another format.\nAdding new formats is dead easy, check out the `code for the SVMlight corpus\n`_ for an example.\n\n## Compatibility with NumPy and SciPy\n\nGensim also contains `efficient utility functions `_\nto help converting from/to numpy matrices\n\n"
]
},
{
@@ -393,7 +393,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "What Next\n---------\n\nRead about `sphx_glr_auto_examples_core_run_topics_and_transformations.py`.\n\nReferences\n----------\n\nFor a complete reference (Want to prune the dictionary to a smaller size?\nOptimize converting between corpora and NumPy/SciPy arrays?), see the `apiref`.\n\n.. [1] This is the same corpus as used in\n `Deerwester et al. (1990): Indexing by Latent Semantic Analysis `_, Table 2.\n\n"
+ "## What Next\n\nRead about `sphx_glr_auto_examples_core_run_topics_and_transformations.py`.\n\n## References\n\nFor a complete reference (Want to prune the dictionary to a smaller size?\nOptimize converting between corpora and NumPy/SciPy arrays?), see the `apiref`.\n\n.. [1] This is the same corpus as used in\n `Deerwester et al. (1990): Indexing by Latent Semantic Analysis `_, Table 2.\n\n"
]
},
{
@@ -424,7 +424,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.6.5"
+ "version": "3.8.5"
}
},
"nbformat": 4,
diff --git a/docs/src/auto_examples/core/run_corpora_and_vector_spaces.py b/docs/src/auto_examples/core/run_corpora_and_vector_spaces.py
index 0a49614123..983a9d1235 100644
--- a/docs/src/auto_examples/core/run_corpora_and_vector_spaces.py
+++ b/docs/src/auto_examples/core/run_corpora_and_vector_spaces.py
@@ -138,7 +138,7 @@
class MyCorpus:
def __iter__(self):
- for line in open('https://radimrehurek.com/gensim/mycorpus.txt'):
+ for line in open('https://radimrehurek.com/mycorpus.txt'):
# assume there's one document per line, tokens separated by whitespace
yield dictionary.doc2bow(line.lower().split())
@@ -154,7 +154,7 @@ def __iter__(self):
# in RAM at once. You can even create the documents on the fly!
###############################################################################
-# Download the sample `mycorpus.txt file here <./mycorpus.txt>`_. The assumption that
+# Download the sample `mycorpus.txt file here `_. The assumption that
# each document occupies one line in a single file is not important; you can mold
# the `__iter__` function to fit your input format, whatever it is.
# Walking directories, parsing XML, accessing the network...
@@ -180,7 +180,7 @@ def __iter__(self):
# Similarly, to construct the dictionary without loading all texts into memory:
# collect statistics about all tokens
-dictionary = corpora.Dictionary(line.lower().split() for line in open('https://radimrehurek.com/gensim/mycorpus.txt'))
+dictionary = corpora.Dictionary(line.lower().split() for line in open('https://radimrehurek.com/mycorpus.txt'))
# remove stop words and words that appear only once
stop_ids = [
dictionary.token2id[stopword]
diff --git a/docs/src/auto_examples/core/run_corpora_and_vector_spaces.py.md5 b/docs/src/auto_examples/core/run_corpora_and_vector_spaces.py.md5
index 935e0357af..174fe2a139 100644
--- a/docs/src/auto_examples/core/run_corpora_and_vector_spaces.py.md5
+++ b/docs/src/auto_examples/core/run_corpora_and_vector_spaces.py.md5
@@ -1 +1 @@
-6b98413399bca9fd1ed8fe420da85692
\ No newline at end of file
+55a8a886f05e5005c5f66d57569ee79d
\ No newline at end of file
diff --git a/docs/src/auto_examples/core/run_corpora_and_vector_spaces.rst b/docs/src/auto_examples/core/run_corpora_and_vector_spaces.rst
index 7f8d25cfec..3cc549dd65 100644
--- a/docs/src/auto_examples/core/run_corpora_and_vector_spaces.rst
+++ b/docs/src/auto_examples/core/run_corpora_and_vector_spaces.rst
@@ -1,12 +1,21 @@
+
+.. DO NOT EDIT.
+.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
+.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
+.. "auto_examples/core/run_corpora_and_vector_spaces.py"
+.. LINE NUMBERS ARE GIVEN BELOW.
+
.. only:: html
.. note::
:class: sphx-glr-download-link-note
- Click :ref:`here ` to download the full example code
- .. rst-class:: sphx-glr-example-title
+ Click :ref:`here `
+ to download the full example code
+
+.. rst-class:: sphx-glr-example-title
- .. _sphx_glr_auto_examples_core_run_corpora_and_vector_spaces.py:
+.. _sphx_glr_auto_examples_core_run_corpora_and_vector_spaces.py:
Corpora and Vector Spaces
@@ -16,6 +25,7 @@ Demonstrates transforming text into a vector space representation.
Also introduces corpus streaming and persistence to disk in various formats.
+.. GENERATED FROM PYTHON SOURCE LINES 9-13
.. code-block:: default
@@ -30,6 +40,8 @@ Also introduces corpus streaming and persistence to disk in various formats.
+.. GENERATED FROM PYTHON SOURCE LINES 14-23
+
First, let’s create a small corpus of nine short documents [1]_:
.. _second example:
@@ -40,6 +52,7 @@ From Strings to Vectors
This time, let's start from documents represented as strings:
+.. GENERATED FROM PYTHON SOURCE LINES 23-35
.. code-block:: default
@@ -62,11 +75,14 @@ This time, let's start from documents represented as strings:
+.. GENERATED FROM PYTHON SOURCE LINES 36-40
+
This is a tiny corpus of nine documents, each consisting of only a single sentence.
First, let's tokenize the documents, remove common words (using a toy stoplist)
as well as words that only appear once in the corpus:
+.. GENERATED FROM PYTHON SOURCE LINES 40-64
.. code-block:: default
@@ -117,6 +133,8 @@ as well as words that only appear once in the corpus:
+.. GENERATED FROM PYTHON SOURCE LINES 65-87
+
Your way of processing the documents will likely vary; here, I only split on whitespace
to tokenize, followed by lowercasing each word. In fact, I use this particular
(simplistic and inefficient) setup to mimic the experiment done in Deerwester et al.'s
@@ -140,6 +158,7 @@ a question-answer pair, in the style of:
It is advantageous to represent the questions only by their (integer) ids. The mapping
between the questions and ids is called a dictionary:
+.. GENERATED FROM PYTHON SOURCE LINES 87-93
.. code-block:: default
@@ -159,21 +178,25 @@ between the questions and ids is called a dictionary:
.. code-block:: none
- 2020-10-28 00:52:02,550 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
- 2020-10-28 00:52:02,550 : INFO : built Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...) from 9 documents (total 29 corpus positions)
- 2020-10-28 00:52:02,550 : INFO : saving Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...) under /tmp/deerwester.dict, separately None
- 2020-10-28 00:52:02,552 : INFO : saved /tmp/deerwester.dict
+ 2021-06-01 10:34:56,824 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
+ 2021-06-01 10:34:56,824 : INFO : built Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...) from 9 documents (total 29 corpus positions)
+ 2021-06-01 10:34:56,834 : INFO : Dictionary lifecycle event {'msg': "built Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...) from 9 documents (total 29 corpus positions)", 'datetime': '2021-06-01T10:34:56.825003', 'gensim': '4.1.0.dev0', 'python': '3.8.5 (default, Jan 27 2021, 15:41:15) \n[GCC 9.3.0]', 'platform': 'Linux-5.4.0-73-generic-x86_64-with-glibc2.29', 'event': 'created'}
+ 2021-06-01 10:34:56,834 : INFO : Dictionary lifecycle event {'fname_or_handle': '/tmp/deerwester.dict', 'separately': 'None', 'sep_limit': 10485760, 'ignore': frozenset(), 'datetime': '2021-06-01T10:34:56.834300', 'gensim': '4.1.0.dev0', 'python': '3.8.5 (default, Jan 27 2021, 15:41:15) \n[GCC 9.3.0]', 'platform': 'Linux-5.4.0-73-generic-x86_64-with-glibc2.29', 'event': 'saving'}
+ 2021-06-01 10:34:56,834 : INFO : saved /tmp/deerwester.dict
Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...)
+.. GENERATED FROM PYTHON SOURCE LINES 94-99
+
Here we assigned a unique integer id to all words appearing in the corpus with the
:class:`gensim.corpora.dictionary.Dictionary` class. This sweeps across the texts, collecting word counts
and relevant statistics. In the end, we see there are twelve distinct words in the
processed corpus, which means each document will be represented by twelve numbers (ie., by a 12-D vector).
To see the mapping between words and their ids:
+.. GENERATED FROM PYTHON SOURCE LINES 99-102
.. code-block:: default
@@ -195,8 +218,11 @@ To see the mapping between words and their ids:
+.. GENERATED FROM PYTHON SOURCE LINES 103-104
+
To actually convert tokenized documents to vectors:
+.. GENERATED FROM PYTHON SOURCE LINES 104-109
.. code-block:: default
@@ -220,12 +246,15 @@ To actually convert tokenized documents to vectors:
+.. GENERATED FROM PYTHON SOURCE LINES 110-115
+
The function :func:`doc2bow` simply counts the number of occurrences of
each distinct word, converts the word to its integer word id
and returns the result as a sparse vector. The sparse vector ``[(0, 1), (1, 1)]``
therefore reads: in the document `"Human computer interaction"`, the words `computer`
(id 0) and `human` (id 1) appear once; the other ten dictionary words appear (implicitly) zero times.
+.. GENERATED FROM PYTHON SOURCE LINES 115-120
.. code-block:: default
@@ -244,16 +273,18 @@ therefore reads: in the document `"Human computer interaction"`, the words `comp
.. code-block:: none
- 2020-10-28 00:52:02,830 : INFO : storing corpus in Matrix Market format to /tmp/deerwester.mm
- 2020-10-28 00:52:02,832 : INFO : saving sparse matrix to /tmp/deerwester.mm
- 2020-10-28 00:52:02,832 : INFO : PROGRESS: saving document #0
- 2020-10-28 00:52:02,834 : INFO : saved 9x12 matrix, density=25.926% (28/108)
- 2020-10-28 00:52:02,834 : INFO : saving MmCorpus index to /tmp/deerwester.mm.index
+ 2021-06-01 10:34:57,074 : INFO : storing corpus in Matrix Market format to /tmp/deerwester.mm
+ 2021-06-01 10:34:57,075 : INFO : saving sparse matrix to /tmp/deerwester.mm
+ 2021-06-01 10:34:57,075 : INFO : PROGRESS: saving document #0
+ 2021-06-01 10:34:57,076 : INFO : saved 9x12 matrix, density=25.926% (28/108)
+ 2021-06-01 10:34:57,076 : INFO : saving MmCorpus index to /tmp/deerwester.mm.index
[[(0, 1), (1, 1), (2, 1)], [(0, 1), (3, 1), (4, 1), (5, 1), (6, 1), (7, 1)], [(2, 1), (5, 1), (7, 1), (8, 1)], [(1, 1), (5, 2), (8, 1)], [(3, 1), (6, 1), (7, 1)], [(9, 1)], [(9, 1), (10, 1)], [(9, 1), (10, 1), (11, 1)], [(4, 1), (10, 1), (11, 1)]]
+.. GENERATED FROM PYTHON SOURCE LINES 121-136
+
By now it should be clear that the vector feature with ``id=10`` stands for the question "How many
times does the word `graph` appear in the document?" and that the answer is "zero" for
the first six documents and "one" for the remaining three.
@@ -270,6 +301,7 @@ Instead, let's assume the documents are stored in a file on disk, one document p
only requires that a corpus must be able to return one document vector at a time:
+.. GENERATED FROM PYTHON SOURCE LINES 136-145
.. code-block:: default
@@ -278,7 +310,7 @@ only requires that a corpus must be able to return one document vector at a time
class MyCorpus:
def __iter__(self):
- for line in open('https://radimrehurek.com/gensim/mycorpus.txt'):
+ for line in open('https://radimrehurek.com/mycorpus.txt'):
# assume there's one document per line, tokens separated by whitespace
yield dictionary.doc2bow(line.lower().split())
@@ -289,11 +321,14 @@ only requires that a corpus must be able to return one document vector at a time
+.. GENERATED FROM PYTHON SOURCE LINES 146-150
+
The full power of Gensim comes from the fact that a corpus doesn't have to be
a ``list``, or a ``NumPy`` array, or a ``Pandas`` dataframe, or whatever.
Gensim *accepts any object that, when iterated over, successively yields
documents*.
+.. GENERATED FROM PYTHON SOURCE LINES 150-156
.. code-block:: default
@@ -310,13 +345,16 @@ documents*.
-Download the sample `mycorpus.txt file here <./mycorpus.txt>`_. The assumption that
+.. GENERATED FROM PYTHON SOURCE LINES 157-163
+
+Download the sample `mycorpus.txt file here `_. The assumption that
each document occupies one line in a single file is not important; you can mold
the `__iter__` function to fit your input format, whatever it is.
Walking directories, parsing XML, accessing the network...
Just parse your input to retrieve a clean list of tokens in each document,
then convert the tokens via a dictionary to their ids and yield the resulting sparse vector inside `__iter__`.
+.. GENERATED FROM PYTHON SOURCE LINES 163-167
.. code-block:: default
@@ -334,15 +372,18 @@ then convert the tokens via a dictionary to their ids and yield the resulting sp
.. code-block:: none
- <__main__.MyCorpus object at 0x11e77bb38>
+ <__main__.MyCorpus object at 0x7f389b5f8520>
+.. GENERATED FROM PYTHON SOURCE LINES 168-171
+
Corpus is now an object. We didn't define any way to print it, so `print` just outputs address
of the object in memory. Not very useful. To see the constituent vectors, let's
iterate over the corpus and print each document vector (one at a time):
+.. GENERATED FROM PYTHON SOURCE LINES 171-175
.. code-block:: default
@@ -373,18 +414,21 @@ iterate over the corpus and print each document vector (one at a time):
+.. GENERATED FROM PYTHON SOURCE LINES 176-181
+
Although the output is the same as for the plain Python list, the corpus is now much
more memory friendly, because at most one vector resides in RAM at a time. Your
corpus can now be as large as you want.
Similarly, to construct the dictionary without loading all texts into memory:
+.. GENERATED FROM PYTHON SOURCE LINES 181-195
.. code-block:: default
# collect statistics about all tokens
- dictionary = corpora.Dictionary(line.lower().split() for line in open('https://radimrehurek.com/gensim/mycorpus.txt'))
+ dictionary = corpora.Dictionary(line.lower().split() for line in open('https://radimrehurek.com/mycorpus.txt'))
# remove stop words and words that appear only once
stop_ids = [
dictionary.token2id[stopword]
@@ -406,13 +450,16 @@ Similarly, to construct the dictionary without loading all texts into memory:
.. code-block:: none
- 2020-10-28 00:52:04,241 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
- 2020-10-28 00:52:04,243 : INFO : built Dictionary(42 unique tokens: ['abc', 'applications', 'computer', 'for', 'human']...) from 9 documents (total 69 corpus positions)
+ 2021-06-01 10:34:58,466 : INFO : adding document #0 to Dictionary(0 unique tokens: [])
+ 2021-06-01 10:34:58,467 : INFO : built Dictionary(42 unique tokens: ['abc', 'applications', 'computer', 'for', 'human']...) from 9 documents (total 69 corpus positions)
+ 2021-06-01 10:34:58,467 : INFO : Dictionary lifecycle event {'msg': "built Dictionary(42 unique tokens: ['abc', 'applications', 'computer', 'for', 'human']...) from 9 documents (total 69 corpus positions)", 'datetime': '2021-06-01T10:34:58.467454', 'gensim': '4.1.0.dev0', 'python': '3.8.5 (default, Jan 27 2021, 15:41:15) \n[GCC 9.3.0]', 'platform': 'Linux-5.4.0-73-generic-x86_64-with-glibc2.29', 'event': 'created'}
Dictionary(12 unique tokens: ['computer', 'human', 'interface', 'response', 'survey']...)
+.. GENERATED FROM PYTHON SOURCE LINES 196-219
+
And that is all there is to it! At least as far as bag-of-words representation is concerned.
Of course, what we do with such a corpus is another question; it is not at all clear
how counting the frequency of distinct words could be useful. As it turns out, it isn't, and
@@ -437,6 +484,7 @@ To save a corpus in the Matrix Market format:
create a toy corpus of 2 documents, as a plain Python list
+.. GENERATED FROM PYTHON SOURCE LINES 219-223
.. code-block:: default
@@ -454,19 +502,22 @@ create a toy corpus of 2 documents, as a plain Python list
.. code-block:: none
- 2020-10-28 00:52:04,368 : INFO : storing corpus in Matrix Market format to /tmp/corpus.mm
- 2020-10-28 00:52:04,370 : INFO : saving sparse matrix to /tmp/corpus.mm
- 2020-10-28 00:52:04,370 : INFO : PROGRESS: saving document #0
- 2020-10-28 00:52:04,370 : INFO : saved 2x2 matrix, density=25.000% (1/4)
- 2020-10-28 00:52:04,370 : INFO : saving MmCorpus index to /tmp/corpus.mm.index
+ 2021-06-01 10:34:58,603 : INFO : storing corpus in Matrix Market format to /tmp/corpus.mm
+ 2021-06-01 10:34:58,604 : INFO : saving sparse matrix to /tmp/corpus.mm
+ 2021-06-01 10:34:58,604 : INFO : PROGRESS: saving document #0
+ 2021-06-01 10:34:58,604 : INFO : saved 2x2 matrix, density=25.000% (1/4)
+ 2021-06-01 10:34:58,604 : INFO : saving MmCorpus index to /tmp/corpus.mm.index
+.. GENERATED FROM PYTHON SOURCE LINES 224-227
+
Other formats include `Joachim's SVMlight format `_
to help converting from/to numpy matrices
+.. GENERATED FROM PYTHON SOURCE LINES 275-282
.. code-block:: default
@@ -655,8 +724,11 @@ to help converting from/to numpy matrices
+.. GENERATED FROM PYTHON SOURCE LINES 283-284
+
and from/to `scipy.sparse` matrices
+.. GENERATED FROM PYTHON SOURCE LINES 284-290
.. code-block:: default
@@ -673,6 +745,8 @@ and from/to `scipy.sparse` matrices
+.. GENERATED FROM PYTHON SOURCE LINES 291-304
+
What Next
---------
@@ -687,6 +761,7 @@ Optimize converting between corpora and NumPy/SciPy arrays?), see the :ref:`apir
.. [1] This is the same corpus as used in
`Deerwester et al. (1990): Indexing by Latent Semantic Analysis `_, Table 2.
+.. GENERATED FROM PYTHON SOURCE LINES 304-310
.. code-block:: default
@@ -710,9 +785,9 @@ Optimize converting between corpora and NumPy/SciPy arrays?), see the :ref:`apir
.. rst-class:: sphx-glr-timing
- **Total running time of the script:** ( 0 minutes 4.010 seconds)
+ **Total running time of the script:** ( 0 minutes 3.242 seconds)
-**Estimated memory usage:** 40 MB
+**Estimated memory usage:** 48 MB
.. _sphx_glr_download_auto_examples_core_run_corpora_and_vector_spaces.py:
diff --git a/docs/src/auto_examples/core/sg_execution_times.rst b/docs/src/auto_examples/core/sg_execution_times.rst
index 9e36b38b09..da5c34f485 100644
--- a/docs/src/auto_examples/core/sg_execution_times.rst
+++ b/docs/src/auto_examples/core/sg_execution_times.rst
@@ -5,10 +5,10 @@
Computation times
=================
-**00:04.010** total execution time for **auto_examples_core** files:
+**00:03.242** total execution time for **auto_examples_core** files:
+--------------------------------------------------------------------------------------------------------------+-----------+---------+
-| :ref:`sphx_glr_auto_examples_core_run_corpora_and_vector_spaces.py` (``run_corpora_and_vector_spaces.py``) | 00:04.010 | 39.8 MB |
+| :ref:`sphx_glr_auto_examples_core_run_corpora_and_vector_spaces.py` (``run_corpora_and_vector_spaces.py``) | 00:03.242 | 48.2 MB |
+--------------------------------------------------------------------------------------------------------------+-----------+---------+
| :ref:`sphx_glr_auto_examples_core_run_core_concepts.py` (``run_core_concepts.py``) | 00:00.000 | 0.0 MB |
+--------------------------------------------------------------------------------------------------------------+-----------+---------+
diff --git a/docs/src/auto_examples/howtos/run_doc.ipynb b/docs/src/auto_examples/howtos/run_doc.ipynb
index 7d3be48ac2..4fab716e66 100644
--- a/docs/src/auto_examples/howtos/run_doc.ipynb
+++ b/docs/src/auto_examples/howtos/run_doc.ipynb
@@ -15,7 +15,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "\nHow to Author Gensim Documentation\n==================================\n\nHow to author documentation for Gensim.\n"
+ "\nHow to Author Gensim Documentation\n==================================\n\nHow to author documentation for Gensim.\n\n"
]
},
{
@@ -61,7 +61,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "Correctness\n-----------\n\nIncorrect documentation can be worse than no documentation at all.\nTake the following steps to ensure correctness:\n\n- Run Python's doctest module on your docstrings\n- Run your documentation scripts from scratch, removing any temporary files/results\n\nUsing data in your documentation\n--------------------------------\n\nSome parts of the documentation require real-world data to be useful.\nFor example, you may need more than just a toy example to demonstrate the benefits of one model over another.\nThis subsection provides some tips for including data in your documentation.\n\nIf possible, use data available via Gensim's\n`downloader API `__.\nThis will reduce the risk of your documentation becoming obsolete because required data is no longer available.\n\nUse the smallest possible dataset: avoid making people unnecessarily load large datasets and models.\nThis will make your documentation faster to run and easier for people to use (they can modify your examples and re-run them quickly).\n\nFinalizing your contribution\n----------------------------\n\nFirst, get Sphinx Gallery to build your documentation::\n\n make -C docs/src html\n\nThis can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.\nOnce this completes successfully, open ``docs/auto_examples/index.html`` in your browser.\nYou should see your new tutorial or guide in the gallery.\n\nOnce your documentation script is working correctly, it's time to add it to the git repository::\n\n git add docs/src/gallery/tutorials/run_example.py\n git add docs/src/auto_examples/tutorials/run_example.{py,py.md5,rst,ipynb}\n git add docs/src/auto_examples/howtos/sg_execution_times.rst\n git commit -m \"enter a helpful commit message here\"\n git push origin branchname\n\n.. Note::\n You may be wondering what all those other files are.\n Sphinx Gallery puts a copy of your Python script in ``auto_examples/tutorials``.\n The .md5 contains MD5 hash of the script to enable easy detection of modifications.\n Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.\n Finally, ``sg_execution_times.rst`` contains the time taken to run each example.\n\nFinally, make a PR on `github `__.\nOne of our friendly maintainers will review it, make suggestions, and eventually merge it.\nYour documentation will then appear in the gallery alongside the rest of the example.\nAt that stage, give yourself a pat on the back: you're done!\n\n"
+ "Correctness\n-----------\n\nIncorrect documentation can be worse than no documentation at all.\nTake the following steps to ensure correctness:\n\n- Run Python's doctest module on your docstrings\n- Run your documentation scripts from scratch, removing any temporary files/results\n\nUsing data in your documentation\n--------------------------------\n\nSome parts of the documentation require real-world data to be useful.\nFor example, you may need more than just a toy example to demonstrate the benefits of one model over another.\nThis subsection provides some tips for including data in your documentation.\n\nIf possible, use data available via Gensim's\n`downloader API `__.\nThis will reduce the risk of your documentation becoming obsolete because required data is no longer available.\n\nUse the smallest possible dataset: avoid making people unnecessarily load large datasets and models.\nThis will make your documentation faster to run and easier for people to use (they can modify your examples and re-run them quickly).\n\nFinalizing your contribution\n----------------------------\n\nFirst, get Sphinx Gallery to build your documentation::\n\n make --directory docs/src html\n\nThis can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.\nOnce this completes successfully, open ``docs/auto_examples/index.html`` in your browser.\nYou should see your new tutorial or guide in the gallery.\n\nOnce your documentation script is working correctly, it's time to add it to the git repository::\n\n git add docs/src/gallery/tutorials/run_example.py\n git add docs/src/auto_examples/tutorials/run_example.{py,py.md5,rst,ipynb}\n git add docs/src/auto_examples/howtos/sg_execution_times.rst\n git commit -m \"enter a helpful commit message here\"\n git push origin branchname\n\n.. Note::\n You may be wondering what all those other files are.\n Sphinx Gallery puts a copy of your Python script in ``auto_examples/tutorials``.\n The .md5 contains MD5 hash of the script to enable easy detection of modifications.\n Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.\n Finally, ``sg_execution_times.rst`` contains the time taken to run each example.\n\nFinally, open a PR at `github `__.\nOne of our friendly maintainers will review it, make suggestions, and eventually merge it.\nYour documentation will then appear in the `gallery `__,\nalongside the rest of the examples. Thanks a lot!\n\n"
]
}
],
@@ -81,7 +81,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.7.3"
+ "version": "3.6.5"
}
},
"nbformat": 4,
diff --git a/docs/src/auto_examples/howtos/run_doc.py b/docs/src/auto_examples/howtos/run_doc.py
index 15e870f1be..dbcd6d91e3 100644
--- a/docs/src/auto_examples/howtos/run_doc.py
+++ b/docs/src/auto_examples/howtos/run_doc.py
@@ -155,7 +155,7 @@
#
# First, get Sphinx Gallery to build your documentation::
#
-# make -C docs/src html
+# make --directory docs/src html
#
# This can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.
# Once this completes successfully, open ``docs/auto_examples/index.html`` in your browser.
@@ -176,7 +176,7 @@
# Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.
# Finally, ``sg_execution_times.rst`` contains the time taken to run each example.
#
-# Finally, make a PR on `github `__.
+# Finally, open a PR at `github `__.
# One of our friendly maintainers will review it, make suggestions, and eventually merge it.
-# Your documentation will then appear in the gallery alongside the rest of the example.
-# At that stage, give yourself a pat on the back: you're done!
+# Your documentation will then appear in the `gallery `__,
+# alongside the rest of the examples. Thanks a lot!
diff --git a/docs/src/auto_examples/howtos/run_doc.py.md5 b/docs/src/auto_examples/howtos/run_doc.py.md5
index 979aa0eb5e..66fecaa7cf 100644
--- a/docs/src/auto_examples/howtos/run_doc.py.md5
+++ b/docs/src/auto_examples/howtos/run_doc.py.md5
@@ -1 +1 @@
-512a76ce743dd12482d21784a76b60fe
\ No newline at end of file
+96cefb1417d54ac8010e38cc739d5ff1
\ No newline at end of file
diff --git a/docs/src/auto_examples/howtos/run_doc.rst b/docs/src/auto_examples/howtos/run_doc.rst
index c763ca1de0..dd4e957315 100644
--- a/docs/src/auto_examples/howtos/run_doc.rst
+++ b/docs/src/auto_examples/howtos/run_doc.rst
@@ -1,10 +1,12 @@
-.. note::
- :class: sphx-glr-download-link-note
+.. only:: html
+
+ .. note::
+ :class: sphx-glr-download-link-note
- Click :ref:`here ` to download the full example code
-.. rst-class:: sphx-glr-example-title
+ Click :ref:`here ` to download the full example code
+ .. rst-class:: sphx-glr-example-title
-.. _sphx_glr_auto_examples_howtos_run_doc.py:
+ .. _sphx_glr_auto_examples_howtos_run_doc.py:
How to Author Gensim Documentation
@@ -78,6 +80,15 @@ At the very top, you need a docstring describing what your script does.
+.. rst-class:: sphx-glr-script-out
+
+ Out:
+
+ .. code-block:: none
+
+
+ '\nTitle\n=====\n\nBrief description.\n'
+
The title is what will show up in the gallery.
@@ -167,7 +178,7 @@ Finalizing your contribution
First, get Sphinx Gallery to build your documentation::
- make -C docs/src html
+ make --directory docs/src html
This can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.
Once this completes successfully, open ``docs/auto_examples/index.html`` in your browser.
@@ -188,17 +199,17 @@ Once your documentation script is working correctly, it's time to add it to the
Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.
Finally, ``sg_execution_times.rst`` contains the time taken to run each example.
-Finally, make a PR on `github `__.
+Finally, open a PR at `github `__.
One of our friendly maintainers will review it, make suggestions, and eventually merge it.
-Your documentation will then appear in the gallery alongside the rest of the example.
-At that stage, give yourself a pat on the back: you're done!
+Your documentation will then appear in the `gallery `__,
+alongside the rest of the examples. Thanks a lot!
.. rst-class:: sphx-glr-timing
- **Total running time of the script:** ( 0 minutes 1.226 seconds)
+ **Total running time of the script:** ( 0 minutes 0.171 seconds)
-**Estimated memory usage:** 9 MB
+**Estimated memory usage:** 6 MB
.. _sphx_glr_download_auto_examples_howtos_run_doc.py:
@@ -211,13 +222,13 @@ At that stage, give yourself a pat on the back: you're done!
- .. container:: sphx-glr-download
+ .. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: run_doc.py `
- .. container:: sphx-glr-download
+ .. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: run_doc.ipynb `
diff --git a/docs/src/auto_examples/howtos/sg_execution_times.rst b/docs/src/auto_examples/howtos/sg_execution_times.rst
index ec9ea90bd7..e13e4ff9dc 100644
--- a/docs/src/auto_examples/howtos/sg_execution_times.rst
+++ b/docs/src/auto_examples/howtos/sg_execution_times.rst
@@ -5,9 +5,14 @@
Computation times
=================
-**00:01.226** total execution time for **auto_examples_howtos** files:
+**00:00.171** total execution time for **auto_examples_howtos** files:
-- **00:01.226**: :ref:`sphx_glr_auto_examples_howtos_run_doc.py` (``run_doc.py``)
-- **00:00.000**: :ref:`sphx_glr_auto_examples_howtos_run_compare_lda.py` (``run_compare_lda.py``)
-- **00:00.000**: :ref:`sphx_glr_auto_examples_howtos_run_doc2vec_imdb.py` (``run_doc2vec_imdb.py``)
-- **00:00.000**: :ref:`sphx_glr_auto_examples_howtos_run_downloader_api.py` (``run_downloader_api.py``)
++----------------------------------------------------------------------------------------+-----------+--------+
+| :ref:`sphx_glr_auto_examples_howtos_run_doc.py` (``run_doc.py``) | 00:00.171 | 6.1 MB |
++----------------------------------------------------------------------------------------+-----------+--------+
+| :ref:`sphx_glr_auto_examples_howtos_run_compare_lda.py` (``run_compare_lda.py``) | 00:00.000 | 0.0 MB |
++----------------------------------------------------------------------------------------+-----------+--------+
+| :ref:`sphx_glr_auto_examples_howtos_run_doc2vec_imdb.py` (``run_doc2vec_imdb.py``) | 00:00.000 | 0.0 MB |
++----------------------------------------------------------------------------------------+-----------+--------+
+| :ref:`sphx_glr_auto_examples_howtos_run_downloader_api.py` (``run_downloader_api.py``) | 00:00.000 | 0.0 MB |
++----------------------------------------------------------------------------------------+-----------+--------+
diff --git a/docs/src/auto_examples/index.rst b/docs/src/auto_examples/index.rst
index 1fa9eeca12..05643de00c 100644
--- a/docs/src/auto_examples/index.rst
+++ b/docs/src/auto_examples/index.rst
@@ -71,7 +71,7 @@ Understanding this functionality is vital for using gensim effectively.
.. raw:: html
-
Testing Gensim
+
+
.. only:: html
@@ -92,7 +92,7 @@ Understanding this functionality is vital for using gensim effectively.
.. raw:: html
-
+
.. only:: html
@@ -169,7 +169,7 @@ Learning-oriented lessons that introduce a particular gensim feature, e.g. a mod
.. raw:: html
-
+
.. only:: html
@@ -190,7 +190,28 @@ Learning-oriented lessons that introduce a particular gensim feature, e.g. a mod
.. raw:: html
-
+
+
+.. only:: html
+
+ .. figure:: /auto_examples/tutorials/images/thumb/sphx_glr_run_ensemblelda_thumb.png
+ :alt: Ensemble LDA
+
+ :ref:`sphx_glr_auto_examples_tutorials_run_ensemblelda.py`
+
+.. raw:: html
+
+
+
+
+.. toctree::
+ :hidden:
+
+ /auto_examples/tutorials/run_ensemblelda
+
+.. raw:: html
+
+
.. only:: html
@@ -288,7 +309,7 @@ These **goal-oriented guides** demonstrate how to **solve a specific problem** u
.. raw:: html
-
+
.. only:: html
@@ -309,7 +330,7 @@ These **goal-oriented guides** demonstrate how to **solve a specific problem** u
.. raw:: html
-
+
.. only:: html
@@ -426,13 +447,13 @@ Blog posts, tutorial videos, hackathons and other useful Gensim resources, from
.. container:: sphx-glr-download sphx-glr-download-python
- :download:`Download all examples in Python source code: auto_examples_python.zip `
+ :download:`Download all examples in Python source code: auto_examples_python.zip `
.. container:: sphx-glr-download sphx-glr-download-jupyter
- :download:`Download all examples in Jupyter notebooks: auto_examples_jupyter.zip `
+ :download:`Download all examples in Jupyter notebooks: auto_examples_jupyter.zip `
.. only:: html
diff --git a/docs/src/auto_examples/tutorials/images/thumb/sphx_glr_run_ensemblelda_thumb.png b/docs/src/auto_examples/tutorials/images/thumb/sphx_glr_run_ensemblelda_thumb.png
new file mode 100644
index 0000000000..233f8e605e
Binary files /dev/null and b/docs/src/auto_examples/tutorials/images/thumb/sphx_glr_run_ensemblelda_thumb.png differ
diff --git a/docs/src/auto_examples/tutorials/run_ensemblelda.ipynb b/docs/src/auto_examples/tutorials/run_ensemblelda.ipynb
new file mode 100644
index 0000000000..7cefa761aa
--- /dev/null
+++ b/docs/src/auto_examples/tutorials/run_ensemblelda.ipynb
@@ -0,0 +1,205 @@
+{
+ "cells": [
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "%matplotlib inline"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\nEnsemble LDA\n============\n\nIntroduces Gensim's EnsembleLda model\n\n\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import logging\nlogging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This tutorial will explain how to use the EnsembleLDA model class.\n\nEnsembleLda is a method of finding and generating stable topics from the results of multiple topic models,\nit can be used to remove topics from your results that are noise and are not reproducible.\n\n\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Corpus\n------\nWe will use the gensim downloader api to get a small corpus for training our ensemble.\n\nThe preprocessing is similar to `sphx_glr_auto_examples_tutorials_run_word2vec.py`,\nso it won't be explained again in detail.\n\n\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import gensim.downloader as api\nfrom gensim.corpora import Dictionary\nfrom nltk.stem.wordnet import WordNetLemmatizer\n\nlemmatizer = WordNetLemmatizer()\ndocs = api.load('text8')\n\ndictionary = Dictionary()\nfor doc in docs:\n dictionary.add_documents([[lemmatizer.lemmatize(token) for token in doc]])\ndictionary.filter_extremes(no_below=20, no_above=0.5)\n\ncorpus = [dictionary.doc2bow(doc) for doc in docs]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Training\n--------\n\nTraining the ensemble works very similar to training a single model,\n\nYou can use any model that is based on LdaModel, such as LdaMulticore, to train the Ensemble.\nIn experiments, LdaMulticore showed better results.\n\n\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "from gensim.models import LdaModel\ntopic_model_class = LdaModel"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Any arbitrary number of models can be used, but it should be a multiple of your workers so that the\nload can be distributed properly. In this example, 4 processes will train 8 models each.\n\n\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "ensemble_workers = 4\nnum_models = 8"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "After training all the models, some distance computations are required which can take quite some\ntime as well. You can speed this up by using workers for that as well.\n\n\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "distance_workers = 4"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "All other parameters that are unknown to EnsembleLda are forwarded to each LDA Model, such as\n\n\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "num_topics = 20\npasses = 2"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now start the training\n\nSince 20 topics were trained on each of the 8 models, we expect there to be 160 different topics.\nThe number of stable topics which are clustered from all those topics is smaller.\n\n\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "from gensim.models import EnsembleLda\nensemble = EnsembleLda(\n corpus=corpus,\n id2word=dictionary,\n num_topics=num_topics,\n passes=passes,\n num_models=num_models,\n topic_model_class=LdaModel,\n ensemble_workers=ensemble_workers,\n distance_workers=distance_workers\n)\n\nprint(len(ensemble.ttda))\nprint(len(ensemble.get_topics()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Tuning\n------\n\nDifferent from LdaModel, the number of resulting topics varies greatly depending on the clustering parameters.\n\nYou can provide those in the ``recluster()`` function or the ``EnsembleLda`` constructor.\n\nPlay around until you get as many topics as you desire, which however may reduce their quality.\nIf your ensemble doesn't have enough topics to begin with, you should make sure to make it large enough.\n\nHaving an epsilon that is smaller than the smallest distance doesn't make sense.\nMake sure to chose one that is within the range of values in ``asymmetric_distance_matrix``.\n\n\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "import numpy as np\nshape = ensemble.asymmetric_distance_matrix.shape\nwithout_diagonal = ensemble.asymmetric_distance_matrix[~np.eye(shape[0], dtype=bool)].reshape(shape[0], -1)\nprint(without_diagonal.min(), without_diagonal.mean(), without_diagonal.max())\n\nensemble.recluster(eps=0.09, min_samples=2, min_cores=2)\n\nprint(len(ensemble.get_topics()))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Increasing the Size\n-------------------\n\nIf you have some models lying around that were trained on a corpus based on the same dictionary,\nthey are compatible and you can add them to the ensemble.\n\nBy setting num_models of the EnsembleLda constructor to 0 you can also create an ensemble that is\nentirely made out of your existing topic models with the following method.\n\nAfterwards the number and quality of stable topics might be different depending on your added topics and parameters.\n\n\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": false
+ },
+ "outputs": [],
+ "source": [
+ "from gensim.models import LdaMulticore\n\nmodel1 = LdaMulticore(\n corpus=corpus,\n id2word=dictionary,\n num_topics=9,\n passes=4,\n)\n\nmodel2 = LdaModel(\n corpus=corpus,\n id2word=dictionary,\n num_topics=11,\n passes=2,\n)\n\n# add_model supports various types of input, check out its docstring\nensemble.add_model(model1)\nensemble.add_model(model2)\n\nensemble.recluster()\n\nprint(len(ensemble.ttda))\nprint(len(ensemble.get_topics()))"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.6.5"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 0
+}
\ No newline at end of file
diff --git a/docs/src/auto_examples/tutorials/run_ensemblelda.py b/docs/src/auto_examples/tutorials/run_ensemblelda.py
new file mode 100644
index 0000000000..aa87d0ecd3
--- /dev/null
+++ b/docs/src/auto_examples/tutorials/run_ensemblelda.py
@@ -0,0 +1,158 @@
+r"""
+Ensemble LDA
+============
+
+Introduces Gensim's EnsembleLda model
+
+"""
+
+import logging
+logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
+
+###############################################################################
+# This tutorial will explain how to use the EnsembleLDA model class.
+#
+# EnsembleLda is a method of finding and generating stable topics from the results of multiple topic models,
+# it can be used to remove topics from your results that are noise and are not reproducible.
+#
+
+###############################################################################
+# Corpus
+# ------
+# We will use the gensim downloader api to get a small corpus for training our ensemble.
+#
+# The preprocessing is similar to :ref:`sphx_glr_auto_examples_tutorials_run_word2vec.py`,
+# so it won't be explained again in detail.
+#
+
+import gensim.downloader as api
+from gensim.corpora import Dictionary
+from nltk.stem.wordnet import WordNetLemmatizer
+
+lemmatizer = WordNetLemmatizer()
+docs = api.load('text8')
+
+dictionary = Dictionary()
+for doc in docs:
+ dictionary.add_documents([[lemmatizer.lemmatize(token) for token in doc]])
+dictionary.filter_extremes(no_below=20, no_above=0.5)
+
+corpus = [dictionary.doc2bow(doc) for doc in docs]
+
+###############################################################################
+# Training
+# --------
+#
+# Training the ensemble works very similar to training a single model,
+#
+# You can use any model that is based on LdaModel, such as LdaMulticore, to train the Ensemble.
+# In experiments, LdaMulticore showed better results.
+#
+
+from gensim.models import LdaModel
+topic_model_class = LdaModel
+
+###############################################################################
+# Any arbitrary number of models can be used, but it should be a multiple of your workers so that the
+# load can be distributed properly. In this example, 4 processes will train 8 models each.
+#
+
+ensemble_workers = 4
+num_models = 8
+
+###############################################################################
+# After training all the models, some distance computations are required which can take quite some
+# time as well. You can speed this up by using workers for that as well.
+#
+
+distance_workers = 4
+
+###############################################################################
+# All other parameters that are unknown to EnsembleLda are forwarded to each LDA Model, such as
+#
+num_topics = 20
+passes = 2
+
+###############################################################################
+# Now start the training
+#
+# Since 20 topics were trained on each of the 8 models, we expect there to be 160 different topics.
+# The number of stable topics which are clustered from all those topics is smaller.
+#
+
+from gensim.models import EnsembleLda
+ensemble = EnsembleLda(
+ corpus=corpus,
+ id2word=dictionary,
+ num_topics=num_topics,
+ passes=passes,
+ num_models=num_models,
+ topic_model_class=LdaModel,
+ ensemble_workers=ensemble_workers,
+ distance_workers=distance_workers
+)
+
+print(len(ensemble.ttda))
+print(len(ensemble.get_topics()))
+
+###############################################################################
+# Tuning
+# ------
+#
+# Different from LdaModel, the number of resulting topics varies greatly depending on the clustering parameters.
+#
+# You can provide those in the ``recluster()`` function or the ``EnsembleLda`` constructor.
+#
+# Play around until you get as many topics as you desire, which however may reduce their quality.
+# If your ensemble doesn't have enough topics to begin with, you should make sure to make it large enough.
+#
+# Having an epsilon that is smaller than the smallest distance doesn't make sense.
+# Make sure to chose one that is within the range of values in ``asymmetric_distance_matrix``.
+#
+
+import numpy as np
+shape = ensemble.asymmetric_distance_matrix.shape
+without_diagonal = ensemble.asymmetric_distance_matrix[~np.eye(shape[0], dtype=bool)].reshape(shape[0], -1)
+print(without_diagonal.min(), without_diagonal.mean(), without_diagonal.max())
+
+ensemble.recluster(eps=0.09, min_samples=2, min_cores=2)
+
+print(len(ensemble.get_topics()))
+
+###############################################################################
+# Increasing the Size
+# -------------------
+#
+# If you have some models lying around that were trained on a corpus based on the same dictionary,
+# they are compatible and you can add them to the ensemble.
+#
+# By setting num_models of the EnsembleLda constructor to 0 you can also create an ensemble that is
+# entirely made out of your existing topic models with the following method.
+#
+# Afterwards the number and quality of stable topics might be different depending on your added topics and parameters.
+#
+
+from gensim.models import LdaMulticore
+
+model1 = LdaMulticore(
+ corpus=corpus,
+ id2word=dictionary,
+ num_topics=9,
+ passes=4,
+)
+
+model2 = LdaModel(
+ corpus=corpus,
+ id2word=dictionary,
+ num_topics=11,
+ passes=2,
+)
+
+# add_model supports various types of input, check out its docstring
+ensemble.add_model(model1)
+ensemble.add_model(model2)
+
+ensemble.recluster()
+
+print(len(ensemble.ttda))
+print(len(ensemble.get_topics()))
diff --git a/docs/src/auto_examples/tutorials/run_ensemblelda.py.md5 b/docs/src/auto_examples/tutorials/run_ensemblelda.py.md5
new file mode 100644
index 0000000000..f09f123fba
--- /dev/null
+++ b/docs/src/auto_examples/tutorials/run_ensemblelda.py.md5
@@ -0,0 +1 @@
+be0c32b18644ebb1a7826764b37ebc01
\ No newline at end of file
diff --git a/docs/src/auto_examples/tutorials/run_ensemblelda.rst b/docs/src/auto_examples/tutorials/run_ensemblelda.rst
new file mode 100644
index 0000000000..f554213238
--- /dev/null
+++ b/docs/src/auto_examples/tutorials/run_ensemblelda.rst
@@ -0,0 +1,3832 @@
+.. only:: html
+
+ .. note::
+ :class: sphx-glr-download-link-note
+
+ Click :ref:`here ` to download the full example code
+ .. rst-class:: sphx-glr-example-title
+
+ .. _sphx_glr_auto_examples_tutorials_run_ensemblelda.py:
+
+
+Ensemble LDA
+============
+
+Introduces Gensim's EnsembleLda model
+
+
+
+.. code-block:: default
+
+
+ import logging
+ logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
+
+
+
+
+
+
+
+
+This tutorial will explain how to use the EnsembleLDA model class.
+
+EnsembleLda is a method of finding and generating stable topics from the results of multiple topic models,
+it can be used to remove topics from your results that are noise and are not reproducible.
+
+
+Corpus
+------
+We will use the gensim downloader api to get a small corpus for training our ensemble.
+
+The preprocessing is similar to :ref:`sphx_glr_auto_examples_tutorials_run_word2vec.py`,
+so it won't be explained again in detail.
+
+
+
+.. code-block:: default
+
+
+ import gensim.downloader as api
+ from gensim.corpora import Dictionary
+ from nltk.stem.wordnet import WordNetLemmatizer
+
+ lemmatizer = WordNetLemmatizer()
+ docs = api.load('text8')
+
+ dictionary = Dictionary()
+ for doc in docs:
+ dictionary.add_documents([[lemmatizer.lemmatize(token) for token in doc]])
+ dictionary.filter_extremes(no_below=20, no_above=0.5)
+
+ corpus = [dictionary.doc2bow(doc) for doc in docs]
+
+
+
+
+
+.. rst-class:: sphx-glr-script-out
+
+ Out:
+
+ .. code-block:: none
+
+ /Volumes/work/workspace/gensim/trunk/gensim/similarities/__init__.py:15: UserWarning: The gensim.similarities.levenshtein submodule is disabled, because the optional Levenshtein package `
+
+
+
+ .. container:: sphx-glr-download sphx-glr-download-jupyter
+
+ :download:`Download Jupyter notebook: run_ensemblelda.ipynb `
+
+
+.. only:: html
+
+ .. rst-class:: sphx-glr-signature
+
+ `Gallery generated by Sphinx-Gallery `_
diff --git a/docs/src/auto_examples/tutorials/sg_execution_times.rst b/docs/src/auto_examples/tutorials/sg_execution_times.rst
index bb21c1ef5c..da986968c9 100644
--- a/docs/src/auto_examples/tutorials/sg_execution_times.rst
+++ b/docs/src/auto_examples/tutorials/sg_execution_times.rst
@@ -5,10 +5,10 @@
Computation times
=================
-**02:47.007** total execution time for **auto_examples_tutorials** files:
+**08:55.221** total execution time for **auto_examples_tutorials** files:
+-------------------------------------------------------------------------------------+-----------+----------+
-| :ref:`sphx_glr_auto_examples_tutorials_run_lda.py` (``run_lda.py``) | 02:47.007 | 657.5 MB |
+| :ref:`sphx_glr_auto_examples_tutorials_run_ensemblelda.py` (``run_ensemblelda.py``) | 08:55.221 | 506.6 MB |
+-------------------------------------------------------------------------------------+-----------+----------+
| :ref:`sphx_glr_auto_examples_tutorials_run_annoy.py` (``run_annoy.py``) | 00:00.000 | 0.0 MB |
+-------------------------------------------------------------------------------------+-----------+----------+
@@ -16,6 +16,8 @@ Computation times
+-------------------------------------------------------------------------------------+-----------+----------+
| :ref:`sphx_glr_auto_examples_tutorials_run_fasttext.py` (``run_fasttext.py``) | 00:00.000 | 0.0 MB |
+-------------------------------------------------------------------------------------+-----------+----------+
+| :ref:`sphx_glr_auto_examples_tutorials_run_lda.py` (``run_lda.py``) | 00:00.000 | 0.0 MB |
++-------------------------------------------------------------------------------------+-----------+----------+
| :ref:`sphx_glr_auto_examples_tutorials_run_scm.py` (``run_scm.py``) | 00:00.000 | 0.0 MB |
+-------------------------------------------------------------------------------------+-----------+----------+
| :ref:`sphx_glr_auto_examples_tutorials_run_wmd.py` (``run_wmd.py``) | 00:00.000 | 0.0 MB |
diff --git a/docs/src/conf.py b/docs/src/conf.py
index 34eb52c9de..5ea73d184d 100644
--- a/docs/src/conf.py
+++ b/docs/src/conf.py
@@ -61,9 +61,9 @@
# built documents.
#
# The short X.Y version.
-version = '4.0.0'
+version = '4.1'
# The full version, including alpha/beta/rc tags.
-release = '4.0.1'
+release = '4.1.0'
# The language for content autogenerated by Sphinx. Refer to documentation
# for a list of supported languages.
diff --git a/docs/src/corpora/opinosiscorpus.rst b/docs/src/corpora/opinosiscorpus.rst
new file mode 100644
index 0000000000..3f62454677
--- /dev/null
+++ b/docs/src/corpora/opinosiscorpus.rst
@@ -0,0 +1,9 @@
+:mod:`corpora.opinosiscorpus` -- Topic related review sentences
+===============================================================
+
+.. automodule:: gensim.corpora.opinosiscorpus
+ :synopsis: Topic related review sentences
+ :members:
+ :inherited-members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/src/gallery/core/run_corpora_and_vector_spaces.py b/docs/src/gallery/core/run_corpora_and_vector_spaces.py
index 0a49614123..983a9d1235 100644
--- a/docs/src/gallery/core/run_corpora_and_vector_spaces.py
+++ b/docs/src/gallery/core/run_corpora_and_vector_spaces.py
@@ -138,7 +138,7 @@
class MyCorpus:
def __iter__(self):
- for line in open('https://radimrehurek.com/gensim/mycorpus.txt'):
+ for line in open('https://radimrehurek.com/mycorpus.txt'):
# assume there's one document per line, tokens separated by whitespace
yield dictionary.doc2bow(line.lower().split())
@@ -154,7 +154,7 @@ def __iter__(self):
# in RAM at once. You can even create the documents on the fly!
###############################################################################
-# Download the sample `mycorpus.txt file here <./mycorpus.txt>`_. The assumption that
+# Download the sample `mycorpus.txt file here `_. The assumption that
# each document occupies one line in a single file is not important; you can mold
# the `__iter__` function to fit your input format, whatever it is.
# Walking directories, parsing XML, accessing the network...
@@ -180,7 +180,7 @@ def __iter__(self):
# Similarly, to construct the dictionary without loading all texts into memory:
# collect statistics about all tokens
-dictionary = corpora.Dictionary(line.lower().split() for line in open('https://radimrehurek.com/gensim/mycorpus.txt'))
+dictionary = corpora.Dictionary(line.lower().split() for line in open('https://radimrehurek.com/mycorpus.txt'))
# remove stop words and words that appear only once
stop_ids = [
dictionary.token2id[stopword]
diff --git a/docs/src/gallery/howtos/run_doc.py b/docs/src/gallery/howtos/run_doc.py
index 15e870f1be..dbcd6d91e3 100644
--- a/docs/src/gallery/howtos/run_doc.py
+++ b/docs/src/gallery/howtos/run_doc.py
@@ -155,7 +155,7 @@
#
# First, get Sphinx Gallery to build your documentation::
#
-# make -C docs/src html
+# make --directory docs/src html
#
# This can take a while if your documentation uses a large dataset, or if you've changed many other tutorials or guides.
# Once this completes successfully, open ``docs/auto_examples/index.html`` in your browser.
@@ -176,7 +176,7 @@
# Gallery also generates .rst (RST for Sphinx) and .ipynb (Jupyter notebook) files from the script.
# Finally, ``sg_execution_times.rst`` contains the time taken to run each example.
#
-# Finally, make a PR on `github `__.
+# Finally, open a PR at `github `__.
# One of our friendly maintainers will review it, make suggestions, and eventually merge it.
-# Your documentation will then appear in the gallery alongside the rest of the example.
-# At that stage, give yourself a pat on the back: you're done!
+# Your documentation will then appear in the `gallery `__,
+# alongside the rest of the examples. Thanks a lot!
diff --git a/docs/src/gallery/tutorials/run_ensemblelda.py b/docs/src/gallery/tutorials/run_ensemblelda.py
new file mode 100644
index 0000000000..aa87d0ecd3
--- /dev/null
+++ b/docs/src/gallery/tutorials/run_ensemblelda.py
@@ -0,0 +1,158 @@
+r"""
+Ensemble LDA
+============
+
+Introduces Gensim's EnsembleLda model
+
+"""
+
+import logging
+logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)
+
+###############################################################################
+# This tutorial will explain how to use the EnsembleLDA model class.
+#
+# EnsembleLda is a method of finding and generating stable topics from the results of multiple topic models,
+# it can be used to remove topics from your results that are noise and are not reproducible.
+#
+
+###############################################################################
+# Corpus
+# ------
+# We will use the gensim downloader api to get a small corpus for training our ensemble.
+#
+# The preprocessing is similar to :ref:`sphx_glr_auto_examples_tutorials_run_word2vec.py`,
+# so it won't be explained again in detail.
+#
+
+import gensim.downloader as api
+from gensim.corpora import Dictionary
+from nltk.stem.wordnet import WordNetLemmatizer
+
+lemmatizer = WordNetLemmatizer()
+docs = api.load('text8')
+
+dictionary = Dictionary()
+for doc in docs:
+ dictionary.add_documents([[lemmatizer.lemmatize(token) for token in doc]])
+dictionary.filter_extremes(no_below=20, no_above=0.5)
+
+corpus = [dictionary.doc2bow(doc) for doc in docs]
+
+###############################################################################
+# Training
+# --------
+#
+# Training the ensemble works very similar to training a single model,
+#
+# You can use any model that is based on LdaModel, such as LdaMulticore, to train the Ensemble.
+# In experiments, LdaMulticore showed better results.
+#
+
+from gensim.models import LdaModel
+topic_model_class = LdaModel
+
+###############################################################################
+# Any arbitrary number of models can be used, but it should be a multiple of your workers so that the
+# load can be distributed properly. In this example, 4 processes will train 8 models each.
+#
+
+ensemble_workers = 4
+num_models = 8
+
+###############################################################################
+# After training all the models, some distance computations are required which can take quite some
+# time as well. You can speed this up by using workers for that as well.
+#
+
+distance_workers = 4
+
+###############################################################################
+# All other parameters that are unknown to EnsembleLda are forwarded to each LDA Model, such as
+#
+num_topics = 20
+passes = 2
+
+###############################################################################
+# Now start the training
+#
+# Since 20 topics were trained on each of the 8 models, we expect there to be 160 different topics.
+# The number of stable topics which are clustered from all those topics is smaller.
+#
+
+from gensim.models import EnsembleLda
+ensemble = EnsembleLda(
+ corpus=corpus,
+ id2word=dictionary,
+ num_topics=num_topics,
+ passes=passes,
+ num_models=num_models,
+ topic_model_class=LdaModel,
+ ensemble_workers=ensemble_workers,
+ distance_workers=distance_workers
+)
+
+print(len(ensemble.ttda))
+print(len(ensemble.get_topics()))
+
+###############################################################################
+# Tuning
+# ------
+#
+# Different from LdaModel, the number of resulting topics varies greatly depending on the clustering parameters.
+#
+# You can provide those in the ``recluster()`` function or the ``EnsembleLda`` constructor.
+#
+# Play around until you get as many topics as you desire, which however may reduce their quality.
+# If your ensemble doesn't have enough topics to begin with, you should make sure to make it large enough.
+#
+# Having an epsilon that is smaller than the smallest distance doesn't make sense.
+# Make sure to chose one that is within the range of values in ``asymmetric_distance_matrix``.
+#
+
+import numpy as np
+shape = ensemble.asymmetric_distance_matrix.shape
+without_diagonal = ensemble.asymmetric_distance_matrix[~np.eye(shape[0], dtype=bool)].reshape(shape[0], -1)
+print(without_diagonal.min(), without_diagonal.mean(), without_diagonal.max())
+
+ensemble.recluster(eps=0.09, min_samples=2, min_cores=2)
+
+print(len(ensemble.get_topics()))
+
+###############################################################################
+# Increasing the Size
+# -------------------
+#
+# If you have some models lying around that were trained on a corpus based on the same dictionary,
+# they are compatible and you can add them to the ensemble.
+#
+# By setting num_models of the EnsembleLda constructor to 0 you can also create an ensemble that is
+# entirely made out of your existing topic models with the following method.
+#
+# Afterwards the number and quality of stable topics might be different depending on your added topics and parameters.
+#
+
+from gensim.models import LdaMulticore
+
+model1 = LdaMulticore(
+ corpus=corpus,
+ id2word=dictionary,
+ num_topics=9,
+ passes=4,
+)
+
+model2 = LdaModel(
+ corpus=corpus,
+ id2word=dictionary,
+ num_topics=11,
+ passes=2,
+)
+
+# add_model supports various types of input, check out its docstring
+ensemble.add_model(model1)
+ensemble.add_model(model2)
+
+ensemble.recluster()
+
+print(len(ensemble.ttda))
+print(len(ensemble.get_topics()))
diff --git a/docs/src/models/ensemblelda.rst b/docs/src/models/ensemblelda.rst
new file mode 100644
index 0000000000..42e63c94be
--- /dev/null
+++ b/docs/src/models/ensemblelda.rst
@@ -0,0 +1,9 @@
+:mod:`models.ensembelda` -- Ensemble Latent Dirichlet Allocation
+================================================================
+
+.. automodule:: gensim.models.ensemblelda
+ :synopsis: Ensemble Latent Dirichlet Allocation
+ :members:
+ :inherited-members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/docs/src/people.rst b/docs/src/people.rst
index f75b652926..99dbd33d9e 100644
--- a/docs/src/people.rst
+++ b/docs/src/people.rst
@@ -48,4 +48,12 @@ Silver Sponsors
Bronze Sponsors
---------------
-`You? `_
+.. figure:: _static/images/eaccidents-logo.png
+ :target: https://eaccidents.com/
+ :width: 50%
+ :alt: EAccidents
+
+.. figure:: _static/images/techtarget-logo.png
+ :target: https://www.techtarget.com/
+ :width: 50%
+ :alt: TechTarget
diff --git a/docs/src/sphinx_rtd_theme/notification.html b/docs/src/sphinx_rtd_theme/notification.html
index e0c483e40a..4b0d7922e1 100644
--- a/docs/src/sphinx_rtd_theme/notification.html
+++ b/docs/src/sphinx_rtd_theme/notification.html
@@ -1,5 +1,5 @@
- You're viewing documentation for Gensim 4.0.0. For Gensim 3.8.3, please visit the old Gensim 3.8.3 documentation.
+ You're viewing documentation for Gensim 4.0.0. For Gensim 3.8.3, please visit the old Gensim 3.8.3 documentation and Migration Guide.
diff --git a/docs/src/wiki.rst b/docs/src/wiki.rst
index 40e7c6343f..800e1b9c65 100644
--- a/docs/src/wiki.rst
+++ b/docs/src/wiki.rst
@@ -236,5 +236,5 @@ into LDA topic distributions:
By the way, improvements to the Wiki markup parsing code are welcome :-)
.. [3] Hoffman, Blei, Bach. 2010. Online learning for Latent Dirichlet Allocation
- [`pdf `_] [`code `_] [`code `_]
diff --git a/gensim/__init__.py b/gensim/__init__.py
index 84e9fc6463..899afc7591 100644
--- a/gensim/__init__.py
+++ b/gensim/__init__.py
@@ -4,7 +4,7 @@
"""
-__version__ = '4.0.1'
+__version__ = '4.1.0'
import logging
diff --git a/gensim/corpora/__init__.py b/gensim/corpora/__init__.py
index 0d51a9b903..7b7044e0b9 100644
--- a/gensim/corpora/__init__.py
+++ b/gensim/corpora/__init__.py
@@ -15,3 +15,4 @@
from .textcorpus import TextCorpus, TextDirectoryCorpus # noqa:F401
from .ucicorpus import UciCorpus # noqa:F401
from .malletcorpus import MalletCorpus # noqa:F401
+from .opinosiscorpus import OpinosisCorpus # noqa:F401
diff --git a/gensim/corpora/dictionary.py b/gensim/corpora/dictionary.py
index 3236dd081e..d954061caf 100644
--- a/gensim/corpora/dictionary.py
+++ b/gensim/corpora/dictionary.py
@@ -10,6 +10,7 @@
from collections.abc import Mapping
import logging
import itertools
+from typing import Optional, List, Tuple
from gensim import utils
@@ -25,9 +26,7 @@ class Dictionary(utils.SaveLoad, Mapping):
Attributes
----------
token2id : dict of (str, int)
- token -> tokenId.
- id2token : dict of (int, str)
- Reverse mapping for token2id, initialized in a lazy manner to save memory (not created until needed).
+ token -> token_id. I.e. the reverse mapping to `self[token_id]`.
cfs : dict of (int, int)
Collection frequencies: token_id -> how many instances of this token are contained in the documents.
dfs : dict of (int, int)
@@ -689,6 +688,30 @@ def load_from_text(fname):
result.dfs[wordid] = int(docfreq)
return result
+ def most_common(self, n: Optional[int] = None) -> List[Tuple[str, int]]:
+ """Return a list of the n most common words and their counts from the most common to the least.
+
+ Words with equal counts are ordered in the increasing order of their ids.
+
+ Parameters
+ ----------
+ n : int or None, optional
+ The number of most common words to be returned. If `None`, all words in the dictionary
+ will be returned. Default is `None`.
+
+ Returns
+ -------
+ most_common : list of (str, int)
+ The n most common words and their counts from the most common to the least.
+
+ """
+ most_common = [
+ (self[word], count)
+ for word, count
+ in sorted(self.cfs.items(), key=lambda x: (-x[1], x[0]))[:n]
+ ]
+ return most_common
+
@staticmethod
def from_corpus(corpus, id2word=None):
"""Create :class:`~gensim.corpora.dictionary.Dictionary` from an existing corpus.
diff --git a/gensim/corpora/lowcorpus.py b/gensim/corpora/lowcorpus.py
index 80dacf8ec0..01b1043a9c 100644
--- a/gensim/corpora/lowcorpus.py
+++ b/gensim/corpora/lowcorpus.py
@@ -11,28 +11,11 @@
from gensim import utils
from gensim.corpora import IndexedCorpus
-
+from gensim.parsing.preprocessing import split_on_space
logger = logging.getLogger(__name__)
-def split_on_space(s):
- """Split line by spaces, used in :class:`gensim.corpora.lowcorpus.LowCorpus`.
-
- Parameters
- ----------
- s : str
- Some line.
-
- Returns
- -------
- list of str
- List of tokens from `s`.
-
- """
- return [word for word in utils.to_unicode(s).strip().split(' ') if word]
-
-
class LowCorpus(IndexedCorpus):
"""Corpus handles input in `GibbsLda++ format
+# Copyright (C) 2021 Radim Rehurek
+# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
+
+
+"""Creates a corpus and dictionary from the Opinosis dataset.
+
+References
+----------
+.. [1] Ganesan, Kavita and Zhai, ChengXiang and Han, Jiawei. Opinosis: a graph-based approach to abstractive
+ summarization of highly redundant opinions [online]. In : Proceedings of the 23rd International Conference on
+ Computational Linguistics. 2010. p. 340-348. Available from: https://kavita-ganesan.com/opinosis/
+"""
+
+import os
+import re
+from gensim.corpora import Dictionary
+from gensim.parsing.porter import PorterStemmer
+from gensim.parsing.preprocessing import STOPWORDS
+
+
+class OpinosisCorpus:
+ """Creates a corpus and dictionary from the Opinosis dataset.
+
+ http://kavita-ganesan.com/opinosis-opinion-dataset/
+
+ This data is organized in folders, each folder containing a few short docs.
+
+ Data can be obtained quickly using the following commands in bash:
+
+ mkdir opinosis && cd opinosis
+ wget https://github.com/kavgan/opinosis/raw/master/OpinosisDataset1.0_0.zip
+ unzip OpinosisDataset1.0_0.zip
+
+ corpus and dictionary can be accessed by using the .corpus and .id2word members
+
+ """
+
+ def __init__(self, path):
+ """Load the downloaded corpus.
+
+ Parameters
+ ----------
+ path : string
+ Path to the extracted zip file. If 'summaries-gold' is in a folder
+ called 'opinosis', then the Path parameter would be 'opinosis',
+ either relative to you current working directory or absolute.
+ """
+ # citation
+ path = os.path.join(path, "summaries-gold")
+ dictionary = Dictionary()
+ corpus = []
+ stemmer = PorterStemmer()
+
+ for directory, b, filenames in os.walk(path):
+ # each subdirectory of path is one collection of reviews to a specific product
+ # now get the corpus/documents
+ for filename in filenames:
+ filepath = directory + os.sep + filename
+ # write down the document and the topicId and split into train and testdata
+ with open(filepath) as file:
+ doc = file.read()
+
+ preprocessed_doc = [
+ stemmer.stem(token) for token in re.findall(r'\w+', doc.lower())
+ if token not in STOPWORDS
+ ]
+
+ dictionary.add_documents([preprocessed_doc])
+ corpus += [dictionary.doc2bow(preprocessed_doc)]
+
+ # and return the results the same way the other corpus generating functions do
+ self.corpus = corpus
+ self.id2word = dictionary
diff --git a/gensim/corpora/textcorpus.py b/gensim/corpora/textcorpus.py
index e5616fe9d7..c2b8b620bf 100644
--- a/gensim/corpora/textcorpus.py
+++ b/gensim/corpora/textcorpus.py
@@ -44,93 +44,15 @@
from gensim import interfaces, utils
from gensim.corpora.dictionary import Dictionary
-from gensim.parsing.preprocessing import STOPWORDS, RE_WHITESPACE
+from gensim.parsing.preprocessing import (
+ remove_stopword_tokens, remove_short_tokens,
+ lower_to_unicode, strip_multiple_whitespaces,
+)
from gensim.utils import deaccent, simple_tokenize
logger = logging.getLogger(__name__)
-def remove_stopwords(tokens, stopwords=STOPWORDS):
- """Remove stopwords using list from `gensim.parsing.preprocessing.STOPWORDS`.
-
- Parameters
- ----------
- tokens : iterable of str
- Sequence of tokens.
- stopwords : iterable of str, optional
- Sequence of stopwords
-
- Returns
- -------
- list of str
- List of tokens without `stopwords`.
-
- """
- return [token for token in tokens if token not in stopwords]
-
-
-def remove_short(tokens, minsize=3):
- """Remove tokens shorter than `minsize` chars.
-
- Parameters
- ----------
- tokens : iterable of str
- Sequence of tokens.
- minsize : int, optimal
- Minimal length of token (include).
-
- Returns
- -------
- list of str
- List of tokens without short tokens.
-
- """
- return [token for token in tokens if len(token) >= minsize]
-
-
-def lower_to_unicode(text, encoding='utf8', errors='strict'):
- """Lowercase `text` and convert to unicode, using :func:`gensim.utils.any2unicode`.
-
- Parameters
- ----------
- text : str
- Input text.
- encoding : str, optional
- Encoding that will be used for conversion.
- errors : str, optional
- Error handling behaviour, used as parameter for `unicode` function (python2 only).
-
- Returns
- -------
- str
- Unicode version of `text`.
-
- See Also
- --------
- :func:`gensim.utils.any2unicode`
- Convert any string to unicode-string.
-
- """
- return utils.to_unicode(text.lower(), encoding, errors)
-
-
-def strip_multiple_whitespaces(s):
- """Collapse multiple whitespace characters into a single space.
-
- Parameters
- ----------
- s : str
- Input string
-
- Returns
- -------
- str
- String with collapsed whitespaces.
-
- """
- return RE_WHITESPACE.sub(" ", s)
-
-
class TextCorpus(interfaces.CorpusABC):
"""Helper class to simplify the pipeline of getting BoW vectors from plain text.
@@ -177,12 +99,12 @@ class TextCorpus(interfaces.CorpusABC):
The default preprocessing consists of:
- #. :func:`~gensim.corpora.textcorpus.lower_to_unicode` - lowercase and convert to unicode (assumes utf8 encoding)
+ #. :func:`~gensim.parsing.preprocessing.lower_to_unicode` - lowercase and convert to unicode (assumes utf8 encoding)
#. :func:`~gensim.utils.deaccent`- deaccent (asciifolding)
- #. :func:`~gensim.corpora.textcorpus.strip_multiple_whitespaces` - collapse multiple whitespaces into a single one
+ #. :func:`~gensim.parsing.preprocessing.strip_multiple_whitespaces` - collapse multiple whitespaces into one
#. :func:`~gensim.utils.simple_tokenize` - tokenize by splitting on whitespace
- #. :func:`~gensim.corpora.textcorpus.remove_short` - remove words less than 3 characters long
- #. :func:`~gensim.corpora.textcorpus.remove_stopwords` - remove stopwords
+ #. :func:`~gensim.parsing.preprocessing.remove_short_tokens` - remove words less than 3 characters long
+ #. :func:`~gensim.parsing.preprocessing.remove_stopword_tokens` - remove stopwords
"""
@@ -204,15 +126,15 @@ def __init__(self, input=None, dictionary=None, metadata=False, character_filter
Each will be applied to the text of each document in order, and should return a single string with
the modified text. For Python 2, the original text will not be unicode, so it may be useful to
convert to unicode as the first character filter.
- If None - using :func:`~gensim.corpora.textcorpus.lower_to_unicode`,
- :func:`~gensim.utils.deaccent` and :func:`~gensim.corpora.textcorpus.strip_multiple_whitespaces`.
+ If None - using :func:`~gensim.parsing.preprocessing.lower_to_unicode`,
+ :func:`~gensim.utils.deaccent` and :func:`~gensim.parsing.preprocessing.strip_multiple_whitespaces`.
tokenizer : callable, optional
Tokenizer for document, if None - using :func:`~gensim.utils.simple_tokenize`.
token_filters : iterable of callable, optional
Each will be applied to the iterable of tokens in order, and should return another iterable of tokens.
These filters can add, remove, or replace tokens, or do nothing at all.
- If None - using :func:`~gensim.corpora.textcorpus.remove_short` and
- :func:`~gensim.corpora.textcorpus.remove_stopwords`.
+ If None - using :func:`~gensim.parsing.preprocessing.remove_short_tokens` and
+ :func:`~gensim.parsing.preprocessing.remove_stopword_tokens`.
Examples
--------
@@ -254,7 +176,7 @@ def __init__(self, input=None, dictionary=None, metadata=False, character_filter
self.token_filters = token_filters
if self.token_filters is None:
- self.token_filters = [remove_short, remove_stopwords]
+ self.token_filters = [remove_short_tokens, remove_stopword_tokens]
self.length = None
self.dictionary = None
diff --git a/gensim/models/__init__.py b/gensim/models/__init__.py
index 8aa19a0465..ac08d1fdb4 100644
--- a/gensim/models/__init__.py
+++ b/gensim/models/__init__.py
@@ -21,6 +21,8 @@
from .ldaseqmodel import LdaSeqModel # noqa:F401
from .fasttext import FastText # noqa:F401
from .translation_matrix import TranslationMatrix, BackMappingTranslationMatrix # noqa:F401
+from .ensemblelda import EnsembleLda # noqa:F401
+from .nmf import Nmf # noqa:F401
from gensim import interfaces, utils
diff --git a/gensim/models/atmodel.py b/gensim/models/atmodel.py
index 2fceb00307..838c7634e3 100755
--- a/gensim/models/atmodel.py
+++ b/gensim/models/atmodel.py
@@ -23,6 +23,9 @@
`_. The model correlates the authorship information with the topics to give a better
insight on the subject knowledge of an author.
+.. _'Online Learning for LDA' by Hoffman et al.: online-lda_
+.. _online-lda: https://papers.neurips.cc/paper/2010/file/71f6278d140af599e06ad9bf1ba03cb0-Paper.pdf
+
Example
-------
.. sourcecode:: pycon
@@ -185,15 +188,30 @@ def __init__(self, corpus=None, num_topics=100, id2word=None, author2doc=None, d
iterations : int, optional
Maximum number of times the model loops over each document.
decay : float, optional
- Controls how old documents are forgotten.
+ A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten
+ when each new document is examined. Corresponds to :math:`\\kappa` from
+ `'Online Learning for LDA' by Hoffman et al.`_
offset : float, optional
- Controls down-weighting of iterations.
- alpha : float, optional
- Hyperparameters for author-topic model.Supports special values of 'asymmetric'
- and 'auto': the former uses a fixed normalized asymmetric 1.0/topicno prior,
- the latter learns an asymmetric prior directly from your data.
- eta : float, optional
- Hyperparameters for author-topic model.
+ Hyper-parameter that controls how much we will slow down the first steps the first few iterations.
+ Corresponds to :math:`\\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_
+ alpha : {float, numpy.ndarray of float, list of float, str}, optional
+ A-priori belief on document-topic distribution, this can be:
+ * scalar for a symmetric prior over document-topic distribution,
+ * 1D array of length equal to num_topics to denote an asymmetric user defined prior for each topic.
+
+ Alternatively default prior selecting strategies can be employed by supplying a string:
+ * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`,
+ * 'asymmetric': Uses a fixed normalized asymmetric prior of `1.0 / (topic_index + sqrt(num_topics))`,
+ * 'auto': Learns an asymmetric prior from the corpus (not available if `distributed==True`).
+ eta : {float, numpy.ndarray of float, list of float, str}, optional
+ A-priori belief on topic-word distribution, this can be:
+ * scalar for a symmetric prior over topic-word distribution,
+ * 1D array of length equal to num_words to denote an asymmetric user defined prior for each word,
+ * matrix of shape (num_topics, num_words) to assign a probability for each word-topic combination.
+
+ Alternatively default prior selecting strategies can be employed by supplying a string:
+ * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`,
+ * 'auto': Learns an asymmetric prior from the corpus.
update_every : int, optional
Make updates in topic probability for latest mini-batch.
eval_every : int, optional
@@ -279,23 +297,17 @@ def __init__(self, corpus=None, num_topics=100, id2word=None, author2doc=None, d
self.init_empty_corpus()
self.alpha, self.optimize_alpha = self.init_dir_prior(alpha, 'alpha')
-
assert self.alpha.shape == (self.num_topics,), \
"Invalid alpha shape. Got shape %s, but expected (%d, )" % (str(self.alpha.shape), self.num_topics)
- if isinstance(eta, str):
- if eta == 'asymmetric':
- raise ValueError("The 'asymmetric' option cannot be used for eta")
-
self.eta, self.optimize_eta = self.init_dir_prior(eta, 'eta')
-
- self.random_state = utils.get_random_state(random_state)
-
assert (self.eta.shape == (self.num_terms,) or self.eta.shape == (self.num_topics, self.num_terms)), (
"Invalid eta shape. Got shape %s, but expected (%d, 1) or (%d, %d)" %
(str(self.eta.shape), self.num_terms, self.num_topics, self.num_terms)
)
+ self.random_state = utils.get_random_state(random_state)
+
# VB constants
self.iterations = iterations
self.gamma_threshold = gamma_threshold
@@ -486,9 +498,12 @@ def inference(self, chunk, author2doc, doc2author, rhot, collect_sstats=False, c
# Update gamma.
# phi is computed implicitly below,
+ dot = np.dot(cts / phinorm, expElogbetad.T)
for ai, a in enumerate(authors_d):
- tilde_gamma[ai, :] = self.alpha + len(self.author2doc[self.id2author[a]])\
- * expElogthetad[ai, :] * np.dot(cts / phinorm, expElogbetad.T)
+ tilde_gamma[ai, :] = (
+ self.alpha
+ + len(self.author2doc[self.id2author[a]]) * expElogthetad[ai, :] * dot
+ )
# Update gamma.
# Interpolation between document d's "local" gamma (tilde_gamma),
@@ -612,15 +627,14 @@ def update(self, corpus=None, author2doc=None, doc2author=None, chunksize=None,
Notes
-----
- This update also supports updating an already trained model (self)
- with new documents from `corpus`: the two models are then merged in proportion to the number of old vs. new
- documents. This feature is still experimental for non-stationary input streams.
+ This update also supports updating an already trained model (`self`) with new documents from `corpus`;
+ the two models are then merged in proportion to the number of old vs. new documents.
+ This feature is still experimental for non-stationary input streams.
- For stationary input (no topic drift in new documents), on the other hand, this equals the online update of
- `Hoffman et al. Stochastic Variational Inference
- `_ and is guaranteed to converge for any `decay`
- in (0.5, 1.0>. Additionally, for smaller `corpus` sizes, an increasing `offset` may be beneficial (see
- Table 1 in Hoffman et al.)
+ For stationary input (no topic drift in new documents), on the other hand, this equals the
+ online update of `'Online Learning for LDA' by Hoffman et al.`_
+ and is guaranteed to converge for any `decay` in (0.5, 1]. Additionally, for smaller corpus sizes, an
+ increasing `offset` may be beneficial (see Table 1 in the same paper).
If update is called with authors that already exist in the model, it will resume training on not only new
documents for that author, but also the previously seen documents. This is necessary for those authors' topic
@@ -647,9 +661,12 @@ def update(self, corpus=None, author2doc=None, doc2author=None, chunksize=None,
chunksize : int, optional
Controls the size of the mini-batches.
decay : float, optional
- Controls how old documents are forgotten.
+ A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten
+ when each new document is examined. Corresponds to :math:`\\kappa` from
+ `'Online Learning for LDA' by Hoffman et al.`_
offset : float, optional
- Controls down-weighting of iterations.
+ Hyper-parameter that controls how much we will slow down the first steps the first few iterations.
+ Corresponds to :math:`\\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_
passes : int, optional
Number of times the model makes a pass over the entire training data.
update_every : int, optional
diff --git a/gensim/models/coherencemodel.py b/gensim/models/coherencemodel.py
index 70fea79804..b3c89640a7 100644
--- a/gensim/models/coherencemodel.py
+++ b/gensim/models/coherencemodel.py
@@ -444,11 +444,14 @@ def topics(self, topics):
self._topics = new_topics
def _ensure_elements_are_ids(self, topic):
- try:
- return np.array([self.dictionary.token2id[token] for token in topic])
- except KeyError: # might be a list of token ids already, but let's verify all in dict
- topic = (self.dictionary.id2token[_id] for _id in topic)
- return np.array([self.dictionary.token2id[token] for token in topic])
+ ids_from_tokens = [self.dictionary.token2id[t] for t in topic if t in self.dictionary.token2id]
+ ids_from_ids = [i for i in topic if i in self.dictionary]
+ if len(ids_from_tokens) > len(ids_from_ids):
+ return np.array(ids_from_tokens)
+ elif len(ids_from_ids) > len(ids_from_tokens):
+ return np.array(ids_from_ids)
+ else:
+ raise ValueError('unable to interpret topic as either a list of tokens or a list of ids')
def _update_accumulator(self, new_topics):
if self._relevant_ids_will_differ(new_topics):
diff --git a/gensim/models/doc2vec.py b/gensim/models/doc2vec.py
index 0dcad7165a..c4b28316b7 100644
--- a/gensim/models/doc2vec.py
+++ b/gensim/models/doc2vec.py
@@ -21,7 +21,7 @@
`_.
For a usage example, see the `Doc2vec tutorial
-`_.
+`_.
**Make sure you have a C compiler before installing Gensim, to use the optimized doc2vec routines** (70x speedup
compared to plain NumPy implementation, https://rare-technologies.com/parallelizing-word2vec-in-python/).
@@ -158,7 +158,7 @@ def count(self, new_val):
class Doc2Vec(Word2Vec):
def __init__(self, documents=None, corpus_file=None, vector_size=100, dm_mean=None, dm=1, dbow_words=0, dm_concat=0,
dm_tag_count=1, dv=None, dv_mapfile=None, comment=None, trim_rule=None, callbacks=(),
- window=5, epochs=10, **kwargs):
+ window=5, epochs=10, shrink_windows=True, **kwargs):
"""Class for training, using and evaluating neural networks described in
`Distributed Representations of Sentences and Documents `_.
@@ -248,6 +248,12 @@ def __init__(self, documents=None, corpus_file=None, vector_size=100, dm_mean=No
callbacks : :obj: `list` of :obj: `~gensim.models.callbacks.CallbackAny2Vec`, optional
List of callbacks that need to be executed/run at specific stages during training.
+ shrink_windows : bool, optional
+ New in 4.1. Experimental.
+ If True, the effective window size is uniformly sampled from [1, `window`]
+ for each target word during training, to match the original word2vec algorithm's
+ approximate weighting of context words by distance. Otherwise, the effective
+ window size is always fixed to `window` words to either side.
Some important internal attributes are the following:
@@ -265,6 +271,7 @@ def __init__(self, documents=None, corpus_file=None, vector_size=100, dm_mean=No
.. sourcecode:: pycon
>>> model.dv['doc003']
+
"""
corpus_iterable = documents
@@ -293,6 +300,7 @@ def __init__(self, documents=None, corpus_file=None, vector_size=100, dm_mean=No
callbacks=callbacks,
window=window,
epochs=epochs,
+ shrink_windows=shrink_windows,
**kwargs,
)
@@ -357,8 +365,10 @@ def reset_from(self, other_model):
self.dv.expandos = other_model.dv.expandos
self.init_weights()
- def _do_train_epoch(self, corpus_file, thread_id, offset, cython_vocab, thread_private_mem, cur_epoch,
- total_examples=None, total_words=None, offsets=None, start_doctags=None, **kwargs):
+ def _do_train_epoch(
+ self, corpus_file, thread_id, offset, cython_vocab, thread_private_mem, cur_epoch,
+ total_examples=None, total_words=None, offsets=None, start_doctags=None, **kwargs
+ ):
work, neu1 = thread_private_mem
doctag_vectors = self.dv.vectors
doctags_lockf = self.dv.vectors_lockf
@@ -425,10 +435,12 @@ def _do_train_job(self, job, alpha, inits):
)
return tally, self._raw_word_count(job)
- def train(self, corpus_iterable=None, corpus_file=None, total_examples=None, total_words=None,
- epochs=None, start_alpha=None, end_alpha=None,
- word_count=0, queue_factor=2, report_delay=1.0, callbacks=(),
- **kwargs):
+ def train(
+ self, corpus_iterable=None, corpus_file=None, total_examples=None, total_words=None,
+ epochs=None, start_alpha=None, end_alpha=None,
+ word_count=0, queue_factor=2, report_delay=1.0, callbacks=(),
+ **kwargs,
+ ):
"""Update the model's neural weights.
To support linear learning-rate decay from (initial) `alpha` to `min_alpha`, and accurate
@@ -576,13 +588,13 @@ def estimated_lookup_memory(self):
"""
return 60 * len(self.dv) + 140 * len(self.dv)
- def infer_vector(self, doc_words, alpha=None, min_alpha=None, epochs=None, steps=None):
+ def infer_vector(self, doc_words, alpha=None, min_alpha=None, epochs=None):
"""Infer a vector for given post-bulk training document.
Notes
-----
Subsequent calls to this function may infer different representations for the same document.
- For a more stable representation, increase the number of steps to assert a stricket convergence.
+ For a more stable representation, increase the number of epochs to assert a stricter convergence.
Parameters
----------
@@ -795,7 +807,7 @@ def load(cls, *args, **kwargs):
return super(Doc2Vec, cls).load(*args, rethrow=True, **kwargs)
except AttributeError as ae:
logger.error(
- "Model load error. Was model saved using code from an older Gensim Version? "
+ "Model load error. Was model saved using code from an older Gensim version? "
"Try loading older model using gensim-3.8.3, then re-saving, to restore "
"compatibility with current code.")
raise ae
@@ -1042,7 +1054,7 @@ def scan_vocab(self, corpus_iterable=None, corpus_file=None, progress_per=10000,
return total_words, corpus_count
- def similarity_unseen_docs(self, doc_words1, doc_words2, alpha=None, min_alpha=None, steps=None):
+ def similarity_unseen_docs(self, doc_words1, doc_words2, alpha=None, min_alpha=None, epochs=None):
"""Compute cosine similarity between two post-bulk out of training documents.
Parameters
@@ -1057,7 +1069,7 @@ def similarity_unseen_docs(self, doc_words1, doc_words2, alpha=None, min_alpha=N
The initial learning rate.
min_alpha : float, optional
Learning rate will linearly drop to `min_alpha` as training progresses.
- steps : int, optional
+ epochs : int, optional
Number of epoch to train the new document.
Returns
@@ -1066,8 +1078,8 @@ def similarity_unseen_docs(self, doc_words1, doc_words2, alpha=None, min_alpha=N
The cosine similarity between `doc_words1` and `doc_words2`.
"""
- d1 = self.infer_vector(doc_words=doc_words1, alpha=alpha, min_alpha=min_alpha, steps=steps)
- d2 = self.infer_vector(doc_words=doc_words2, alpha=alpha, min_alpha=min_alpha, steps=steps)
+ d1 = self.infer_vector(doc_words=doc_words1, alpha=alpha, min_alpha=min_alpha, epochs=epochs)
+ d2 = self.infer_vector(doc_words=doc_words2, alpha=alpha, min_alpha=min_alpha, epochs=epochs)
return np.dot(matutils.unitvec(d1), matutils.unitvec(d2))
diff --git a/gensim/models/doc2vec_corpusfile.pyx b/gensim/models/doc2vec_corpusfile.pyx
index 40bf20bdd3..9216d13bd4 100644
--- a/gensim/models/doc2vec_corpusfile.pyx
+++ b/gensim/models/doc2vec_corpusfile.pyx
@@ -59,7 +59,7 @@ cdef void prepare_c_structures_for_batch(
int *effective_words, unsigned long long *next_random, cvocab_t *vocab,
np.uint32_t *indexes, int *codelens, np.uint8_t **codes, np.uint32_t **points,
np.uint32_t *reduced_windows, int *document_len, int train_words,
- int docvecs_count, int doc_tag,
+ int docvecs_count, int doc_tag, int shrink_windows,
) nogil:
cdef VocabItem predict_word
cdef string token
@@ -87,8 +87,12 @@ cdef void prepare_c_structures_for_batch(
document_len[0] = i
if train_words and reduced_windows != NULL:
- for i in range(document_len[0]):
- reduced_windows[i] = random_int32(next_random) % window
+ if shrink_windows:
+ for i in range(document_len[0]):
+ reduced_windows[i] = random_int32(next_random) % window
+ else:
+ for i in range(document_len[0]):
+ reduced_windows[i] = 0
if doc_tag < docvecs_count:
effective_words[0] += 1
@@ -160,6 +164,7 @@ def d2v_train_epoch_dbow(
cdef long long total_documents = 0
cdef long long total_effective_words = 0, total_words = 0
cdef int sent_idx, idx_start, idx_end
+ cdef int shrink_windows = int(model.shrink_windows)
cdef vector[string] doc_words
cdef long long _doc_tag = start_doctag
@@ -183,7 +188,7 @@ def d2v_train_epoch_dbow(
prepare_c_structures_for_batch(
doc_words, c.sample, c.hs, c.window, &total_words, &effective_words,
&c.next_random, vocab.get_vocab_ptr(), c.indexes, c.codelens, c.codes, c.points,
- c.reduced_windows, &document_len, c.train_words, c.docvecs_count, _doc_tag)
+ c.reduced_windows, &document_len, c.train_words, c.docvecs_count, _doc_tag, shrink_windows)
for i in range(document_len):
if c.train_words: # simultaneous skip-gram wordvec-training
@@ -300,6 +305,7 @@ def d2v_train_epoch_dm(
cdef long long total_effective_words = 0, total_words = 0
cdef int sent_idx, idx_start, idx_end
cdef REAL_t count, inv_count = 1.0
+ cdef int shrink_windows = int(model.shrink_windows)
cdef vector[string] doc_words
cdef long long _doc_tag = start_doctag
@@ -323,7 +329,7 @@ def d2v_train_epoch_dm(
prepare_c_structures_for_batch(
doc_words, c.sample, c.hs, c.window, &total_words, &effective_words, &c.next_random,
vocab.get_vocab_ptr(), c.indexes, c.codelens, c.codes, c.points, c.reduced_windows,
- &document_len, c.train_words, c.docvecs_count, _doc_tag)
+ &document_len, c.train_words, c.docvecs_count, _doc_tag, shrink_windows)
for i in range(document_len):
j = i - c.window + c.reduced_windows[i]
@@ -453,6 +459,7 @@ def d2v_train_epoch_dm_concat(
cdef long long total_documents = 0
cdef long long total_effective_words = 0, total_words = 0
cdef int sent_idx, idx_start, idx_end
+ cdef int shrink_windows = int(model.shrink_windows)
cdef vector[string] doc_words
cdef long long _doc_tag = start_doctag
@@ -490,7 +497,8 @@ def d2v_train_epoch_dm_concat(
prepare_c_structures_for_batch(
doc_words, c.sample, c.hs, c.window, &total_words, &effective_words,
&c.next_random, vocab.get_vocab_ptr(), c.indexes, c.codelens, c.codes,
- c.points, NULL, &document_len, c.train_words, c.docvecs_count, _doc_tag)
+ c.points, NULL, &document_len, c.train_words, c.docvecs_count, _doc_tag,
+ shrink_windows)
for i in range(document_len):
j = i - c.window # negative OK: will pad with null word
diff --git a/gensim/models/doc2vec_inner.pyx b/gensim/models/doc2vec_inner.pyx
index 23ede53c90..1657c59787 100644
--- a/gensim/models/doc2vec_inner.pyx
+++ b/gensim/models/doc2vec_inner.pyx
@@ -365,8 +365,12 @@ def train_document_dbow(model, doc_words, doctag_indexes, alpha, work=None,
if c.train_words:
# single randint() call avoids a big thread-synchronization slowdown
- for i, item in enumerate(model.random.randint(0, c.window, c.document_len)):
- c.reduced_windows[i] = item
+ if model.shrink_windows:
+ for i, item in enumerate(model.random.randint(0, c.window, c.document_len)):
+ c.reduced_windows[i] = item
+ else:
+ for i in range(c.document_len):
+ c.reduced_windows[i] = 0
for i in range(c.doctag_len):
c.doctag_indexes[i] = doctag_indexes[i]
@@ -497,8 +501,12 @@ def train_document_dm(model, doc_words, doctag_indexes, alpha, work=None, neu1=N
c.document_len = i
# single randint() call avoids a big thread-sync slowdown
- for i, item in enumerate(model.random.randint(0, c.window, c.document_len)):
- c.reduced_windows[i] = item
+ if model.shrink_windows:
+ for i, item in enumerate(model.random.randint(0, c.window, c.document_len)):
+ c.reduced_windows[i] = item
+ else:
+ for i in range(c.document_len):
+ c.reduced_windows[i] = 0
for i in range(c.doctag_len):
c.doctag_indexes[i] = doctag_indexes[i]
diff --git a/gensim/models/ensemblelda.py b/gensim/models/ensemblelda.py
new file mode 100644
index 0000000000..39d7e06620
--- /dev/null
+++ b/gensim/models/ensemblelda.py
@@ -0,0 +1,1366 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Authors: Tobias Brigl , Alex Salles ,
+# Alex Loosley , Data Reply Munich
+# Copyright (C) 2021 Radim Rehurek
+# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
+
+
+"""Ensemble Latent Dirichlet Allocation (eLDA), an algorithm for extracting reliable topics.
+
+The aim of topic modelling is to find a set of topics that represent the global structure of a corpus of documents. One
+issue that occurs with topics extracted from an NMF or LDA model is reproducibility. That is, if the topic model is
+trained repeatedly allowing only the random seed to change, would the same (or similar) topic representation be reliably
+learned. Unreliable topics are undesireable because they are not a good representation of the corpus.
+
+Ensemble LDA addresses the issue by training an ensemble of topic models and throwing out topics that do not reoccur
+across the ensemble. In this regard, the topics extracted are more reliable and there is the added benefit over many
+topic models that the user does not need to know the exact number of topics ahead of time.
+
+For more information, see the :ref:`citation section ` below, watch our `Machine Learning Prague 2019 talk
+`_,
+or view our `Machine Learning Summer School poster
+`_.
+
+Usage examples
+--------------
+
+Train an ensemble of LdaModels using a Gensim corpus:
+
+.. sourcecode:: pycon
+
+ >>> from gensim.test.utils import common_texts
+ >>> from gensim.corpora.dictionary import Dictionary
+ >>> from gensim.models import EnsembleLda
+ >>>
+ >>> # Create a corpus from a list of texts
+ >>> common_dictionary = Dictionary(common_texts)
+ >>> common_corpus = [common_dictionary.doc2bow(text) for text in common_texts]
+ >>>
+ >>> # Train the model on the corpus. corpus has to be provided as a
+ >>> # keyword argument, as they are passed through to the children.
+ >>> elda = EnsembleLda(corpus=common_corpus, id2word=common_dictionary, num_topics=10, num_models=4)
+
+Save a model to disk, or reload a pre-trained model:
+
+.. sourcecode:: pycon
+
+ >>> from gensim.test.utils import datapath
+ >>>
+ >>> # Save model to disk.
+ >>> temp_file = datapath("model")
+ >>> elda.save(temp_file)
+ >>>
+ >>> # Load a potentially pretrained model from disk.
+ >>> elda = EnsembleLda.load(temp_file)
+
+Query, the model using new, unseen documents:
+
+.. sourcecode:: pycon
+
+ >>> # Create a new corpus, made of previously unseen documents.
+ >>> other_texts = [
+ ... ['computer', 'time', 'graph'],
+ ... ['survey', 'response', 'eps'],
+ ... ['human', 'system', 'computer']
+ ... ]
+ >>> other_corpus = [common_dictionary.doc2bow(text) for text in other_texts]
+ >>>
+ >>> unseen_doc = other_corpus[0]
+ >>> vector = elda[unseen_doc] # get topic probability distribution for a document
+
+Increase the ensemble size by adding a new model. Make sure it uses the same dictionary:
+
+.. sourcecode:: pycon
+
+ >>> from gensim.models import LdaModel
+ >>> elda.add_model(LdaModel(common_corpus, id2word=common_dictionary, num_topics=10))
+ >>> elda.recluster()
+ >>> vector = elda[unseen_doc]
+
+To optimize the ensemble for your specific case, the children can be clustered again using
+different hyperparameters:
+
+.. sourcecode:: pycon
+
+ >>> elda.recluster(eps=0.2)
+
+.. _Citation:
+
+Citation
+--------
+BRIGL, Tobias, 2019, Extracting Reliable Topics using Ensemble Latent Dirichlet Allocation [Bachelor Thesis].
+Technische Hochschule Ingolstadt. Munich: Data Reply GmbH. Available from:
+https://www.sezanzeb.de/machine_learning/ensemble_LDA/
+
+"""
+import logging
+import os
+from multiprocessing import Process, Pipe, ProcessError
+import importlib
+from typing import Set, Optional, List
+
+import numpy as np
+from scipy.spatial.distance import cosine
+from dataclasses import dataclass
+
+from gensim import utils
+from gensim.models import ldamodel, ldamulticore, basemodel
+from gensim.utils import SaveLoad
+
+
+logger = logging.getLogger(__name__)
+
+# _COSINE_DISTANCE_CALCULATION_THRESHOLD is used so that cosine distance calculations can be sped up by skipping
+# distance calculations for highly masked topic-term distributions
+_COSINE_DISTANCE_CALCULATION_THRESHOLD = 0.05
+
+# nps max random state of 2**32 - 1 is too large for windows
+_MAX_RANDOM_STATE = np.iinfo(np.int32).max
+
+
+@dataclass
+class Topic:
+ is_core: bool # if the topic has enough neighbors
+ neighboring_labels: Set[int] # which other clusters are close by
+ neighboring_topic_indices: Set[int] # which other topics are close by
+ label: Optional[int] # to which cluster this topic belongs
+ num_neighboring_labels: int # how many different labels a core has as parents
+ valid_neighboring_labels: Set[int] # A set of labels of close by clusters that are large enough
+
+
+@dataclass
+class Cluster:
+ max_num_neighboring_labels: int # the max number of parent labels among each topic of a given cluster
+ neighboring_labels: List[Set[int]] # a concatenated list of the neighboring_labels sets of each topic
+ label: int # the unique identifier of the cluster
+ num_cores: int # how many topics in the cluster are cores
+
+
+def _is_valid_core(topic):
+ """Check if the topic is a valid core, i.e. no neighboring valid cluster is overlapping with it.
+
+ Parameters
+ ----------
+ topic : :class:`Topic`
+ topic to validate
+
+ """
+ return topic.is_core and (topic.valid_neighboring_labels == {topic.label})
+
+
+def _remove_from_all_sets(label, clusters):
+ """Remove a label from every set in "neighboring_labels" for each core in ``clusters``."""
+ for cluster in clusters:
+ for neighboring_labels_set in cluster.neighboring_labels:
+ if label in neighboring_labels_set:
+ neighboring_labels_set.remove(label)
+
+
+def _contains_isolated_cores(label, cluster, min_cores):
+ """Check if the cluster has at least ``min_cores`` of cores that belong to no other cluster."""
+ return sum([neighboring_labels == {label} for neighboring_labels in cluster.neighboring_labels]) >= min_cores
+
+
+def _aggregate_topics(grouped_by_labels):
+ """Aggregate the labeled topics to a list of clusters.
+
+ Parameters
+ ----------
+ grouped_by_labels : dict of (int, list of :class:`Topic`)
+ The return value of _group_by_labels. A mapping of the label to a list of each topic which belongs to the
+ label.
+
+ Returns
+ -------
+ list of :class:`Cluster`
+ It is sorted by max_num_neighboring_labels in descending order. There is one single element for each cluster.
+
+ """
+ clusters = []
+
+ for label, topics in grouped_by_labels.items():
+ max_num_neighboring_labels = 0
+ neighboring_labels = [] # will be a list of sets
+
+ for topic in topics:
+ max_num_neighboring_labels = max(topic.num_neighboring_labels, max_num_neighboring_labels)
+ neighboring_labels.append(topic.neighboring_labels)
+
+ neighboring_labels = [x for x in neighboring_labels if len(x) > 0]
+
+ clusters.append(Cluster(
+ max_num_neighboring_labels=max_num_neighboring_labels,
+ neighboring_labels=neighboring_labels,
+ label=label,
+ num_cores=len([topic for topic in topics if topic.is_core]),
+ ))
+
+ logger.info("found %s clusters", len(clusters))
+
+ return clusters
+
+
+def _group_by_labels(cbdbscan_topics):
+ """Group all the learned cores by their label, which was assigned in the cluster_model.
+
+ Parameters
+ ----------
+ cbdbscan_topics : list of :class:`Topic`
+ A list of topic data resulting from fitting a :class:`~CBDBSCAN` object.
+ After calling .fit on a CBDBSCAN model, the results can be retrieved from it by accessing the .results
+ member, which can be used as the argument to this function. It is a list of infos gathered during
+ the clustering step and each element in the list corresponds to a single topic.
+
+ Returns
+ -------
+ dict of (int, list of :class:`Topic`)
+ A mapping of the label to a list of topics that belong to that particular label. Also adds
+ a new member to each topic called num_neighboring_labels, which is the number of
+ neighboring_labels of that topic.
+
+ """
+ grouped_by_labels = {}
+
+ for topic in cbdbscan_topics:
+ if topic.is_core:
+ topic.num_neighboring_labels = len(topic.neighboring_labels)
+
+ label = topic.label
+ if label not in grouped_by_labels:
+ grouped_by_labels[label] = []
+ grouped_by_labels[label].append(topic)
+
+ return grouped_by_labels
+
+
+def _teardown(pipes, processes, i):
+ """Close pipes and terminate processes.
+
+ Parameters
+ ----------
+ pipes : {list of :class:`multiprocessing.Pipe`}
+ list of pipes that the processes use to communicate with the parent
+ processes : {list of :class:`multiprocessing.Process`}
+ list of worker processes
+ """
+ for parent_conn, child_conn in pipes:
+ child_conn.close()
+ parent_conn.close()
+
+ for process in processes:
+ if process.is_alive():
+ process.terminate()
+ del process
+
+
+def mass_masking(a, threshold=None):
+ """Original masking method. Returns a new binary mask."""
+ if threshold is None:
+ threshold = 0.95
+
+ sorted_a = np.sort(a)[::-1]
+ largest_mass = sorted_a.cumsum() < threshold
+ smallest_valid = sorted_a[largest_mass][-1]
+ return a >= smallest_valid
+
+
+def rank_masking(a, threshold=None):
+ """Faster masking method. Returns a new binary mask."""
+ if threshold is None:
+ threshold = 0.11
+
+ return a > np.sort(a)[::-1][int(len(a) * threshold)]
+
+
+def _validate_clusters(clusters, min_cores):
+ """Check which clusters from the cbdbscan step are significant enough. is_valid is set accordingly."""
+ # Clusters with noisy invalid neighbors may have a harder time being marked as stable, so start with the
+ # easy ones and potentially already remove some noise by also sorting smaller clusters to the front.
+ # This clears up the clusters a bit before checking the ones with many neighbors.
+ def _cluster_sort_key(cluster):
+ return cluster.max_num_neighboring_labels, cluster.num_cores, cluster.label
+
+ sorted_clusters = sorted(clusters, key=_cluster_sort_key, reverse=False)
+
+ for cluster in sorted_clusters:
+ cluster.is_valid = None
+ if cluster.num_cores < min_cores:
+ cluster.is_valid = False
+ _remove_from_all_sets(cluster.label, sorted_clusters)
+
+ # now that invalid clusters are removed, check which clusters contain enough cores that don't belong to any
+ # other cluster.
+ for cluster in [cluster for cluster in sorted_clusters if cluster.is_valid is None]:
+ label = cluster.label
+ if _contains_isolated_cores(label, cluster, min_cores):
+ cluster.is_valid = True
+ else:
+ cluster.is_valid = False
+ _remove_from_all_sets(label, sorted_clusters)
+
+ return [cluster for cluster in sorted_clusters if cluster.is_valid]
+
+
+def _generate_topic_models_multiproc(ensemble, num_models, ensemble_workers):
+ """Generate the topic models to form the ensemble in a multiprocessed way.
+
+ Depending on the used topic model this can result in a speedup.
+
+ Parameters
+ ----------
+ ensemble: EnsembleLda
+ the ensemble
+ num_models : int
+ how many models to train in the ensemble
+ ensemble_workers : int
+ into how many processes to split the models will be set to max(workers, num_models), to avoid workers that
+ are supposed to train 0 models.
+
+ to get maximum performance, set to the number of your cores, if non-parallelized models are being used in
+ the ensemble (LdaModel).
+
+ For LdaMulticore, the performance gain is small and gets larger for a significantly smaller corpus.
+ In that case, ensemble_workers=2 can be used.
+
+ """
+ # the way random_states is handled needs to prevent getting different results when multiprocessing is on,
+ # or getting the same results in every lda children. so it is solved by generating a list of state seeds before
+ # multiprocessing is started.
+ random_states = [ensemble.random_state.randint(_MAX_RANDOM_STATE) for _ in range(num_models)]
+
+ # each worker has to work on at least one model.
+ # Don't spawn idle workers:
+ workers = min(ensemble_workers, num_models)
+
+ # create worker processes:
+ # from what I know this is basically forking with a jump to a target function in each child
+ # so modifying the ensemble object will not modify the one in the parent because of no shared memory
+ processes = []
+ pipes = []
+ num_models_unhandled = num_models # how many more models need to be trained by workers?
+
+ for i in range(workers):
+ parent_conn, child_conn = Pipe()
+ num_subprocess_models = 0
+ if i == workers - 1: # i is a index, hence -1
+ # is this the last worker that needs to be created?
+ # then task that worker with all the remaining models
+ num_subprocess_models = num_models_unhandled
+ else:
+ num_subprocess_models = int(num_models_unhandled / (workers - i))
+
+ # get the chunk from the random states that is meant to be for those models
+ random_states_for_worker = random_states[-num_models_unhandled:][:num_subprocess_models]
+
+ args = (ensemble, num_subprocess_models, random_states_for_worker, child_conn)
+ try:
+ process = Process(target=_generate_topic_models_worker, args=args)
+ processes.append(process)
+ pipes.append((parent_conn, child_conn))
+ process.start()
+ num_models_unhandled -= num_subprocess_models
+ except ProcessError:
+ logger.error(f"could not start process {i}")
+ _teardown(pipes, processes)
+ raise
+
+ # aggregate results
+ # will also block until workers are finished
+ for parent_conn, _ in pipes:
+ answer = parent_conn.recv()
+ parent_conn.close()
+ # this does basically the same as the _generate_topic_models function (concatenate all the ttdas):
+ if not ensemble.memory_friendly_ttda:
+ ensemble.tms += answer
+ ttda = np.concatenate([m.get_topics() for m in answer])
+ else:
+ ttda = answer
+ ensemble.ttda = np.concatenate([ensemble.ttda, ttda])
+
+ for process in processes:
+ process.terminate()
+
+
+def _generate_topic_models(ensemble, num_models, random_states=None):
+ """Train the topic models that form the ensemble.
+
+ Parameters
+ ----------
+ ensemble: EnsembleLda
+ the ensemble
+ num_models : int
+ number of models to be generated
+ random_states : list
+ list of numbers or np.random.RandomState objects. Will be autogenerated based on the ensembles
+ RandomState if None (default).
+ """
+ if random_states is None:
+ random_states = [ensemble.random_state.randint(_MAX_RANDOM_STATE) for _ in range(num_models)]
+
+ assert len(random_states) == num_models
+
+ kwargs = ensemble.gensim_kw_args.copy()
+
+ tm = None # remember one of the topic models from the following
+ # loop, in order to collect some properties from it afterwards.
+
+ for i in range(num_models):
+ kwargs["random_state"] = random_states[i]
+
+ tm = ensemble.get_topic_model_class()(**kwargs)
+
+ # adds the lambda (that is the unnormalized get_topics) to ttda, which is
+ # a list of all those lambdas
+ ensemble.ttda = np.concatenate([ensemble.ttda, tm.get_topics()])
+
+ # only saves the model if it is not "memory friendly"
+ if not ensemble.memory_friendly_ttda:
+ ensemble.tms += [tm]
+
+ # use one of the tms to get some info that will be needed later
+ ensemble.sstats_sum = tm.state.sstats.sum()
+ ensemble.eta = tm.eta
+
+
+def _generate_topic_models_worker(ensemble, num_models, random_states, pipe):
+ """Wrapper for _generate_topic_models to write the results into a pipe.
+
+ This is intended to be used inside a subprocess."""
+ #
+ # Same as _generate_topic_models, but runs in a separate subprocess, and
+ # sends the updated ensemble state to the parent subprocess via a pipe.
+ #
+ logger.info(f"spawned worker to generate {num_models} topic models")
+
+ _generate_topic_models(ensemble=ensemble, num_models=num_models, random_states=random_states)
+
+ # send the ttda that is in the child/workers version of the memory into the pipe
+ # available, after _generate_topic_models has been called in the worker
+ if ensemble.memory_friendly_ttda:
+ # remember that this code is inside the worker processes memory,
+ # so self.ttda is the ttda of only a chunk of models
+ pipe.send(ensemble.ttda)
+ else:
+ pipe.send(ensemble.tms)
+
+ pipe.close()
+
+
+def _calculate_asymmetric_distance_matrix_chunk(
+ ttda1,
+ ttda2,
+ start_index,
+ masking_method,
+ masking_threshold,
+):
+ """Calculate an (asymmetric) distance from each topic in ``ttda1`` to each topic in ``ttda2``.
+
+ Parameters
+ ----------
+ ttda1 and ttda2: 2D arrays of floats
+ Two ttda matrices that are going to be used for distance calculation. Each row in ttda corresponds to one
+ topic. Each cell in the resulting matrix corresponds to the distance between a topic pair.
+ start_index : int
+ this function might be used in multiprocessing, so start_index has to be set as ttda1 is a chunk of the
+ complete ttda in that case. start_index would be 0 if ``ttda1 == self.ttda``. When self.ttda is split into
+ two pieces, each 100 ttdas long, then start_index should be be 100. default is 0
+ masking_method: function
+
+ masking_threshold: float
+
+ Returns
+ -------
+ 2D numpy.ndarray of floats
+ Asymmetric distance matrix of size ``len(ttda1)`` by ``len(ttda2)``.
+
+ """
+ # initialize the distance matrix. ndarray is faster than zeros
+ distances = np.ndarray((len(ttda1), len(ttda2)))
+
+ if ttda1.shape[0] > 0 and ttda2.shape[0] > 0:
+ # the worker might not have received a ttda because it was chunked up too much
+
+ # some help to find a better threshold by useful log messages
+ avg_mask_size = 0
+
+ # now iterate over each topic
+ for ttd1_idx, ttd1 in enumerate(ttda1):
+ # create mask from ttd1 that removes noise from a and keeps the largest terms
+ mask = masking_method(ttd1, masking_threshold)
+ ttd1_masked = ttd1[mask]
+
+ avg_mask_size += mask.sum()
+
+ # now look at every possible pair for topic a:
+ for ttd2_idx, ttd2 in enumerate(ttda2):
+ # distance to itself is 0
+ if ttd1_idx + start_index == ttd2_idx:
+ distances[ttd1_idx][ttd2_idx] = 0
+ continue
+
+ # now mask b based on a, which will force the shape of a onto b
+ ttd2_masked = ttd2[mask]
+
+ # Smart distance calculation avoids calculating cosine distance for highly masked topic-term
+ # distributions that will have distance values near 1.
+ if ttd2_masked.sum() <= _COSINE_DISTANCE_CALCULATION_THRESHOLD:
+ distance = 1
+ else:
+ distance = cosine(ttd1_masked, ttd2_masked)
+
+ distances[ttd1_idx][ttd2_idx] = distance
+
+ percent = round(100 * avg_mask_size / ttda1.shape[0] / ttda1.shape[1], 1)
+ logger.info(f'the given threshold of {masking_threshold} covered on average {percent}% of tokens')
+
+ return distances
+
+
+def _asymmetric_distance_matrix_worker(
+ worker_id,
+ entire_ttda,
+ ttdas_sent,
+ n_ttdas,
+ masking_method,
+ masking_threshold,
+ pipe,
+):
+ """Worker that computes the distance to all other nodes from a chunk of nodes."""
+ logger.info(f"spawned worker {worker_id} to generate {n_ttdas} rows of the asymmetric distance matrix")
+ # the chunk of ttda that's going to be calculated:
+ ttda1 = entire_ttda[ttdas_sent:ttdas_sent + n_ttdas]
+ distance_chunk = _calculate_asymmetric_distance_matrix_chunk(
+ ttda1=ttda1,
+ ttda2=entire_ttda,
+ start_index=ttdas_sent,
+ masking_method=masking_method,
+ masking_threshold=masking_threshold,
+ )
+ pipe.send((worker_id, distance_chunk)) # remember that this code is inside the workers memory
+ pipe.close()
+
+
+def _calculate_assymetric_distance_matrix_multiproc(
+ workers,
+ entire_ttda,
+ masking_method,
+ masking_threshold,
+):
+ processes = []
+ pipes = []
+ ttdas_sent = 0
+
+ for i in range(workers):
+ try:
+ parent_conn, child_conn = Pipe()
+
+ # Load Balancing, for example if there are 9 ttdas and 4 workers, the load will be balanced 2, 2, 2, 3.
+ n_ttdas = 0
+ if i == workers - 1: # i is a index, hence -1
+ # is this the last worker that needs to be created?
+ # then task that worker with all the remaining models
+ n_ttdas = len(entire_ttda) - ttdas_sent
+ else:
+ n_ttdas = int((len(entire_ttda) - ttdas_sent) / (workers - i))
+
+ args = (i, entire_ttda, ttdas_sent, n_ttdas, masking_method, masking_threshold, child_conn)
+ process = Process(target=_asymmetric_distance_matrix_worker, args=args)
+ ttdas_sent += n_ttdas
+
+ processes.append(process)
+ pipes.append((parent_conn, child_conn))
+ process.start()
+ except ProcessError:
+ logger.error(f"could not start process {i}")
+ _teardown(pipes, processes)
+ raise
+
+ distances = []
+ # note, that the following loop maintains order in how the ttda will be concatenated
+ # which is very important. Ordering in ttda has to be the same as when using only one process
+ for parent_conn, _ in pipes:
+ worker_id, distance_chunk = parent_conn.recv()
+ parent_conn.close() # child conn will be closed from inside the worker
+ # this does basically the same as the _generate_topic_models function (concatenate all the ttdas):
+ distances.append(distance_chunk)
+
+ for process in processes:
+ process.terminate()
+
+ return np.concatenate(distances)
+
+
+class EnsembleLda(SaveLoad):
+ """Ensemble Latent Dirichlet Allocation (eLDA), a method of training a topic model ensemble.
+
+ Extracts stable topics that are consistently learned across multiple LDA models. eLDA has the added benefit that
+ the user does not need to know the exact number of topics the topic model should extract ahead of time.
+
+ """
+
+ def __init__(
+ self, topic_model_class="ldamulticore", num_models=3,
+ min_cores=None, # default value from _generate_stable_topics()
+ epsilon=0.1, ensemble_workers=1, memory_friendly_ttda=True,
+ min_samples=None, masking_method=mass_masking, masking_threshold=None,
+ distance_workers=1, random_state=None, **gensim_kw_args,
+ ):
+ """Create and train a new EnsembleLda model.
+
+ Will start training immediatelly, except if iterations, passes or num_models is 0 or if the corpus is missing.
+
+ Parameters
+ ----------
+ topic_model_class : str, topic model, optional
+ Examples:
+ * 'ldamulticore' (default, recommended)
+ * 'lda'
+ * ldamodel.LdaModel
+ * ldamulticore.LdaMulticore
+ ensemble_workers : int, optional
+ Spawns that many processes and distributes the models from the ensemble to those as evenly as possible.
+ num_models should be a multiple of ensemble_workers.
+
+ Setting it to 0 or 1 will both use the non-multiprocessing version. Default: 1
+ num_models : int, optional
+ How many LDA models to train in this ensemble.
+ Default: 3
+ min_cores : int, optional
+ Minimum cores a cluster of topics has to contain so that it is recognized as stable topic.
+ epsilon : float, optional
+ Defaults to 0.1. Epsilon for the CBDBSCAN clustering that generates the stable topics.
+ ensemble_workers : int, optional
+ Spawns that many processes and distributes the models from the ensemble to those as evenly as possible.
+ num_models should be a multiple of ensemble_workers.
+
+ Setting it to 0 or 1 will both use the nonmultiprocessing version. Default: 1
+ memory_friendly_ttda : boolean, optional
+ If True, the models in the ensemble are deleted after training and only a concatenation of each model's
+ topic term distribution (called ttda) is kept to save memory.
+
+ Defaults to True. When False, trained models are stored in a list in self.tms, and no models that are not
+ of a gensim model type can be added to this ensemble using the add_model function.
+
+ If False, any topic term matrix can be suplied to add_model.
+ min_samples : int, optional
+ Required int of nearby topics for a topic to be considered as 'core' in the CBDBSCAN clustering.
+ masking_method : function, optional
+ Choose one of :meth:`~gensim.models.ensemblelda.mass_masking` (default) or
+ :meth:`~gensim.models.ensemblelda.rank_masking` (percentile, faster).
+
+ For clustering, distances between topic-term distributions are asymmetric. In particular, the distance
+ (technically a divergence) from distribution A to B is more of a measure of if A is contained in B. At a
+ high level, this involves using distribution A to mask distribution B and then calculating the cosine
+ distance between the two. The masking can be done in two ways:
+
+ 1. mass: forms mask by taking the top ranked terms until their cumulative mass reaches the
+ 'masking_threshold'
+
+ 2. rank: forms mask by taking the top ranked terms (by mass) until the 'masking_threshold' is reached.
+ For example, a ranking threshold of 0.11 means the top 0.11 terms by weight are used to form a mask.
+ masking_threshold : float, optional
+ Default: None, which uses ``0.95`` for "mass", and ``0.11`` for masking_method "rank". In general, too
+ small a mask threshold leads to inaccurate calculations (no signal) and too big a mask leads to noisy
+ distance calculations. Defaults are often a good sweet spot for this hyperparameter.
+ distance_workers : int, optional
+ When ``distance_workers`` is ``None``, it defaults to ``os.cpu_count()`` for maximum performance. Default is
+ 1, which is not multiprocessed. Set to ``> 1`` to enable multiprocessing.
+ **gensim_kw_args
+ Parameters for each gensim model (e.g. :py:class:`gensim.models.LdaModel`) in the ensemble.
+
+ """
+
+ if "id2word" not in gensim_kw_args:
+ gensim_kw_args["id2word"] = None
+ if "corpus" not in gensim_kw_args:
+ gensim_kw_args["corpus"] = None
+
+ if gensim_kw_args["id2word"] is None and not gensim_kw_args["corpus"] is None:
+ logger.warning("no word id mapping provided; initializing from corpus, assuming identity")
+ gensim_kw_args["id2word"] = utils.dict_from_corpus(gensim_kw_args["corpus"])
+ if gensim_kw_args["id2word"] is None and gensim_kw_args["corpus"] is None:
+ raise ValueError(
+ "at least one of corpus/id2word must be specified, to establish "
+ "input space dimensionality. Corpus should be provided using the "
+ "`corpus` keyword argument."
+ )
+
+ if type(topic_model_class) == type and issubclass(topic_model_class, ldamodel.LdaModel):
+ self.topic_model_class = topic_model_class
+ else:
+ kinds = {
+ "lda": ldamodel.LdaModel,
+ "ldamulticore": ldamulticore.LdaMulticore
+ }
+ if topic_model_class not in kinds:
+ raise ValueError(
+ "topic_model_class should be one of 'lda', 'ldamulticode' or a model "
+ "inheriting from LdaModel"
+ )
+ self.topic_model_class = kinds[topic_model_class]
+
+ self.num_models = num_models
+ self.gensim_kw_args = gensim_kw_args
+
+ self.memory_friendly_ttda = memory_friendly_ttda
+
+ self.distance_workers = distance_workers
+ self.masking_threshold = masking_threshold
+ self.masking_method = masking_method
+
+ # this will provide the gensim api to the ensemble basically
+ self.classic_model_representation = None
+
+ # the ensembles state
+ self.random_state = utils.get_random_state(random_state)
+ self.sstats_sum = 0
+ self.eta = None
+ self.tms = []
+ # initialize empty 2D topic term distribution array (ttda) (number of topics x number of terms)
+ self.ttda = np.empty((0, len(gensim_kw_args["id2word"])))
+ self.asymmetric_distance_matrix_outdated = True
+
+ # in case the model will not train due to some
+ # parameters, stop here and don't train.
+ if num_models <= 0:
+ return
+ if gensim_kw_args.get("corpus") is None:
+ return
+ if "iterations" in gensim_kw_args and gensim_kw_args["iterations"] <= 0:
+ return
+ if "passes" in gensim_kw_args and gensim_kw_args["passes"] <= 0:
+ return
+
+ logger.info(f"generating {num_models} topic models using {ensemble_workers} workers")
+
+ if ensemble_workers > 1:
+ _generate_topic_models_multiproc(self, num_models, ensemble_workers)
+ else:
+ _generate_topic_models(self, num_models)
+
+ self._generate_asymmetric_distance_matrix()
+ self._generate_topic_clusters(epsilon, min_samples)
+ self._generate_stable_topics(min_cores)
+
+ # create model that can provide the usual gensim api to the stable topics from the ensemble
+ self.generate_gensim_representation()
+
+ def get_topic_model_class(self):
+ """Get the class that is used for :meth:`gensim.models.EnsembleLda.generate_gensim_representation`."""
+ if self.topic_model_class is None:
+ instruction = (
+ 'Try setting topic_model_class manually to what the individual models were based on, '
+ 'e.g. LdaMulticore.'
+ )
+ try:
+ module = importlib.import_module(self.topic_model_module_string)
+ self.topic_model_class = getattr(module, self.topic_model_class_string)
+ del self.topic_model_module_string
+ del self.topic_model_class_string
+ except ModuleNotFoundError:
+ logger.error(
+ f'Could not import the "{self.topic_model_class_string}" module in order to provide the '
+ f'"{self.topic_model_class_string}" class as "topic_model_class" attribute. {instruction}'
+ )
+ except AttributeError:
+ logger.error(
+ f'Could not import the "{self.topic_model_class_string}" class from the '
+ f'"{self.topic_model_module_string}" module in order to set the "topic_model_class" attribute. '
+ f'{instruction}'
+ )
+ return self.topic_model_class
+
+ def save(self, *args, **kwargs):
+ if self.get_topic_model_class() is not None:
+ self.topic_model_module_string = self.topic_model_class.__module__
+ self.topic_model_class_string = self.topic_model_class.__name__
+ kwargs['ignore'] = frozenset(kwargs.get('ignore', ())).union(('topic_model_class', ))
+ super(EnsembleLda, self).save(*args, **kwargs)
+
+ save.__doc__ = SaveLoad.save.__doc__
+
+ def convert_to_memory_friendly(self):
+ """Remove the stored gensim models and only keep their ttdas.
+
+ This frees up memory, but you won't have access to the individual models anymore if you intended to use them
+ outside of the ensemble.
+ """
+ self.tms = []
+ self.memory_friendly_ttda = True
+
+ def generate_gensim_representation(self):
+ """Create a gensim model from the stable topics.
+
+ The returned representation is an Gensim LdaModel (:py:class:`gensim.models.LdaModel`) that has been
+ instantiated with an A-priori belief on word probability, eta, that represents the topic-term distributions of
+ any stable topics the were found by clustering over the ensemble of topic distributions.
+
+ When no stable topics have been detected, None is returned.
+
+ Returns
+ -------
+ :py:class:`gensim.models.LdaModel`
+ A Gensim LDA Model classic_model_representation for which:
+ ``classic_model_representation.get_topics() == self.get_topics()``
+
+ """
+ logger.info("generating classic gensim model representation based on results from the ensemble")
+
+ sstats_sum = self.sstats_sum
+ # if sstats_sum (which is the number of words actually) should be wrong for some fantastic funny reason
+ # that makes you want to peel your skin off, recreate it (takes a while):
+ if sstats_sum == 0 and "corpus" in self.gensim_kw_args and not self.gensim_kw_args["corpus"] is None:
+ for document in self.gensim_kw_args["corpus"]:
+ for token in document:
+ sstats_sum += token[1]
+ self.sstats_sum = sstats_sum
+
+ stable_topics = self.get_topics()
+
+ num_stable_topics = len(stable_topics)
+
+ if num_stable_topics == 0:
+ logger.error(
+ "the model did not detect any stable topic. You can try to adjust epsilon: "
+ "recluster(eps=...)"
+ )
+ self.classic_model_representation = None
+ return
+
+ # create a new gensim model
+ params = self.gensim_kw_args.copy()
+ params["eta"] = self.eta
+ params["num_topics"] = num_stable_topics
+ # adjust params in a way that no training happens
+ params["passes"] = 0 # no training
+ # iterations is needed for inference, pass it to the model
+
+ classic_model_representation = self.get_topic_model_class()(**params)
+
+ # when eta was None, use what gensim generates as default eta for the following tasks:
+ eta = classic_model_representation.eta
+ if sstats_sum == 0:
+ sstats_sum = classic_model_representation.state.sstats.sum()
+ self.sstats_sum = sstats_sum
+
+ # the following is important for the denormalization
+ # to generate the proper sstats for the new gensim model:
+ # transform to dimensionality of stable_topics. axis=1 is summed
+ eta_sum = 0
+ if isinstance(eta, (int, float)):
+ eta_sum = [eta * len(stable_topics[0])] * num_stable_topics
+ else:
+ if len(eta.shape) == 1: # [e1, e2, e3]
+ eta_sum = [[eta.sum()]] * num_stable_topics
+ if len(eta.shape) > 1: # [[e11, e12, ...], [e21, e22, ...], ...]
+ eta_sum = np.array(eta.sum(axis=1)[:, None])
+
+ # the factor, that will be used when get_topics() is used, for normalization
+ # will never change, because the sum for eta as well as the sum for sstats is constant.
+ # Therefore predicting normalization_factor becomes super easy.
+ # corpus is a mapping of id to occurrences
+
+ # so one can also easily calculate the
+ # right sstats, so that get_topics() will return the stable topics no
+ # matter eta.
+
+ normalization_factor = np.array([[sstats_sum / num_stable_topics]] * num_stable_topics) + eta_sum
+
+ sstats = stable_topics * normalization_factor
+ sstats -= eta
+
+ classic_model_representation.state.sstats = sstats.astype(np.float32)
+ # fix expElogbeta.
+ classic_model_representation.sync_state()
+
+ self.classic_model_representation = classic_model_representation
+
+ return classic_model_representation
+
+ def add_model(self, target, num_new_models=None):
+ """Add the topic term distribution array (ttda) of another model to the ensemble.
+
+ This way, multiple topic models can be connected to an ensemble manually. Make sure that all the models use
+ the exact same dictionary/idword mapping.
+
+ In order to generate new stable topics afterwards, use:
+ 2. ``self.``:meth:`~gensim.models.ensemblelda.EnsembleLda.recluster`
+
+ The ttda of another ensemble can also be used, in that case set ``num_new_models`` to the ``num_models``
+ parameter of the ensemble, that means the number of classic models in the ensemble that generated the ttda.
+ This is important, because that information is used to estimate "min_samples" for _generate_topic_clusters.
+
+ If you trained this ensemble in the past with a certain Dictionary that you want to reuse for other
+ models, you can get it from: ``self.id2word``.
+
+ Parameters
+ ----------
+ target : {see description}
+ 1. A single EnsembleLda object
+ 2. List of EnsembleLda objects
+ 3. A single Gensim topic model (e.g. (:py:class:`gensim.models.LdaModel`)
+ 4. List of Gensim topic models
+
+ if memory_friendly_ttda is True, target can also be:
+ 5. topic-term-distribution-array
+
+ example: [[0.1, 0.1, 0.8], [...], ...]
+
+ [topic1, topic2, ...]
+ with topic being an array of probabilities:
+ [token1, token2, ...]
+
+ token probabilities in a single topic sum to one, therefore, all the words sum to len(ttda)
+
+ num_new_models : integer, optional
+ the model keeps track of how many models were used in this ensemble. Set higher if ttda contained topics
+ from more than one model. Default: None, which takes care of it automatically.
+
+ If target is a 2D-array of float values, it assumes 1.
+
+ If the ensemble has ``memory_friendly_ttda`` set to False, then it will always use the number of models in
+ the target parameter.
+
+ """
+ # If the model has never seen a ttda before, initialize.
+ # If it has, append.
+
+ # Be flexible. Can be a single element or a list of elements
+ # make sure it is a numpy array
+ if not isinstance(target, (np.ndarray, list)):
+ target = np.array([target])
+ else:
+ target = np.array(target)
+ assert len(target) > 0
+
+ if self.memory_friendly_ttda:
+ # for memory friendly models/ttdas, append the ttdas to itself
+
+ detected_num_models = 0
+ ttda = []
+
+ # 1. ttda array, because that's the only accepted input that contains numbers
+ if isinstance(target.dtype.type(), (np.number, float)):
+ ttda = target
+ detected_num_models = 1
+
+ # 2. list of ensemblelda objects
+ elif isinstance(target[0], type(self)):
+ ttda = np.concatenate([ensemble.ttda for ensemble in target], axis=0)
+ detected_num_models = sum([ensemble.num_models for ensemble in target])
+
+ # 3. list of gensim models
+ elif isinstance(target[0], basemodel.BaseTopicModel):
+ ttda = np.concatenate([model.get_topics() for model in target], axis=0)
+ detected_num_models = len(target)
+
+ # unknown
+ else:
+ raise ValueError(f"target is of unknown type or a list of unknown types: {type(target[0])}")
+
+ # new models were added, increase num_models
+ # if the user didn't provide a custon numer to use
+ if num_new_models is None:
+ self.num_models += detected_num_models
+ else:
+ self.num_models += num_new_models
+
+ else: # memory unfriendly ensembles
+ ttda = []
+
+ # 1. ttda array
+ if isinstance(target.dtype.type(), (np.number, float)):
+ raise ValueError(
+ 'ttda arrays cannot be added to ensembles, for which memory_friendly_ttda=False, '
+ 'you can call convert_to_memory_friendly, but it will discard the stored gensim '
+ 'models and only keep the relevant topic term distributions from them.'
+ )
+
+ # 2. list of ensembles
+ elif isinstance(target[0], type(self)):
+ for ensemble in target:
+ self.tms += ensemble.tms
+ ttda = np.concatenate([ensemble.ttda for ensemble in target], axis=0)
+
+ # 3. list of gensim models
+ elif isinstance(target[0], basemodel.BaseTopicModel):
+ self.tms += target.tolist()
+ ttda = np.concatenate([model.get_topics() for model in target], axis=0)
+
+ # unknown
+ else:
+ raise ValueError(f"target is of unknown type or a list of unknown types: {type(target[0])}")
+
+ # in this case, len(self.tms) should
+ # always match self.num_models
+ if num_new_models is not None and num_new_models + self.num_models != len(self.tms):
+ logger.info(
+ 'num_new_models will be ignored. num_models should match the number of '
+ 'stored models for a memory unfriendly ensemble'
+ )
+ self.num_models = len(self.tms)
+
+ logger.info(f"ensemble contains {self.num_models} models and {len(self.ttda)} topics now")
+
+ if self.ttda.shape[1] != ttda.shape[1]:
+ raise ValueError(
+ f"target ttda dimensions do not match. Topics must be {self.ttda.shape[-1]} but was {ttda.shape[-1]} "
+ f"elements large"
+ )
+
+ self.ttda = np.append(self.ttda, ttda, axis=0)
+
+ # tell recluster that the distance matrix needs to be regenerated
+ self.asymmetric_distance_matrix_outdated = True
+
+ def _generate_asymmetric_distance_matrix(self):
+ """Calculate the pairwise distance matrix for all the ttdas from the ensemble.
+
+ Returns the asymmetric pairwise distance matrix that is used in the DBSCAN clustering.
+
+ Afterwards, the model needs to be reclustered for this generated matrix to take effect.
+
+ """
+ workers = self.distance_workers
+
+ # matrix is up to date afterwards
+ self.asymmetric_distance_matrix_outdated = False
+
+ logger.info(f"generating a {len(self.ttda)} x {len(self.ttda)} asymmetric distance matrix...")
+
+ if workers is not None and workers <= 1:
+ self.asymmetric_distance_matrix = _calculate_asymmetric_distance_matrix_chunk(
+ ttda1=self.ttda,
+ ttda2=self.ttda,
+ start_index=0,
+ masking_method=self.masking_method,
+ masking_threshold=self.masking_threshold,
+ )
+ else:
+ # best performance on 2-core machine: 2 workers
+ if workers is None:
+ workers = os.cpu_count()
+
+ self.asymmetric_distance_matrix = _calculate_assymetric_distance_matrix_multiproc(
+ workers=workers,
+ entire_ttda=self.ttda,
+ masking_method=self.masking_method,
+ masking_threshold=self.masking_threshold,
+ )
+
+ def _generate_topic_clusters(self, eps=0.1, min_samples=None):
+ """Run the CBDBSCAN algorithm on all the detected topics and label them with label-indices.
+
+ The final approval and generation of stable topics is done in ``_generate_stable_topics()``.
+
+ Parameters
+ ----------
+ eps : float
+ dbscan distance scale
+ min_samples : int, optional
+ defaults to ``int(self.num_models / 2)``, dbscan min neighbours threshold required to consider
+ a topic to be a core. Should scale with the number of models, ``self.num_models``
+
+ """
+ if min_samples is None:
+ min_samples = int(self.num_models / 2)
+ logger.info("fitting the clustering model, using %s for min_samples", min_samples)
+ else:
+ logger.info("fitting the clustering model")
+
+ self.cluster_model = CBDBSCAN(eps=eps, min_samples=min_samples)
+ self.cluster_model.fit(self.asymmetric_distance_matrix)
+
+ def _generate_stable_topics(self, min_cores=None):
+ """Generate stable topics out of the clusters.
+
+ The function finds clusters of topics using a variant of DBScan. If a cluster has enough core topics
+ (c.f. parameter ``min_cores``), then this cluster represents a stable topic. The stable topic is specifically
+ calculated as the average over all topic-term distributions of the core topics in the cluster.
+
+ This function is the last step that has to be done in the ensemble. After this step is complete,
+ Stable topics can be retrieved afterwards using the :meth:`~gensim.models.ensemblelda.EnsembleLda.get_topics`
+ method.
+
+ Parameters
+ ----------
+ min_cores : int
+ Minimum number of core topics needed to form a cluster that represents a stable topic.
+ Using ``None`` defaults to ``min_cores = min(3, max(1, int(self.num_models /4 +1)))``
+
+ """
+ # min_cores being 0 makes no sense. there has to be a core for a cluster
+ # or there is no cluster
+ if min_cores == 0:
+ min_cores = 1
+
+ if min_cores is None:
+ # min_cores is a number between 1 and 3, depending on the number of models
+ min_cores = min(3, max(1, int(self.num_models / 4 + 1)))
+ logger.info("generating stable topics, using %s for min_cores", min_cores)
+ else:
+ logger.info("generating stable topics")
+
+ cbdbscan_topics = self.cluster_model.results
+
+ grouped_by_labels = _group_by_labels(cbdbscan_topics)
+ clusters = _aggregate_topics(grouped_by_labels)
+ valid_clusters = _validate_clusters(clusters, min_cores)
+ valid_cluster_labels = {cluster.label for cluster in valid_clusters}
+
+ for topic in cbdbscan_topics:
+ topic.valid_neighboring_labels = {
+ label for label in topic.neighboring_labels
+ if label in valid_cluster_labels
+ }
+
+ # keeping only VALID cores
+ valid_core_mask = np.vectorize(_is_valid_core)(cbdbscan_topics)
+ valid_topics = self.ttda[valid_core_mask]
+ topic_labels = np.array([topic.label for topic in cbdbscan_topics])[valid_core_mask]
+ unique_labels = np.unique(topic_labels)
+
+ num_stable_topics = len(unique_labels)
+ stable_topics = np.empty((num_stable_topics, len(self.id2word)))
+
+ # for each cluster
+ for label_index, label in enumerate(unique_labels):
+ # mean of all the topics that are of that cluster
+ topics_of_cluster = np.array([topic for t, topic in enumerate(valid_topics) if topic_labels[t] == label])
+ stable_topics[label_index] = topics_of_cluster.mean(axis=0)
+
+ self.valid_clusters = valid_clusters
+ self.stable_topics = stable_topics
+
+ logger.info("found %s stable topics", len(stable_topics))
+
+ def recluster(self, eps=0.1, min_samples=None, min_cores=None):
+ """Reapply CBDBSCAN clustering and stable topic generation.
+
+ Stable topics can be retrieved using :meth:`~gensim.models.ensemblelda.EnsembleLda.get_topics`.
+
+ Parameters
+ ----------
+ eps : float
+ epsilon for the CBDBSCAN algorithm, having the same meaning as in classic DBSCAN clustering.
+ default: ``0.1``
+ min_samples : int
+ The minimum number of samples in the neighborhood of a topic to be considered a core in CBDBSCAN.
+ default: ``int(self.num_models / 2)``
+ min_cores : int
+ how many cores a cluster has to have, to be treated as stable topic. That means, how many topics
+ that look similar have to be present, so that the average topic in those is used as stable topic.
+ default: ``min(3, max(1, int(self.num_models /4 +1)))``
+
+ """
+ # if new models were added to the ensemble, the distance matrix needs to be generated again
+ if self.asymmetric_distance_matrix_outdated:
+ logger.info("asymmetric distance matrix is outdated due to add_model")
+ self._generate_asymmetric_distance_matrix()
+
+ # Run CBDBSCAN to get topic clusters:
+ self._generate_topic_clusters(eps, min_samples)
+
+ # Interpret the results of CBDBSCAN to identify stable topics:
+ self._generate_stable_topics(min_cores)
+
+ # Create gensim LdaModel representation of topic model with stable topics (can be used for inference):
+ self.generate_gensim_representation()
+
+ # GENSIM API
+ # to make using the ensemble in place of a gensim model as easy as possible
+
+ def get_topics(self):
+ """Return only the stable topics from the ensemble.
+
+ Returns
+ -------
+ 2D Numpy.numpy.ndarray of floats
+ List of stable topic term distributions
+
+ """
+ return self.stable_topics
+
+ def _ensure_gensim_representation(self):
+ """Check if stable topics and the internal gensim representation exist. Raise an error if not."""
+ if self.classic_model_representation is None:
+ if len(self.stable_topics) == 0:
+ raise ValueError("no stable topic was detected")
+ else:
+ raise ValueError("use generate_gensim_representation() first")
+
+ def __getitem__(self, i):
+ """See :meth:`gensim.models.LdaModel.__getitem__`."""
+ self._ensure_gensim_representation()
+ return self.classic_model_representation[i]
+
+ def inference(self, *posargs, **kwargs):
+ """See :meth:`gensim.models.LdaModel.inference`."""
+ self._ensure_gensim_representation()
+ return self.classic_model_representation.inference(*posargs, **kwargs)
+
+ def log_perplexity(self, *posargs, **kwargs):
+ """See :meth:`gensim.models.LdaModel.log_perplexity`."""
+ self._ensure_gensim_representation()
+ return self.classic_model_representation.log_perplexity(*posargs, **kwargs)
+
+ def print_topics(self, *posargs, **kwargs):
+ """See :meth:`gensim.models.LdaModel.print_topics`."""
+ self._ensure_gensim_representation()
+ return self.classic_model_representation.print_topics(*posargs, **kwargs)
+
+ @property
+ def id2word(self):
+ """Return the :py:class:`gensim.corpora.dictionary.Dictionary` object used in the model."""
+ return self.gensim_kw_args["id2word"]
+
+
+class CBDBSCAN:
+ """A Variation of the DBSCAN algorithm called Checkback DBSCAN (CBDBSCAN).
+
+ The algorithm works based on DBSCAN-like parameters 'eps' and 'min_samples' that respectively define how far a
+ "nearby" point is, and the minimum number of nearby points needed to label a candidate datapoint a core of a
+ cluster. (See https://scikit-learn.org/stable/modules/generated/sklearn.cluster.DBSCAN.html).
+
+ The algorithm works as follows:
+
+ 1. (A)symmetric distance matrix provided at fit-time (called 'amatrix').
+ For the sake of example below, assume the there are only five topics (amatrix contains distances with dim 5x5),
+ T_1, T_2, T_3, T_4, T_5:
+ 2. Start by scanning a candidate topic with respect to a parent topic
+ (e.g. T_1 with respect to parent None)
+ 3. Check which topics are nearby the candidate topic using 'self.eps' as a threshold and call them neighbours
+ (e.g. assume T_3, T_4, and T_5 are nearby and become neighbours)
+ 4. If there are more neighbours than 'self.min_samples', the candidate topic becomes a core candidate for a cluster
+ (e.g. if 'min_samples'=1, then T_1 becomes the first core of a cluster)
+ 5. If candidate is a core, CheckBack (CB) to find the fraction of neighbours that are either the parent or the
+ parent's neighbours. If this fraction is more than 75%, give the candidate the same label as its parent.
+ (e.g. in the trivial case there is no parent (or neighbours of that parent), a new incremental label is given)
+ 6. If candidate is a core, recursively scan the next nearby topic (e.g. scan T_3) labeling the previous topic as
+ the parent and the previous neighbours as the parent_neighbours - repeat steps 2-6:
+
+ 2. (e.g. Scan candidate T_3 with respect to parent T_1 that has parent_neighbours T_3, T_4, and T_5)
+ 3. (e.g. T5 is the only neighbour)
+ 4. (e.g. number of neighbours is 1, therefore candidate T_3 becomes a core)
+ 5. (e.g. CheckBack finds that two of the four parent and parent neighbours are neighbours of candidate T_3.
+ Therefore the candidate T_3 does NOT get the same label as its parent T_1)
+ 6. (e.g. Scan candidate T_5 with respect to parent T_3 that has parent_neighbours T_5)
+
+ The CB step has the effect that it enforces cluster compactness and allows the model to avoid creating clusters for
+ unstable topics made of a composition of multiple stable topics.
+
+ """
+
+ def __init__(self, eps, min_samples):
+ """Create a new CBDBSCAN object. Call fit in order to train it on an asymmetric distance matrix.
+
+ Parameters
+ ----------
+ eps : float
+ epsilon for the CBDBSCAN algorithm, having the same meaning as in classic DBSCAN clustering.
+ min_samples : int
+ The minimum number of samples in the neighborhood of a topic to be considered a core in CBDBSCAN.
+
+ """
+ self.eps = eps
+ self.min_samples = min_samples
+
+ def fit(self, amatrix):
+ """Apply the algorithm to an asymmetric distance matrix."""
+ self.next_label = 0
+
+ topic_clustering_results = [
+ Topic(
+ is_core=False,
+ neighboring_labels=set(),
+ neighboring_topic_indices=set(),
+ label=None,
+ num_neighboring_labels=0,
+ valid_neighboring_labels=set()
+ ) for i in range(len(amatrix))
+ ]
+
+ amatrix_copy = amatrix.copy()
+
+ # to avoid the problem of comparing the topic with itself
+ np.fill_diagonal(amatrix_copy, 1)
+
+ min_distance_per_topic = [(distance, index) for index, distance in enumerate(amatrix_copy.min(axis=1))]
+ min_distance_per_topic_sorted = sorted(min_distance_per_topic, key=lambda distance: distance[0])
+ ordered_min_similarity = [index for distance, index in min_distance_per_topic_sorted]
+
+ def scan_topic(topic_index, current_label=None, parent_neighbors=None):
+ """Extend the cluster in one direction.
+
+ Results are accumulated to ``self.results``.
+
+ Parameters
+ ----------
+ topic_index : int
+ The topic that might be added to the existing cluster, or which might create a new cluster if necessary.
+ current_label : int
+ The label of the cluster that might be suitable for ``topic_index``
+
+ """
+ neighbors_sorted = sorted(
+ [
+ (distance, index)
+ for index, distance in enumerate(amatrix_copy[topic_index])
+ ],
+ key=lambda x: x[0],
+ )
+ neighboring_topic_indices = [index for distance, index in neighbors_sorted if distance < self.eps]
+
+ num_neighboring_topics = len(neighboring_topic_indices)
+
+ # If the number of neighbor indices of a topic is large enough, it is considered a core.
+ # This also takes neighbor indices that already are identified as core in count.
+ if num_neighboring_topics >= self.min_samples:
+ # This topic is a core!
+ topic_clustering_results[topic_index].is_core = True
+
+ # if current_label is none, then this is the first core
+ # of a new cluster (hence next_label is used)
+ if current_label is None:
+ # next_label is initialized with 0 in fit() for the first cluster
+ current_label = self.next_label
+ self.next_label += 1
+
+ else:
+ # In case the core has a parent, check the distance to the parents neighbors (since the matrix is
+ # asymmetric, it takes return distances into account here)
+ # If less than 25% of the elements are close enough, then create a new cluster rather than further
+ # growing the current cluster in that direction.
+ close_parent_neighbors_mask = amatrix_copy[topic_index][parent_neighbors] < self.eps
+
+ if close_parent_neighbors_mask.mean() < 0.25:
+ # start new cluster by changing current_label
+ current_label = self.next_label
+ self.next_label += 1
+
+ topic_clustering_results[topic_index].label = current_label
+
+ for neighboring_topic_index in neighboring_topic_indices:
+ if topic_clustering_results[neighboring_topic_index].label is None:
+ ordered_min_similarity.remove(neighboring_topic_index)
+ # try to extend the cluster into the direction of the neighbor
+ scan_topic(neighboring_topic_index, current_label, neighboring_topic_indices + [topic_index])
+
+ topic_clustering_results[neighboring_topic_index].neighboring_topic_indices.add(topic_index)
+ topic_clustering_results[neighboring_topic_index].neighboring_labels.add(current_label)
+
+ else:
+ # this topic is not a core!
+ if current_label is None:
+ topic_clustering_results[topic_index].label = -1
+ else:
+ topic_clustering_results[topic_index].label = current_label
+
+ # elements are going to be removed from that array in scan_topic, do until it is empty
+ while len(ordered_min_similarity) != 0:
+ next_topic_index = ordered_min_similarity.pop(0)
+ scan_topic(next_topic_index)
+
+ self.results = topic_clustering_results
diff --git a/gensim/models/fasttext.py b/gensim/models/fasttext.py
index 2c2750d70c..a94bc17f27 100644
--- a/gensim/models/fasttext.py
+++ b/gensim/models/fasttext.py
@@ -276,7 +276,7 @@ def __init__(self, sentences=None, corpus_file=None, sg=0, hs=0, vector_size=100
max_vocab_size=None, word_ngrams=1, sample=1e-3, seed=1, workers=3, min_alpha=0.0001,
negative=5, ns_exponent=0.75, cbow_mean=1, hashfxn=hash, epochs=5, null_word=0, min_n=3, max_n=6,
sorted_vocab=1, bucket=2000000, trim_rule=None, batch_words=MAX_WORDS_IN_BATCH, callbacks=(),
- max_final_vocab=None):
+ max_final_vocab=None, shrink_windows=True,):
"""Train, use and evaluate word representations learned using the method
described in `Enriching Word Vectors with Subword Information `_,
aka FastText.
@@ -301,7 +301,7 @@ def __init__(self, sentences=None, corpus_file=None, sg=0, hs=0, vector_size=100
uninitialized).
min_count : int, optional
The model ignores all words with total frequency lower than this.
- size : int, optional
+ vector_size : int, optional
Dimensionality of the word vectors.
window : int, optional
The maximum distance between the current and predicted word within a sentence.
@@ -385,6 +385,12 @@ def __init__(self, sentences=None, corpus_file=None, sg=0, hs=0, vector_size=100
``min_count```. If the specified ``min_count`` is more than the
automatically calculated ``min_count``, the former will be used.
Set to ``None`` if not required.
+ shrink_windows : bool, optional
+ New in 4.1. Experimental.
+ If True, the effective window size is uniformly sampled from [1, `window`]
+ for each target word during training, to match the original word2vec algorithm's
+ approximate weighting of context words by distance. Otherwise, the effective
+ window size is always fixed to `window` words to either side.
Examples
--------
@@ -432,7 +438,8 @@ def __init__(self, sentences=None, corpus_file=None, sg=0, hs=0, vector_size=100
max_vocab_size=max_vocab_size, max_final_vocab=max_final_vocab,
min_count=min_count, sample=sample, sorted_vocab=sorted_vocab,
null_word=null_word, ns_exponent=ns_exponent, hashfxn=hashfxn,
- seed=seed, hs=hs, negative=negative, cbow_mean=cbow_mean, min_alpha=min_alpha)
+ seed=seed, hs=hs, negative=negative, cbow_mean=cbow_mean,
+ min_alpha=min_alpha, shrink_windows=shrink_windows)
def _init_post_load(self, hidden_output):
num_vectors = len(self.wv.vectors)
@@ -489,16 +496,20 @@ def estimate_memory(self, vocab_size=None, report=None):
)
return report
- def _do_train_epoch(self, corpus_file, thread_id, offset, cython_vocab, thread_private_mem, cur_epoch,
- total_examples=None, total_words=None, **kwargs):
+ def _do_train_epoch(
+ self, corpus_file, thread_id, offset, cython_vocab, thread_private_mem, cur_epoch,
+ total_examples=None, total_words=None, **kwargs,
+ ):
work, neu1 = thread_private_mem
if self.sg:
- examples, tally, raw_tally = train_epoch_sg(self, corpus_file, offset, cython_vocab, cur_epoch,
- total_examples, total_words, work, neu1)
+ examples, tally, raw_tally = train_epoch_sg(
+ self, corpus_file, offset, cython_vocab, cur_epoch, total_examples, total_words, work, neu1,
+ )
else:
- examples, tally, raw_tally = train_epoch_cbow(self, corpus_file, offset, cython_vocab, cur_epoch,
- total_examples, total_words, work, neu1)
+ examples, tally, raw_tally = train_epoch_cbow(
+ self, corpus_file, offset, cython_vocab, cur_epoch, total_examples, total_words, work, neu1,
+ )
return examples, tally, raw_tally
diff --git a/gensim/models/fasttext_corpusfile.pyx b/gensim/models/fasttext_corpusfile.pyx
index e5ec611aa0..5d275b42b6 100644
--- a/gensim/models/fasttext_corpusfile.pyx
+++ b/gensim/models/fasttext_corpusfile.pyx
@@ -46,7 +46,8 @@ cdef void prepare_c_structures_for_batch(
vector[vector[string]] &sentences, int sample, int hs, int window, long long *total_words,
int *effective_words, int *effective_sentences, unsigned long long *next_random, cvocab_t *vocab,
int *sentence_idx, np.uint32_t *indexes, int *codelens, np.uint8_t **codes, np.uint32_t **points,
- np.uint32_t *reduced_windows, int *subwords_idx_len, np.uint32_t **subwords_idx) nogil:
+ np.uint32_t *reduced_windows, int *subwords_idx_len, np.uint32_t **subwords_idx, int shrink_windows,
+ ) nogil:
cdef VocabItem word
cdef string token
cdef vector[string] sent
@@ -88,8 +89,12 @@ cdef void prepare_c_structures_for_batch(
break
# precompute "reduced window" offsets in a single randint() call
- for i in range(effective_words[0]):
- reduced_windows[i] = random_int32(next_random) % window
+ if shrink_windows:
+ for i in range(effective_words[0]):
+ reduced_windows[i] = random_int32(next_random) % window
+ else:
+ for i in range(effective_words[0]):
+ reduced_windows[i] = 0
def train_epoch_sg(
@@ -136,6 +141,7 @@ def train_epoch_sg(
cdef long long total_sentences = 0
cdef long long total_effective_words = 0, total_words = 0
cdef int sent_idx, idx_start, idx_end
+ cdef int shrink_windows = int(model.shrink_windows)
init_ft_config(&c, model, _alpha, _work, _l1)
@@ -153,7 +159,7 @@ def train_epoch_sg(
prepare_c_structures_for_batch(
sentences, c.sample, c.hs, c.window, &total_words, &effective_words, &effective_sentences,
&c.next_random, vocab.get_vocab_ptr(), c.sentence_idx, c.indexes, c.codelens,
- c.codes, c.points, c.reduced_windows, c.subwords_idx_len, c.subwords_idx)
+ c.codes, c.points, c.reduced_windows, c.subwords_idx_len, c.subwords_idx, shrink_windows)
for sent_idx in range(effective_sentences):
idx_start = c.sentence_idx[sent_idx]
@@ -226,6 +232,7 @@ def train_epoch_cbow(model, corpus_file, offset, _cython_vocab, _cur_epoch, _exp
cdef long long total_sentences = 0
cdef long long total_effective_words = 0, total_words = 0
cdef int sent_idx, idx_start, idx_end
+ cdef int shrink_windows = int(model.shrink_windows)
init_ft_config(&c, model, _alpha, _work, _neu1)
@@ -243,7 +250,7 @@ def train_epoch_cbow(model, corpus_file, offset, _cython_vocab, _cur_epoch, _exp
prepare_c_structures_for_batch(
sentences, c.sample, c.hs, c.window, &total_words, &effective_words, &effective_sentences,
&c.next_random, vocab.get_vocab_ptr(), c.sentence_idx, c.indexes, c.codelens,
- c.codes, c.points, c.reduced_windows, c.subwords_idx_len, c.subwords_idx)
+ c.codes, c.points, c.reduced_windows, c.subwords_idx_len, c.subwords_idx, shrink_windows)
for sent_idx in range(effective_sentences):
idx_start = c.sentence_idx[sent_idx]
diff --git a/gensim/models/fasttext_inner.pyx b/gensim/models/fasttext_inner.pyx
index e71ed6f31d..e27bd62feb 100644
--- a/gensim/models/fasttext_inner.pyx
+++ b/gensim/models/fasttext_inner.pyx
@@ -601,8 +601,12 @@ def train_batch_any(model, sentences, alpha, _work, _neu1):
num_words, num_sentences = populate_ft_config(&c, model.wv, model.wv.buckets_word, sentences)
# precompute "reduced window" offsets in a single randint() call
- for i, randint in enumerate(model.random.randint(0, c.window, num_words)):
- c.reduced_windows[i] = randint
+ if model.shrink_windows:
+ for i, randint in enumerate(model.random.randint(0, c.window, num_words)):
+ c.reduced_windows[i] = randint
+ else:
+ for i in range(num_words):
+ c.reduced_windows[i] = 0
# release GIL & train on all sentences in the batch
with nogil:
diff --git a/gensim/models/keyedvectors.py b/gensim/models/keyedvectors.py
index daa5482184..b5debb21c1 100644
--- a/gensim/models/keyedvectors.py
+++ b/gensim/models/keyedvectors.py
@@ -171,6 +171,7 @@
import itertools
import warnings
from numbers import Integral
+from typing import Iterable
from numpy import (
dot, float32 as REAL, double, array, zeros, vstack,
@@ -178,6 +179,7 @@
)
import numpy as np
from scipy import stats
+from scipy.spatial.distance import cdist
from gensim import utils, matutils # utility fnc for pickling, common scipy operations etc
from gensim.corpora.dictionary import Dictionary
@@ -187,7 +189,21 @@
logger = logging.getLogger(__name__)
-KEY_TYPES = (str, int, np.integer)
+_KEY_TYPES = (str, int, np.integer)
+
+_EXTENDED_KEY_TYPES = (str, int, np.integer, np.ndarray)
+
+
+def _ensure_list(value):
+ """Ensure that the specified value is wrapped in a list, for those supported cases
+ where we also accept a single key or vector."""
+ if value is None:
+ return []
+
+ if isinstance(value, _KEY_TYPES) or (isinstance(value, ndarray) and len(value.shape) == 1):
+ return [value]
+
+ return value
class KeyedVectors(utils.SaveLoad):
@@ -375,7 +391,7 @@ def __getitem__(self, key_or_keys):
Vector representation for `key_or_keys` (1D if `key_or_keys` is single key, otherwise - 2D).
"""
- if isinstance(key_or_keys, KEY_TYPES):
+ if isinstance(key_or_keys, _KEY_TYPES):
return self.get_vector(key_or_keys)
return vstack([self.get_vector(key) for key in key_or_keys])
@@ -489,7 +505,7 @@ def add_vectors(self, keys, weights, extras=None, replace=False):
if True - replace vectors, otherwise - keep old vectors.
"""
- if isinstance(keys, KEY_TYPES):
+ if isinstance(keys, _KEY_TYPES):
keys = [keys]
weights = np.array(weights).reshape(1, -1)
elif isinstance(weights, list):
@@ -727,10 +743,9 @@ def most_similar(
if isinstance(topn, Integral) and topn < 1:
return []
- if positive is None:
- positive = []
- if negative is None:
- negative = []
+ # allow passing a single string-key or vector for the positive/negative arguments
+ positive = _ensure_list(positive)
+ negative = _ensure_list(negative)
self.fill_norms()
clip_end = clip_end or len(self.vectors)
@@ -739,18 +754,14 @@ def most_similar(
clip_start = 0
clip_end = restrict_vocab
- if isinstance(positive, KEY_TYPES) and not negative:
- # allow calls like most_similar('dog'), as a shorthand for most_similar(['dog'])
- positive = [positive]
-
# add weights for each key, if not already present; default to 1.0 for positive and -1.0 for negative keys
positive = [
- (item, 1.0) if isinstance(item, KEY_TYPES + (ndarray,))
- else item for item in positive
+ (item, 1.0) if isinstance(item, _EXTENDED_KEY_TYPES) else item
+ for item in positive
]
negative = [
- (item, -1.0) if isinstance(item, KEY_TYPES + (ndarray,))
- else item for item in negative
+ (item, -1.0) if isinstance(item, _EXTENDED_KEY_TYPES) else item
+ for item in negative
]
# compute the weighted average of all keys
@@ -901,23 +912,16 @@ def wmdistance(self, document1, document2, norm=True):
# Both documents are composed of a single unique token => zero distance.
return 0.0
- # Sets for faster look-up.
- docset1 = set(document1)
- docset2 = set(document2)
+ doclist1 = list(set(document1))
+ doclist2 = list(set(document2))
+ v1 = np.array([self.get_vector(token, norm=norm) for token in doclist1])
+ v2 = np.array([self.get_vector(token, norm=norm) for token in doclist2])
+ doc1_indices = dictionary.doc2idx(doclist1)
+ doc2_indices = dictionary.doc2idx(doclist2)
# Compute distance matrix.
distance_matrix = zeros((vocab_len, vocab_len), dtype=double)
- for i, t1 in dictionary.items():
- if t1 not in docset1:
- continue
-
- for j, t2 in dictionary.items():
- if t2 not in docset2 or distance_matrix[i, j] != 0.0:
- continue
-
- # Compute Euclidean distance between (potentially unit-normed) word vectors.
- distance_matrix[i, j] = distance_matrix[j, i] = np.sqrt(
- np_sum((self.get_vector(t1, norm=norm) - self.get_vector(t2, norm=norm))**2))
+ distance_matrix[np.ix_(doc1_indices, doc2_indices)] = cdist(v1, v2)
if abs(np_sum(distance_matrix)) < 1e-8:
# `emd` gets stuck if the distance matrix contains only zeros.
@@ -974,21 +978,16 @@ def most_similar_cosmul(self, positive=None, negative=None, topn=10):
if isinstance(topn, Integral) and topn < 1:
return []
- if positive is None:
- positive = []
- if negative is None:
- negative = []
+ # allow passing a single string-key or vector for the positive/negative arguments
+ positive = _ensure_list(positive)
+ negative = _ensure_list(negative)
self.fill_norms()
- if isinstance(positive, str) and not negative:
- # allow calls like most_similar_cosmul('dog'), as a shorthand for most_similar_cosmul(['dog'])
- positive = [positive]
-
all_words = {
self.get_index(word) for word in positive + negative
if not isinstance(word, ndarray) and word in self.key_to_index
- }
+ }
positive = [
self.get_vector(word, norm=True) if isinstance(word, str) else word
@@ -1106,7 +1105,7 @@ def distances(self, word_or_vector, other_words=()):
If either `word_or_vector` or any word in `other_words` is absent from vocab.
"""
- if isinstance(word_or_vector, KEY_TYPES):
+ if isinstance(word_or_vector, _KEY_TYPES):
input_vector = self.get_vector(word_or_vector)
else:
input_vector = word_or_vector
@@ -1695,6 +1694,70 @@ def intersect_word2vec_format(self, fname, lockf=0.0, binary=False, encoding='ut
msg=f"merged {overlap_count} vectors into {self.vectors.shape} matrix from {fname}",
)
+ def vectors_for_all(self, keys: Iterable, allow_inference: bool = True,
+ copy_vecattrs: bool = False) -> 'KeyedVectors':
+ """Produce vectors for all given keys as a new :class:`KeyedVectors` object.
+
+ Notes
+ -----
+ The keys will always be deduplicated. For optimal performance, you should not pass entire
+ corpora to the method. Instead, you should construct a dictionary of unique words in your
+ corpus:
+
+ >>> from collections import Counter
+ >>> import itertools
+ >>>
+ >>> from gensim.models import FastText
+ >>> from gensim.test.utils import datapath, common_texts
+ >>>
+ >>> model_corpus_file = datapath('lee_background.cor') # train word vectors on some corpus
+ >>> model = FastText(corpus_file=model_corpus_file, vector_size=20, min_count=1)
+ >>> corpus = common_texts # infer word vectors for words from another corpus
+ >>> word_counts = Counter(itertools.chain.from_iterable(corpus)) # count words in your corpus
+ >>> words_by_freq = (k for k, v in word_counts.most_common())
+ >>> word_vectors = model.wv.vectors_for_all(words_by_freq) # create word-vectors for words in your corpus
+
+ Parameters
+ ----------
+ keys : iterable
+ The keys that will be vectorized.
+ allow_inference : bool, optional
+ In subclasses such as :class:`~gensim.models.fasttext.FastTextKeyedVectors`,
+ vectors for out-of-vocabulary keys (words) may be inferred. Default is True.
+ copy_vecattrs : bool, optional
+ Additional attributes set via the :meth:`KeyedVectors.set_vecattr` method
+ will be preserved in the produced :class:`KeyedVectors` object. Default is False.
+ To ensure that *all* the produced vectors will have vector attributes assigned,
+ you should set `allow_inference=False`.
+
+ Returns
+ -------
+ keyedvectors : :class:`~gensim.models.keyedvectors.KeyedVectors`
+ Vectors for all the given keys.
+
+ """
+ # Pick only the keys that actually exist & deduplicate them.
+ # We keep the original key order, to improve cache locality, for performance.
+ vocab, seen = [], set()
+ for key in keys:
+ if key not in seen:
+ seen.add(key)
+ if key in (self if allow_inference else self.key_to_index):
+ vocab.append(key)
+
+ kv = KeyedVectors(self.vector_size, len(vocab), dtype=self.vectors.dtype)
+
+ for key in vocab: # produce and index vectors for all the given keys
+ weights = self[key]
+ _add_word_to_kv(kv, None, key, weights, len(vocab))
+ if copy_vecattrs:
+ for attr in self.expandos:
+ try:
+ kv.set_vecattr(key, attr, self.get_vecattr(key, attr))
+ except KeyError:
+ pass
+ return kv
+
def _upconvert_old_d2vkv(self):
"""Convert a deserialized older Doc2VecKeyedVectors instance to latest generic KeyedVectors"""
self.vocab = self.doctags
diff --git a/gensim/models/ldamodel.py b/gensim/models/ldamodel.py
index 439aeb91fb..6691ddcc31 100755
--- a/gensim/models/ldamodel.py
+++ b/gensim/models/ldamodel.py
@@ -4,7 +4,7 @@
# Copyright (C) 2011 Radim Rehurek
# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
-"""Optimized `Latent Dirichlet Allocation (LDA) ` in Python.
+"""Optimized `Latent Dirichlet Allocation (LDA) `_ in Python.
For a faster implementation of LDA (parallelized for multicore machines), see also :mod:`gensim.models.ldamulticore`.
@@ -13,9 +13,13 @@
for online training.
The core estimation code is based on the `onlineldavb.py script
-`_, by `Hoffman, Blei, Bach:
-Online Learning for Latent Dirichlet Allocation, NIPS 2010
-`_.
+`_, by
+Matthew D. Hoffman, David M. Blei, Francis Bach:
+`'Online Learning for Latent Dirichlet Allocation', NIPS 2010`_.
+
+.. _'Online Learning for Latent Dirichlet Allocation', NIPS 2010: online-lda_
+.. _'Online Learning for LDA' by Hoffman et al.: online-lda_
+.. _online-lda: https://papers.neurips.cc/paper/2010/file/71f6278d140af599e06ad9bf1ba03cb0-Paper.pdf
The algorithm:
@@ -198,8 +202,7 @@ def blend(self, rhot, other, targetsize=None):
The number of documents is stretched in both state objects, so that they are of comparable magnitude.
This procedure corresponds to the stochastic gradient update from
- `Hoffman et al. :"Online Learning for Latent Dirichlet Allocation"
- `_, see equations (5) and (9).
+ `'Online Learning for LDA' by Hoffman et al.`_, see equations (5) and (9).
Parameters
----------
@@ -311,8 +314,7 @@ def load(cls, fname, *args, **kwargs):
class LdaModel(interfaces.TransformationABC, basemodel.BaseTopicModel):
- """Train and use Online Latent Dirichlet Allocation (OLDA) models as presented in
- `Hoffman et al. :"Online Learning for Latent Dirichlet Allocation" `_.
+ """Train and use Online Latent Dirichlet Allocation model as presented in `'Online Learning for LDA' by Hoffman et al.`_
Examples
-------
@@ -374,30 +376,31 @@ def __init__(self, corpus=None, num_topics=100, id2word=None,
update_every : int, optional
Number of documents to be iterated through for each update.
Set to 0 for batch learning, > 1 for online iterative learning.
- alpha : {numpy.ndarray, str}, optional
- Can be set to an 1D array of length equal to the number of expected topics that expresses
- our a-priori belief for each topics' probability.
- Alternatively default prior selecting strategies can be employed by supplying a string:
+ alpha : {float, numpy.ndarray of float, list of float, str}, optional
+ A-priori belief on document-topic distribution, this can be:
+ * scalar for a symmetric prior over document-topic distribution,
+ * 1D array of length equal to num_topics to denote an asymmetric user defined prior for each topic.
- * 'symmetric': Default; uses a fixed symmetric prior per topic,
+ Alternatively default prior selecting strategies can be employed by supplying a string:
+ * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`,
* 'asymmetric': Uses a fixed normalized asymmetric prior of `1.0 / (topic_index + sqrt(num_topics))`,
* 'auto': Learns an asymmetric prior from the corpus (not available if `distributed==True`).
- eta : {float, np.array, str}, optional
- A-priori belief on word probability, this can be:
+ eta : {float, numpy.ndarray of float, list of float, str}, optional
+ A-priori belief on topic-word distribution, this can be:
+ * scalar for a symmetric prior over topic-word distribution,
+ * 1D array of length equal to num_words to denote an asymmetric user defined prior for each word,
+ * matrix of shape (num_topics, num_words) to assign a probability for each word-topic combination.
- * scalar for a symmetric prior over topic/word probability,
- * vector of length num_words to denote an asymmetric user defined probability for each word,
- * matrix of shape (num_topics, num_words) to assign a probability for each word-topic combination,
- * the string 'auto' to learn the asymmetric prior from the data.
+ Alternatively default prior selecting strategies can be employed by supplying a string:
+ * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`,
+ * 'auto': Learns an asymmetric prior from the corpus.
decay : float, optional
A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten
- when each new document is examined. Corresponds to Kappa from
- `Matthew D. Hoffman, David M. Blei, Francis Bach:
- "Online Learning for Latent Dirichlet Allocation NIPS'10" `_.
+ when each new document is examined.
+ Corresponds to :math:`\\kappa` from `'Online Learning for LDA' by Hoffman et al.`_
offset : float, optional
Hyper-parameter that controls how much we will slow down the first steps the first few iterations.
- Corresponds to Tau_0 from `Matthew D. Hoffman, David M. Blei, Francis Bach:
- "Online Learning for Latent Dirichlet Allocation NIPS'10" `_.
+ Corresponds to :math:`\\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_
eval_every : int, optional
Log perplexity is estimated every that many updates. Setting this to one slows down training by ~2x.
iterations : int, optional
@@ -409,7 +412,7 @@ def __init__(self, corpus=None, num_topics=100, id2word=None,
random_state : {np.random.RandomState, int}, optional
Either a randomState object or a seed to generate one. Useful for reproducibility.
ns_conf : dict of (str, object), optional
- Key word parameters propagated to :func:`gensim.utils.getNS` to get a Pyro4 Nameserved.
+ Key word parameters propagated to :func:`gensim.utils.getNS` to get a Pyro4 nameserver.
Only used if `distributed` is set to True.
minimum_phi_value : float, optional
if `per_word_topics` is True, this represents a lower bound on the term probabilities.
@@ -459,22 +462,16 @@ def __init__(self, corpus=None, num_topics=100, id2word=None,
self.callbacks = callbacks
self.alpha, self.optimize_alpha = self.init_dir_prior(alpha, 'alpha')
-
assert self.alpha.shape == (self.num_topics,), \
"Invalid alpha shape. Got shape %s, but expected (%d, )" % (str(self.alpha.shape), self.num_topics)
- if isinstance(eta, str):
- if eta == 'asymmetric':
- raise ValueError("The 'asymmetric' option cannot be used for eta")
-
self.eta, self.optimize_eta = self.init_dir_prior(eta, 'eta')
-
- self.random_state = utils.get_random_state(random_state)
-
assert self.eta.shape == (self.num_terms,) or self.eta.shape == (self.num_topics, self.num_terms), (
"Invalid eta shape. Got shape %s, but expected (%d, 1) or (%d, %d)" %
(str(self.eta.shape), self.num_terms, self.num_topics, self.num_terms))
+ self.random_state = utils.get_random_state(random_state)
+
# VB constants
self.iterations = iterations
self.gamma_threshold = gamma_threshold
@@ -531,24 +528,36 @@ def init_dir_prior(self, prior, name):
Parameters
----------
- prior : {str, list of float, numpy.ndarray of float, float}
- A-priori belief on word probability. If `name` == 'eta' then the prior can be:
+ prior : {float, numpy.ndarray of float, list of float, str}
+ A-priori belief on document-topic distribution. If `name` == 'alpha', then the prior can be:
+ * scalar for a symmetric prior over document-topic distribution,
+ * 1D array of length equal to num_topics to denote an asymmetric user defined prior for each topic.
- * scalar for a symmetric prior over topic/word probability,
- * vector of length num_words to denote an asymmetric user defined probability for each word,
- * matrix of shape (num_topics, num_words) to assign a probability for each word-topic combination,
- * the string 'auto' to learn the asymmetric prior from the data.
+ Alternatively default prior selecting strategies can be employed by supplying a string:
+ * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`,
+ * 'asymmetric': Uses a fixed normalized asymmetric prior of `1.0 / (topic_index + sqrt(num_topics))`,
+ * 'auto': Learns an asymmetric prior from the corpus (not available if `distributed==True`).
- If `name` == 'alpha', then the prior can be:
+ A-priori belief on topic-word distribution. If `name` == 'eta' then the prior can be:
+ * scalar for a symmetric prior over topic-word distribution,
+ * 1D array of length equal to num_words to denote an asymmetric user defined prior for each word,
+ * matrix of shape (num_topics, num_words) to assign a probability for each word-topic combination.
- * an 1D array of length equal to the number of expected topics,
- * 'symmetric': Uses a fixed symmetric prior per topic,
- * 'asymmetric': Uses a fixed normalized asymmetric prior of `1.0 / (topic_index + sqrt(num_topics))`,
+ Alternatively default prior selecting strategies can be employed by supplying a string:
+ * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`,
* 'auto': Learns an asymmetric prior from the corpus.
name : {'alpha', 'eta'}
Whether the `prior` is parameterized by the alpha vector (1 parameter per topic)
or by the eta (1 parameter per unique term in the vocabulary).
+ Returns
+ -------
+ init_prior: numpy.ndarray
+ Initialized Dirichlet prior:
+ If 'alpha' was provided as `name` the shape is (self.num_topics, ).
+ If 'eta' was provided as `name` the shape is (len(self.id2word), ).
+ is_auto: bool
+ Flag that shows if hyperparameter optimization should be used or not.
"""
if prior is None:
prior = 'symmetric'
@@ -570,6 +579,8 @@ def init_dir_prior(self, prior, name):
dtype=self.dtype, count=prior_shape,
)
elif prior == 'asymmetric':
+ if name == 'eta':
+ raise ValueError("The 'asymmetric' option cannot be used for eta")
init_prior = np.fromiter(
(1.0 / (i + np.sqrt(prior_shape)) for i in range(prior_shape)),
dtype=self.dtype, count=prior_shape,
@@ -632,7 +643,7 @@ def inference(self, chunk, collect_sstats=False):
"""Given a chunk of sparse document vectors, estimate gamma (parameters controlling the topic weights)
for each document in the chunk.
- This function does not modify the model The whole input chunk of document is assumed to fit in RAM;
+ This function does not modify the model. The whole input chunk of document is assumed to fit in RAM;
chunking of a large corpus must be done earlier in the pipeline. Avoids computing the `phi` variational
parameter directly using the optimization presented in
`Lee, Seung: Algorithms for non-negative matrix factorization"
@@ -693,7 +704,7 @@ def inference(self, chunk, collect_sstats=False):
expElogthetad = expElogtheta[d, :]
expElogbetad = self.expElogbeta[:, ids]
- # The optimal phi_{dwk} is proportional to expElogthetad_k * expElogbetad_w.
+ # The optimal phi_{dwk} is proportional to expElogthetad_k * expElogbetad_kw.
# phinorm is the normalizer.
# TODO treat zeros explicitly, instead of adding epsilon?
phinorm = np.dot(expElogthetad, expElogbetad) + epsilon
@@ -849,13 +860,15 @@ def update(self, corpus, chunksize=None, decay=None, offset=None,
Notes
-----
- This update also supports updating an already trained model with new documents; the two models are then merged
- in proportion to the number of old vs. new documents. This feature is still experimental for non-stationary
- input streams. For stationary input (no topic drift in new documents), on the other hand, this equals the
- online update of `Matthew D. Hoffman, David M. Blei, Francis Bach:
- "Online Learning for Latent Dirichlet Allocation NIPS'10" `_.
- and is guaranteed to converge for any `decay` in (0.5, 1.0). Additionally, for smaller corpus sizes, an
- increasing `offset` may be beneficial (see Table 1 in the same paper).
+ This update also supports updating an already trained model (`self`) with new documents from `corpus`;
+ the two models are then merged in proportion to the number of old vs. new documents.
+ This feature is still experimental for non-stationary input streams.
+
+ For stationary input (no topic drift in new documents), on the other hand,
+ this equals the online update of `'Online Learning for LDA' by Hoffman et al.`_
+ and is guaranteed to converge for any `decay` in (0.5, 1].
+ Additionally, for smaller corpus sizes,
+ an increasing `offset` may be beneficial (see Table 1 in the same paper).
Parameters
----------
@@ -866,13 +879,11 @@ def update(self, corpus, chunksize=None, decay=None, offset=None,
Number of documents to be used in each training chunk.
decay : float, optional
A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten
- when each new document is examined. Corresponds to Kappa from
- `Matthew D. Hoffman, David M. Blei, Francis Bach:
- "Online Learning for Latent Dirichlet Allocation NIPS'10" `_.
+ when each new document is examined. Corresponds to :math:`\\kappa` from
+ `'Online Learning for LDA' by Hoffman et al.`_
offset : float, optional
Hyper-parameter that controls how much we will slow down the first steps the first few iterations.
- Corresponds to Tau_0 from `Matthew D. Hoffman, David M. Blei, Francis Bach:
- "Online Learning for Latent Dirichlet Allocation NIPS'10" `_.
+ Corresponds to :math:`\\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_
passes : int, optional
Number of passes through the corpus during training.
update_every : int, optional
diff --git a/gensim/models/ldamulticore.py b/gensim/models/ldamulticore.py
index b65fcdd240..fdb5ce70a9 100644
--- a/gensim/models/ldamulticore.py
+++ b/gensim/models/ldamulticore.py
@@ -38,8 +38,13 @@
unseen documents. The model can also be updated with new documents for online training.
The core estimation code is based on the `onlineldavb.py script
-`_, by `Hoffman, Blei, Bach:
-Online Learning for Latent Dirichlet Allocation, NIPS 2010 `_.
+`_, by
+Matthew D. Hoffman, David M. Blei, Francis Bach:
+`'Online Learning for Latent Dirichlet Allocation', NIPS 2010`_.
+
+.. _'Online Learning for Latent Dirichlet Allocation', NIPS 2010: online-lda_
+.. _'Online Learning for LDA' by Hoffman et al.: online-lda_
+.. _online-lda: https://papers.neurips.cc/paper/2010/file/71f6278d140af599e06ad9bf1ba03cb0-Paper.pdf
Usage examples
--------------
@@ -128,28 +133,30 @@ def __init__(self, corpus=None, num_topics=100, id2word=None, workers=None,
Number of documents to be used in each training chunk.
passes : int, optional
Number of passes through the corpus during training.
- alpha : {np.ndarray, str}, optional
- Can be set to an 1D array of length equal to the number of expected topics that expresses
- our a-priori belief for the each topics' probability.
- Alternatively default prior selecting strategies can be employed by supplying a string:
+ alpha : {float, numpy.ndarray of float, list of float, str}, optional
+ A-priori belief on document-topic distribution, this can be:
+ * scalar for a symmetric prior over document-topic distribution,
+ * 1D array of length equal to num_topics to denote an asymmetric user defined prior for each topic.
- * 'asymmetric': Uses a fixed normalized asymmetric prior of `1.0 / topicno`.
- eta : {float, np.array, str}, optional
- A-priori belief on word probability, this can be:
+ Alternatively default prior selecting strategies can be employed by supplying a string:
+ * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`,
+ * 'asymmetric': Uses a fixed normalized asymmetric prior of `1.0 / (topic_index + sqrt(num_topics))`.
+ eta : {float, numpy.ndarray of float, list of float, str}, optional
+ A-priori belief on topic-word distribution, this can be:
+ * scalar for a symmetric prior over topic-word distribution,
+ * 1D array of length equal to num_words to denote an asymmetric user defined prior for each word,
+ * matrix of shape (num_topics, num_words) to assign a probability for each word-topic combination.
- * scalar for a symmetric prior over topic/word probability,
- * vector of length num_words to denote an asymmetric user defined probability for each word,
- * matrix of shape (num_topics, num_words) to assign a probability for each word-topic combination,
- * the string 'auto' to learn the asymmetric prior from the data.
+ Alternatively default prior selecting strategies can be employed by supplying a string:
+ * 'symmetric': (default) Uses a fixed symmetric prior of `1.0 / num_topics`,
+ * 'auto': Learns an asymmetric prior from the corpus.
decay : float, optional
A number between (0.5, 1] to weight what percentage of the previous lambda value is forgotten
- when each new document is examined. Corresponds to Kappa from
- `Matthew D. Hoffman, David M. Blei, Francis Bach:
- "Online Learning for Latent Dirichlet Allocation NIPS'10" `_.
+ when each new document is examined. Corresponds to :math:`\\kappa` from
+ `'Online Learning for LDA' by Hoffman et al.`_
offset : float, optional
Hyper-parameter that controls how much we will slow down the first steps the first few iterations.
- Corresponds to Tau_0 from `Matthew D. Hoffman, David M. Blei, Francis Bach:
- "Online Learning for Latent Dirichlet Allocation NIPS'10" `_.
+ Corresponds to :math:`\\tau_0` from `'Online Learning for LDA' by Hoffman et al.`_
eval_every : int, optional
Log perplexity is estimated every that many updates. Setting this to one slows down training by ~2x.
iterations : int, optional
@@ -174,7 +181,7 @@ def __init__(self, corpus=None, num_topics=100, id2word=None, workers=None,
self.batch = batch
if isinstance(alpha, str) and alpha == 'auto':
- raise NotImplementedError("auto-tuning alpha not implemented in multicore LDA; use plain LdaModel.")
+ raise NotImplementedError("auto-tuning alpha not implemented in LdaMulticore; use plain LdaModel.")
super(LdaMulticore, self).__init__(
corpus=corpus, num_topics=num_topics,
@@ -194,14 +201,13 @@ def update(self, corpus, chunks_as_numpy=False):
Notes
-----
- This update also supports updating an already trained model (`self`)
- with new documents from `corpus`; the two models are then merged in
- proportion to the number of old vs. new documents. This feature is still
- experimental for non-stationary input streams.
+ This update also supports updating an already trained model (`self`) with new documents from `corpus`;
+ the two models are then merged in proportion to the number of old vs. new documents.
+ This feature is still experimental for non-stationary input streams.
For stationary input (no topic drift in new documents), on the other hand,
- this equals the online update of Hoffman et al. and is guaranteed to
- converge for any `decay` in (0.5, 1.0>.
+ this equals the online update of `'Online Learning for LDA' by Hoffman et al.`_
+ and is guaranteed to converge for any `decay` in (0.5, 1].
Parameters
----------
diff --git a/gensim/models/ldaseqmodel.py b/gensim/models/ldaseqmodel.py
index 100aed748f..0f222c9c6c 100644
--- a/gensim/models/ldaseqmodel.py
+++ b/gensim/models/ldaseqmodel.py
@@ -5,8 +5,8 @@
# Based on Copyright (C) 2016 Radim Rehurek
"""Lda Sequence model, inspired by `David M. Blei, John D. Lafferty: "Dynamic Topic Models"
-`_ .
-The original C/C++ implementation can be found on `blei-lab/dtm `.
+`_.
+The original C/C++ implementation can be found on `blei-lab/dtm `_.
TODO: The next steps to take this forward would be:
diff --git a/gensim/models/lsi_dispatcher.py b/gensim/models/lsi_dispatcher.py
index b593e94cd3..2265dc7811 100755
--- a/gensim/models/lsi_dispatcher.py
+++ b/gensim/models/lsi_dispatcher.py
@@ -278,7 +278,11 @@ def exit(self):
logging.basicConfig(format='%(asctime)s - %(levelname)s - %(message)s', level=logging.INFO)
parser = argparse.ArgumentParser(description=__doc__[:-135], formatter_class=argparse.RawTextHelpFormatter)
parser.add_argument(
- 'maxsize', type=int, help='Maximum number of jobs to be kept pre-fetched in the queue.', default=MAX_JOBS_QUEUE
+ 'maxsize',
+ nargs='?',
+ type=int,
+ help='Maximum number of jobs to be kept pre-fetched in the queue.',
+ default=MAX_JOBS_QUEUE,
)
args = parser.parse_args()
diff --git a/gensim/models/lsimodel.py b/gensim/models/lsimodel.py
index 97cc921f34..06055722e1 100644
--- a/gensim/models/lsimodel.py
+++ b/gensim/models/lsimodel.py
@@ -670,7 +670,9 @@ def show_topic(self, topicno, topn=10):
c = np.asarray(self.projection.u.T[topicno, :]).flatten()
norm = np.sqrt(np.sum(np.dot(c, c)))
most = matutils.argsort(np.abs(c), topn, reverse=True)
- return [(self.id2word[val], 1.0 * c[val] / norm) for val in most]
+
+ # Output only (word, score) pairs for `val`s that are within `self.id2word`. See #3090 for details.
+ return [(self.id2word[val], 1.0 * c[val] / norm) for val in most if val in self.id2word]
def show_topics(self, num_topics=-1, num_words=10, log=False, formatted=True):
"""Get the most significant topics.
diff --git a/gensim/models/nmf.py b/gensim/models/nmf.py
index 626d9ca16a..132a0f8774 100644
--- a/gensim/models/nmf.py
+++ b/gensim/models/nmf.py
@@ -38,6 +38,7 @@
.. sourcecode:: pycon
+ >>> from gensim.models import Nmf
>>> from gensim.test.utils import common_texts
>>> from gensim.corpora.dictionary import Dictionary
>>>
diff --git a/gensim/models/phrases.py b/gensim/models/phrases.py
index 8e20333b8f..c95682fa5e 100644
--- a/gensim/models/phrases.py
+++ b/gensim/models/phrases.py
@@ -364,7 +364,7 @@ def load(cls, *args, **kwargs):
}
elif isinstance(component, tuple): # 3.8 => 4.0: phrasegram keys are strings, not tuples with bytestrings
model.phrasegrams = {
- str(model.delimiter.join(component), encoding='utf8'): score
+ str(model.delimiter.join(key), encoding='utf8'): val
for key, val in phrasegrams.items()
}
except StopIteration:
@@ -391,15 +391,13 @@ def load(cls, *args, **kwargs):
raise ValueError(f'failed to load {cls.__name__} model, unknown scoring "{model.scoring}"')
# common_terms didn't exist pre-3.?, and was renamed to connector in 4.0.0.
- if hasattr(model, "common_terms"):
- model.connector_words = model.common_terms
- del model.common_terms
- else:
- logger.warning(
- 'older version of %s loaded without common_terms attribute, setting connector_words to an empty set',
- cls.__name__,
- )
- model.connector_words = frozenset()
+ if not hasattr(model, "connector_words"):
+ if hasattr(model, "common_terms"):
+ model.connector_words = model.common_terms
+ del model.common_terms
+ else:
+ logger.warning('loaded older version of %s, setting connector_words to an empty set', cls.__name__)
+ model.connector_words = frozenset()
if not hasattr(model, 'corpus_word_count'):
logger.warning('older version of %s loaded without corpus_word_count', cls.__name__)
diff --git a/gensim/models/word2vec.py b/gensim/models/word2vec.py
index 23c5b90429..356f711408 100755
--- a/gensim/models/word2vec.py
+++ b/gensim/models/word2vec.py
@@ -240,7 +240,7 @@ def __init__(
max_vocab_size=None, sample=1e-3, seed=1, workers=3, min_alpha=0.0001,
sg=0, hs=0, negative=5, ns_exponent=0.75, cbow_mean=1, hashfxn=hash, epochs=5, null_word=0,
trim_rule=None, sorted_vocab=1, batch_words=MAX_WORDS_IN_BATCH, compute_loss=False, callbacks=(),
- comment=None, max_final_vocab=None,
+ comment=None, max_final_vocab=None, shrink_windows=True,
):
"""Train, use and evaluate neural networks described in https://code.google.com/p/word2vec/.
@@ -345,6 +345,12 @@ def __init__(
:meth:`~gensim.models.word2vec.Word2Vec.get_latest_training_loss`.
callbacks : iterable of :class:`~gensim.models.callbacks.CallbackAny2Vec`, optional
Sequence of callbacks to be executed at specific stages during training.
+ shrink_windows : bool, optional
+ New in 4.1. Experimental.
+ If True, the effective window size is uniformly sampled from [1, `window`]
+ for each target word during training, to match the original word2vec algorithm's
+ approximate weighting of context words by distance. Otherwise, the effective
+ window size is always fixed to `window` words to either side.
Examples
--------
@@ -377,6 +383,7 @@ def __init__(
self.min_alpha = float(min_alpha)
self.window = int(window)
+ self.shrink_windows = bool(shrink_windows)
self.random = np.random.RandomState(seed)
self.hs = int(hs)
@@ -522,7 +529,7 @@ def build_vocab_from_freq(
# to be directly the raw vocab
raw_vocab = word_freq
logger.info(
- "collected %i different raw word, with total frequency of %i",
+ "collected %i unique word types, with total frequency of %i",
len(raw_vocab), sum(raw_vocab.values()),
)
@@ -553,7 +560,7 @@ def _scan_vocab(self, sentences, progress_per, trim_rule):
if sentence_no % progress_per == 0:
logger.info(
"PROGRESS: at sentence #%i, processed %i words, keeping %i word types",
- sentence_no, total_words, len(vocab)
+ sentence_no, total_words, len(vocab),
)
for word in sentence:
vocab[word] += 1
@@ -604,8 +611,8 @@ def prepare_vocab(
# set effective_min_count to min_count in case max_final_vocab isn't set
self.effective_min_count = min_count
- # if max_final_vocab is specified instead of min_count
- # pick a min_count which satisfies max_final_vocab as well as possible
+ # If max_final_vocab is specified instead of min_count,
+ # pick a min_count which satisfies max_final_vocab as well as possible.
if self.max_final_vocab is not None:
sorted_vocab = sorted(self.raw_vocab.keys(), key=lambda word: self.raw_vocab[word], reverse=True)
calc_min_count = 1
@@ -910,12 +917,12 @@ def _do_train_epoch(
if self.sg:
examples, tally, raw_tally = train_epoch_sg(
self, corpus_file, offset, cython_vocab, cur_epoch,
- total_examples, total_words, work, neu1, self.compute_loss,
+ total_examples, total_words, work, neu1, self.compute_loss
)
else:
examples, tally, raw_tally = train_epoch_cbow(
self, corpus_file, offset, cython_vocab, cur_epoch,
- total_examples, total_words, work, neu1, self.compute_loss,
+ total_examples, total_words, work, neu1, self.compute_loss
)
return examples, tally, raw_tally
@@ -1039,7 +1046,7 @@ def train(
msg=(
f"training model with {self.workers} workers on {len(self.wv)} vocabulary and "
f"{self.layer1_size} features, using sg={self.sg} hs={self.hs} sample={self.sample} "
- f"negative={self.negative} window={self.window}"
+ f"negative={self.negative} window={self.window} shrink_windows={self.shrink_windows}"
),
)
@@ -1799,8 +1806,9 @@ def predict_output_word(self, context_words_list, topn=10):
Parameters
----------
- context_words_list : list of str
- List of context words.
+ context_words_list : list of (str and/or int)
+ List of context words, which may be words themselves (str)
+ or their index in `self.wv.vectors` (int).
topn : int, optional
Return `topn` words and their probabilities.
@@ -1818,8 +1826,8 @@ def predict_output_word(self, context_words_list, topn=10):
if not hasattr(self.wv, 'vectors') or not hasattr(self, 'syn1neg'):
raise RuntimeError("Parameters required for predicting the output words not found.")
-
word2_indices = [self.wv.get_index(w) for w in context_words_list if w in self.wv]
+
if not word2_indices:
logger.warning("All the input context words are out-of-vocabulary for the current model.")
return None
@@ -1830,7 +1838,7 @@ def predict_output_word(self, context_words_list, topn=10):
# propagate hidden -> output and take softmax to get probabilities
prob_values = np.exp(np.dot(l1, self.syn1neg.T))
- prob_values /= sum(prob_values)
+ prob_values /= np.sum(prob_values)
top_indices = matutils.argsort(prob_values, topn=topn, reverse=True)
# returning the most probable output words with their probabilities
return [(self.wv.index_to_key[index1], prob_values[index1]) for index1 in top_indices]
@@ -1970,6 +1978,8 @@ def _load_specials(self, *args, **kwargs):
self.syn1 = self.syn1
del self.syn1
del self.trainables
+ if not hasattr(self, 'shrink_windows'):
+ self.shrink_windows = True
def get_latest_training_loss(self):
"""Get current value of the training loss.
diff --git a/gensim/models/word2vec_corpusfile.pyx b/gensim/models/word2vec_corpusfile.pyx
index 19b9b8c165..5d7f5004e4 100644
--- a/gensim/models/word2vec_corpusfile.pyx
+++ b/gensim/models/word2vec_corpusfile.pyx
@@ -186,7 +186,9 @@ cdef void prepare_c_structures_for_batch(
vector[vector[string]] &sentences, int sample, int hs, int window, long long *total_words,
int *effective_words, int *effective_sentences, unsigned long long *next_random,
cvocab_t *vocab, int *sentence_idx, np.uint32_t *indexes, int *codelens,
- np.uint8_t **codes, np.uint32_t **points, np.uint32_t *reduced_windows) nogil:
+ np.uint8_t **codes, np.uint32_t **points, np.uint32_t *reduced_windows,
+ int shrink_windows,
+ ) nogil:
cdef VocabItem word
cdef string token
cdef vector[string] sent
@@ -224,8 +226,12 @@ cdef void prepare_c_structures_for_batch(
break # TODO: log warning, tally overflow?
# precompute "reduced window" offsets in a single randint() call
- for i in range(effective_words[0]):
- reduced_windows[i] = random_int32(next_random) % window
+ if shrink_windows:
+ for i in range(effective_words[0]):
+ reduced_windows[i] = random_int32(next_random) % window
+ else:
+ for i in range(effective_words[0]):
+ reduced_windows[i] = 0
cdef REAL_t get_alpha(REAL_t alpha, REAL_t end_alpha, int cur_epoch, int num_epochs) nogil:
@@ -250,7 +256,7 @@ cdef REAL_t get_next_alpha(
def train_epoch_sg(model, corpus_file, offset, _cython_vocab, _cur_epoch, _expected_examples, _expected_words, _work,
- _neu1, compute_loss):
+ _neu1, compute_loss,):
"""Train Skipgram model for one epoch by training on an input stream. This function is used only in multistream mode.
Called internally from :meth:`~gensim.models.word2vec.Word2Vec.train`.
@@ -295,6 +301,7 @@ def train_epoch_sg(model, corpus_file, offset, _cython_vocab, _cur_epoch, _expec
cdef long long total_sentences = 0
cdef long long total_effective_words = 0, total_words = 0
cdef int sent_idx, idx_start, idx_end
+ cdef int shrink_windows = int(model.shrink_windows)
init_w2v_config(&c, model, _alpha, compute_loss, _work)
@@ -311,7 +318,7 @@ def train_epoch_sg(model, corpus_file, offset, _cython_vocab, _cur_epoch, _expec
prepare_c_structures_for_batch(
sentences, c.sample, c.hs, c.window, &total_words, &effective_words, &effective_sentences,
&c.next_random, vocab.get_vocab_ptr(), c.sentence_idx, c.indexes,
- c.codelens, c.codes, c.points, c.reduced_windows)
+ c.codelens, c.codes, c.points, c.reduced_windows, shrink_windows)
for sent_idx in range(effective_sentences):
idx_start = c.sentence_idx[sent_idx]
@@ -350,7 +357,7 @@ def train_epoch_sg(model, corpus_file, offset, _cython_vocab, _cur_epoch, _expec
def train_epoch_cbow(model, corpus_file, offset, _cython_vocab, _cur_epoch, _expected_examples, _expected_words, _work,
- _neu1, compute_loss):
+ _neu1, compute_loss,):
"""Train CBOW model for one epoch by training on an input stream. This function is used only in multistream mode.
Called internally from :meth:`~gensim.models.word2vec.Word2Vec.train`.
@@ -395,6 +402,7 @@ def train_epoch_cbow(model, corpus_file, offset, _cython_vocab, _cur_epoch, _exp
cdef long long total_sentences = 0
cdef long long total_effective_words = 0, total_words = 0
cdef int sent_idx, idx_start, idx_end
+ cdef int shrink_windows = int(model.shrink_windows)
init_w2v_config(&c, model, _alpha, compute_loss, _work, _neu1)
@@ -411,7 +419,7 @@ def train_epoch_cbow(model, corpus_file, offset, _cython_vocab, _cur_epoch, _exp
prepare_c_structures_for_batch(
sentences, c.sample, c.hs, c.window, &total_words, &effective_words,
&effective_sentences, &c.next_random, vocab.get_vocab_ptr(), c.sentence_idx,
- c.indexes, c.codelens, c.codes, c.points, c.reduced_windows)
+ c.indexes, c.codelens, c.codes, c.points, c.reduced_windows, shrink_windows)
for sent_idx in range(effective_sentences):
idx_start = c.sentence_idx[sent_idx]
diff --git a/gensim/models/word2vec_inner.pyx b/gensim/models/word2vec_inner.pyx
index 50bfc803bd..ffdc908b5c 100755
--- a/gensim/models/word2vec_inner.pyx
+++ b/gensim/models/word2vec_inner.pyx
@@ -112,12 +112,12 @@ cdef void w2v_fast_sentence_sg_hs(
"""
cdef long long a, b
- cdef long long row1 = word2_index * size, row2, sgn
+ cdef long long row1 = word2_index * size, row2, sgn
cdef REAL_t f, g, f_dot, lprob
memset(work, 0, size * cython.sizeof(REAL_t))
for b in range(codelen):
- row2 = word_point[b] * size
+ row2 = word_point[b] * size
f_dot = our_dot(&size, &syn0[row1], &ONE, &syn1[row2], &ONE)
if f_dot <= -MAX_EXP or f_dot >= MAX_EXP:
continue
@@ -206,7 +206,7 @@ cdef unsigned long long w2v_fast_sentence_sg_neg(
"""
cdef long long a
- cdef long long row1 = word2_index * size, row2
+ cdef long long row1 = word2_index * size, row2
cdef unsigned long long modulo = 281474976710655ULL
cdef REAL_t f, g, label, f_dot, log_e_f_dot
cdef np.uint32_t target_index
@@ -225,7 +225,7 @@ cdef unsigned long long w2v_fast_sentence_sg_neg(
continue
label = 0.0
- row2 = target_index * size
+ row2 = target_index * size
f_dot = our_dot(&size, &syn0[row1], &ONE, &syn1neg[row2], &ONE)
if f_dot <= -MAX_EXP or f_dot >= MAX_EXP:
continue
@@ -309,7 +309,7 @@ cdef void w2v_fast_sentence_cbow_hs(
continue
else:
count += ONEF
- our_saxpy(&size, &ONEF, &syn0[indexes[m] * size], &ONE, neu1, &ONE)
+ our_saxpy(&size, &ONEF, &syn0[indexes[m] * size], &ONE, neu1, &ONE)
if count > (0.5):
inv_count = ONEF/count
if cbow_mean:
@@ -317,7 +317,7 @@ cdef void w2v_fast_sentence_cbow_hs(
memset(work, 0, size * cython.sizeof(REAL_t))
for b in range(codelens[i]):
- row2 = word_point[b] * size
+ row2 = word_point[b] * size
f_dot = our_dot(&size, neu1, &ONE, &syn1[row2], &ONE)
if f_dot <= -MAX_EXP or f_dot >= MAX_EXP:
continue
@@ -342,7 +342,7 @@ cdef void w2v_fast_sentence_cbow_hs(
if m == i:
continue
else:
- our_saxpy(&size, &words_lockf[indexes[m] % lockf_len], work, &ONE, &syn0[indexes[m] * size], &ONE)
+ our_saxpy(&size, &words_lockf[indexes[m] % lockf_len], work, &ONE, &syn0[indexes[m] * size], &ONE)
cdef unsigned long long w2v_fast_sentence_cbow_neg(
@@ -416,7 +416,7 @@ cdef unsigned long long w2v_fast_sentence_cbow_neg(
continue
else:
count += ONEF
- our_saxpy(&size, &ONEF, &syn0[indexes[m] * size], &ONE, neu1, &ONE)
+ our_saxpy(&size, &ONEF, &syn0[indexes[m] * size], &ONE, neu1, &ONE)
if count > (0.5):
inv_count = ONEF/count
if cbow_mean:
@@ -435,7 +435,7 @@ cdef unsigned long long w2v_fast_sentence_cbow_neg(
continue
label = 0.0
- row2 = target_index * size
+ row2 = target_index * size
f_dot = our_dot(&size, neu1, &ONE, &syn1neg[row2], &ONE)
if f_dot <= -MAX_EXP or f_dot >= MAX_EXP:
continue
@@ -459,7 +459,7 @@ cdef unsigned long long w2v_fast_sentence_cbow_neg(
if m == i:
continue
else:
- our_saxpy(&size, &words_lockf[indexes[m] % lockf_len], work, &ONE, &syn0[indexes[m]*size], &ONE)
+ our_saxpy(&size, &words_lockf[indexes[m] % lockf_len], work, &ONE, &syn0[indexes[m] * size], &ONE)
return next_random
@@ -566,8 +566,12 @@ def train_batch_sg(model, sentences, alpha, _work, compute_loss):
break # TODO: log warning, tally overflow?
# precompute "reduced window" offsets in a single randint() call
- for i, item in enumerate(model.random.randint(0, c.window, effective_words)):
- c.reduced_windows[i] = item
+ if model.shrink_windows:
+ for i, item in enumerate(model.random.randint(0, c.window, effective_words)):
+ c.reduced_windows[i] = item
+ else:
+ for i in range(effective_words):
+ c.reduced_windows[i] = 0
# release GIL & train on all sentences
with nogil:
@@ -662,8 +666,12 @@ def train_batch_cbow(model, sentences, alpha, _work, _neu1, compute_loss):
break # TODO: log warning, tally overflow?
# precompute "reduced window" offsets in a single randint() call
- for i, item in enumerate(model.random.randint(0, c.window, effective_words)):
- c.reduced_windows[i] = item
+ if model.shrink_windows:
+ for i, item in enumerate(model.random.randint(0, c.window, effective_words)):
+ c.reduced_windows[i] = item
+ else:
+ for i in range(effective_words):
+ c.reduced_windows[i] = 0
# release GIL & train on all sentences
with nogil:
@@ -784,11 +792,11 @@ cdef void score_pair_sg_hs(
const np.uint32_t word2_index, REAL_t *work) nogil:
cdef long long b
- cdef long long row1 = word2_index * size, row2, sgn
+ cdef long long row1 = word2_index * size, row2, sgn
cdef REAL_t f
for b in range(codelen):
- row2 = word_point[b] * size
+ row2 = word_point[b] * size
f = our_dot(&size, &syn0[row1], &ONE, &syn1[row2], &ONE)
sgn = (-1)**word_code[b] # ch function: 0-> 1, 1 -> -1
f *= sgn
@@ -889,14 +897,14 @@ cdef void score_pair_cbow_hs(
continue
else:
count += ONEF
- our_saxpy(&size, &ONEF, &syn0[indexes[m] * size], &ONE, neu1, &ONE)
+ our_saxpy(&size, &ONEF, &syn0[indexes[m] * size], &ONE, neu1, &ONE)
if count > (0.5):
inv_count = ONEF/count
if cbow_mean:
sscal(&size, &inv_count, neu1, &ONE)
for b in range(codelens[i]):
- row2 = word_point[b] * size
+ row2 = word_point[b] * size
f = our_dot(&size, neu1, &ONE, &syn1[row2], &ONE)
sgn = (-1)**word_code[b] # ch function: 0-> 1, 1 -> -1
f *= sgn
diff --git a/gensim/parsing/__init__.py b/gensim/parsing/__init__.py
index 5bbf84239e..c608bf399b 100644
--- a/gensim/parsing/__init__.py
+++ b/gensim/parsing/__init__.py
@@ -1,8 +1,18 @@
"""This package contains functions to preprocess raw text"""
from .porter import PorterStemmer # noqa:F401
-from .preprocessing import (remove_stopwords, strip_punctuation, strip_punctuation2, # noqa:F401
- strip_tags, strip_short, strip_numeric,
- strip_non_alphanum, strip_multiple_whitespaces,
- split_alphanum, stem_text, preprocess_string,
- preprocess_documents, read_file, read_files)
+from .preprocessing import ( # noqa:F401
+ preprocess_documents,
+ preprocess_string,
+ read_file,
+ read_files,
+ remove_stopwords,
+ split_alphanum,
+ stem_text,
+ strip_multiple_whitespaces,
+ strip_non_alphanum,
+ strip_numeric,
+ strip_punctuation,
+ strip_short,
+ strip_tags,
+)
diff --git a/gensim/parsing/preprocessing.py b/gensim/parsing/preprocessing.py
index 777ca46e8e..bb96b1ec2f 100644
--- a/gensim/parsing/preprocessing.py
+++ b/gensim/parsing/preprocessing.py
@@ -68,17 +68,20 @@
RE_WHITESPACE = re.compile(r"(\s)+", re.UNICODE)
-def remove_stopwords(s):
+def remove_stopwords(s, stopwords=None):
"""Remove :const:`~gensim.parsing.preprocessing.STOPWORDS` from `s`.
Parameters
----------
s : str
+ stopwords : iterable of str, optional
+ Sequence of stopwords
+ If None - using :const:`~gensim.parsing.preprocessing.STOPWORDS`
Returns
-------
str
- Unicode string without :const:`~gensim.parsing.preprocessing.STOPWORDS`.
+ Unicode string without `stopwords`.
Examples
--------
@@ -90,11 +93,33 @@ def remove_stopwords(s):
"""
s = utils.to_unicode(s)
- return " ".join(w for w in s.split() if w not in STOPWORDS)
+ return " ".join(remove_stopword_tokens(s.split(), stopwords))
+
+
+def remove_stopword_tokens(tokens, stopwords=None):
+ """Remove stopword tokens using list `stopwords`.
+
+ Parameters
+ ----------
+ tokens : iterable of str
+ Sequence of tokens.
+ stopwords : iterable of str, optional
+ Sequence of stopwords
+ If None - using :const:`~gensim.parsing.preprocessing.STOPWORDS`
+
+ Returns
+ -------
+ list of str
+ List of tokens without `stopwords`.
+
+ """
+ if stopwords is None:
+ stopwords = STOPWORDS
+ return [token for token in tokens if token not in stopwords]
def strip_punctuation(s):
- """Replace punctuation characters with spaces in `s` using :const:`~gensim.parsing.preprocessing.RE_PUNCT`.
+ """Replace ASCII punctuation characters with spaces in `s` using :const:`~gensim.parsing.preprocessing.RE_PUNCT`.
Parameters
----------
@@ -115,12 +140,10 @@ def strip_punctuation(s):
"""
s = utils.to_unicode(s)
+ # For unicode enhancement options see https://github.com/RaRe-Technologies/gensim/issues/2962
return RE_PUNCT.sub(" ", s)
-strip_punctuation2 = strip_punctuation
-
-
def strip_tags(s):
"""Remove tags from `s` using :const:`~gensim.parsing.preprocessing.RE_TAGS`.
@@ -172,7 +195,26 @@ def strip_short(s, minsize=3):
"""
s = utils.to_unicode(s)
- return " ".join(e for e in s.split() if len(e) >= minsize)
+ return " ".join(remove_short_tokens(s.split(), minsize))
+
+
+def remove_short_tokens(tokens, minsize=3):
+ """Remove tokens shorter than `minsize` chars.
+
+ Parameters
+ ----------
+ tokens : iterable of str
+ Sequence of tokens.
+ minsize : int, optimal
+ Minimal length of token (include).
+
+ Returns
+ -------
+ list of str
+ List of tokens without short tokens.
+ """
+
+ return [token for token in tokens if len(token) >= minsize]
def strip_numeric(s):
@@ -310,6 +352,49 @@ def stem_text(text):
stem = stem_text
+def lower_to_unicode(text, encoding='utf8', errors='strict'):
+ """Lowercase `text` and convert to unicode, using :func:`gensim.utils.any2unicode`.
+
+ Parameters
+ ----------
+ text : str
+ Input text.
+ encoding : str, optional
+ Encoding that will be used for conversion.
+ errors : str, optional
+ Error handling behaviour, used as parameter for `unicode` function (python2 only).
+
+ Returns
+ -------
+ str
+ Unicode version of `text`.
+
+ See Also
+ --------
+ :func:`gensim.utils.any2unicode`
+ Convert any string to unicode-string.
+
+ """
+ return utils.to_unicode(text.lower(), encoding, errors)
+
+
+def split_on_space(s):
+ """Split line by spaces, used in :class:`gensim.corpora.lowcorpus.LowCorpus`.
+
+ Parameters
+ ----------
+ s : str
+ Some line.
+
+ Returns
+ -------
+ list of str
+ List of tokens from `s`.
+
+ """
+ return [word for word in utils.to_unicode(s).strip().split(' ') if word]
+
+
DEFAULT_FILTERS = [
lambda x: x.lower(), strip_tags, strip_punctuation,
strip_multiple_whitespaces, strip_numeric,
diff --git a/gensim/scripts/make_wiki_online.py b/gensim/scripts/make_wiki_online.py
index 0ec9704724..e5ee11283a 100755
--- a/gensim/scripts/make_wiki_online.py
+++ b/gensim/scripts/make_wiki_online.py
@@ -16,7 +16,7 @@
* `OUTPUT_PREFIX_wordids.txt`: mapping between words and their integer ids
* `OUTPUT_PREFIX_bow.mm`: bag-of-words (word counts) representation, in
- Matrix Matrix format
+ Matrix Market format
* `OUTPUT_PREFIX_tfidf.mm`: TF-IDF representation
* `OUTPUT_PREFIX.tfidf_model`: TF-IDF model dump
@@ -35,7 +35,6 @@
python -m gensim.scripts.make_wikicorpus ~/gensim/results/enwiki-latest-pages-articles.xml.bz2 ~/gensim/results/wiki
"""
-
import logging
import os.path
import sys
@@ -43,13 +42,11 @@
from gensim.corpora import Dictionary, HashDictionary, MmCorpus, WikiCorpus
from gensim.models import TfidfModel
-
# Wiki is first scanned for all distinct word types (~7M). The types that
# appear in more than 10% of articles are removed and from the rest, the
# DEFAULT_DICT_SIZE most frequent types are kept.
DEFAULT_DICT_SIZE = 100000
-
if __name__ == '__main__':
program = os.path.basename(sys.argv[0])
logger = logging.getLogger(program)
diff --git a/gensim/scripts/make_wikicorpus.py b/gensim/scripts/make_wikicorpus.py
index 66056cf10b..b76c8c2bd5 100755
--- a/gensim/scripts/make_wikicorpus.py
+++ b/gensim/scripts/make_wikicorpus.py
@@ -18,7 +18,7 @@
* `OUTPUT_PREFIX_bow.mm`: bag-of-words (word counts) representation in Matrix Market format
* `OUTPUT_PREFIX_bow.mm.index`: index for `OUTPUT_PREFIX_bow.mm`
* `OUTPUT_PREFIX_bow.mm.metadata.cpickle`: titles of documents
-* `OUTPUT_PREFIX_tfidf.mm`: TF-IDF representation in Matix Market format
+* `OUTPUT_PREFIX_tfidf.mm`: TF-IDF representation in Matrix Market format
* `OUTPUT_PREFIX_tfidf.mm.index`: index for `OUTPUT_PREFIX_tfidf.mm`
* `OUTPUT_PREFIX.tfidf_model`: TF-IDF model
diff --git a/gensim/similarities/__init__.py b/gensim/similarities/__init__.py
index 00464d29c3..3fdf94bd3a 100644
--- a/gensim/similarities/__init__.py
+++ b/gensim/similarities/__init__.py
@@ -3,19 +3,7 @@
"""
# bring classes directly into package namespace, to save some typing
-import warnings
-try:
- import Levenshtein # noqa:F401
-except ImportError:
- msg = (
- "The gensim.similarities.levenshtein submodule is disabled, because the optional "
- "Levenshtein package
+# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
+# Code adapted from TinyFastSS (public domain), https://github.com/fujimotos/TinyFastSS
+
+"""Fast approximate string similarity search using the FastSS algorithm."""
+
+import itertools
+
+from cpython.ref cimport PyObject
+
+
+DEF MAX_WORD_LENGTH = 1000 # Maximum allowed word length, in characters. Must fit in the C `int` range.
+
+
+cdef extern from *:
+ """
+ #define WIDTH int
+ #define MAX_WORD_LENGTH 1000
+
+ int ceditdist(PyObject * s1, PyObject * s2, WIDTH maximum) {
+ WIDTH row1[MAX_WORD_LENGTH + 1];
+ WIDTH row2[MAX_WORD_LENGTH + 1];
+ WIDTH * CYTHON_RESTRICT pos_new;
+ WIDTH * CYTHON_RESTRICT pos_old;
+ int row_flip = 1; /* Does pos_new represent row1 or row2? */
+ int kind1 = PyUnicode_KIND(s1); /* How many bytes per unicode codepoint? */
+ int kind2 = PyUnicode_KIND(s2);
+
+ WIDTH len_s1 = (WIDTH)PyUnicode_GET_LENGTH(s1);
+ WIDTH len_s2 = (WIDTH)PyUnicode_GET_LENGTH(s2);
+ if (len_s1 > len_s2) {
+ PyObject * tmp = s1; s1 = s2; s2 = tmp;
+ const WIDTH tmpi = len_s1; len_s1 = len_s2; len_s2 = tmpi;
+ }
+ if (len_s2 - len_s1 > maximum) return maximum + 1;
+ if (len_s2 > MAX_WORD_LENGTH) return -1;
+ void * s1_data = PyUnicode_DATA(s1);
+ void * s2_data = PyUnicode_DATA(s2);
+
+ for (WIDTH tmpi = 0; tmpi <= len_s1; tmpi++) row2[tmpi] = tmpi;
+
+ for (WIDTH i2 = 0; i2 < len_s2; i2++) {
+ int all_bad = i2 >= maximum;
+ const Py_UCS4 ch = PyUnicode_READ(kind2, s2_data, i2);
+ row_flip = 1 - row_flip;
+ if (row_flip) {
+ pos_new = row2; pos_old = row1;
+ } else {
+ pos_new = row1; pos_old = row2;
+ }
+ *pos_new = i2 + 1;
+
+ for (WIDTH i1 = 0; i1 < len_s1; i1++) {
+ WIDTH val = *(pos_old++);
+ if (ch != PyUnicode_READ(kind1, s1_data, i1)) {
+ const WIDTH _val1 = *pos_old;
+ const WIDTH _val2 = *pos_new;
+ if (_val1 < val) val = _val1;
+ if (_val2 < val) val = _val2;
+ val += 1;
+ }
+ *(++pos_new) = val;
+ if (all_bad && val <= maximum) all_bad = 0;
+ }
+ if (all_bad) return maximum + 1;
+ }
+
+ return row_flip ? row2[len_s1] : row1[len_s1];
+ }
+ """
+ int ceditdist(PyObject *s1, PyObject *s2, int maximum)
+
+
+def editdist(s1: str, s2: str, max_dist=None):
+ """
+ Return the Levenshtein distance between two strings.
+
+ Use `max_dist` to control the maximum distance you care about. If the actual distance is larger
+ than `max_dist`, editdist will return early, with the value `max_dist+1`.
+ This is a performance optimization – for example if anything above distance 2 is uninteresting
+ to your application, call editdist with `max_dist=2` and ignore any return value greater than 2.
+
+ Leave `max_dist=None` (default) to always return the full Levenshtein distance (slower).
+
+ """
+ if s1 == s2:
+ return 0
+
+ result = ceditdist(s1, s2, MAX_WORD_LENGTH if max_dist is None else int(max_dist))
+ if result >= 0:
+ return result
+ elif result == -1:
+ raise ValueError(f"editdist doesn't support strings longer than {MAX_WORD_LENGTH} characters")
+ else:
+ raise ValueError(f"editdist returned an error: {result}")
+
+
+def indexkeys(word, max_dist):
+ """Return the set of index keys ("variants") of a word.
+
+ >>> indexkeys('aiu', 1)
+ {'aiu', 'iu', 'au', 'ai'}
+ """
+ res = set()
+ wordlen = len(word)
+ limit = min(max_dist, wordlen) + 1
+
+ for dist in range(limit):
+ for variant in itertools.combinations(word, wordlen - dist):
+ res.add(''.join(variant))
+
+ return res
+
+
+def set2bytes(s):
+ """Serialize a set of unicode strings into bytes.
+
+ >>> set2byte({u'a', u'b', u'c'})
+ b'a\x00b\x00c'
+ """
+ return '\x00'.join(s).encode('utf8')
+
+
+def bytes2set(b):
+ """Deserialize bytes into a set of unicode strings.
+
+ >>> bytes2set(b'a\x00b\x00c')
+ {u'a', u'b', u'c'}
+ """
+ return set(b.decode('utf8').split('\x00')) if b else set()
+
+
+class FastSS:
+
+ def __init__(self, words=None, max_dist=2):
+ """
+ Create a FastSS index. The index will contain encoded variants of all
+ indexed words, allowing fast "fuzzy string similarity" queries.
+
+ max_dist: maximum allowed edit distance of an indexed word to a query word. Keep
+ max_dist<=3 for sane performance.
+
+ """
+ self.db = {}
+ self.max_dist = max_dist
+ if words:
+ for word in words:
+ self.add(word)
+
+ def __str__(self):
+ return "%s" % (self.__class__.__name__, self.max_dist, len(self.db), )
+
+ def __contains__(self, word):
+ bkey = word.encode('utf8')
+ if bkey in self.db:
+ return word in bytes2set(self.db[bkey])
+ return False
+
+ def add(self, word):
+ """Add a string to the index."""
+ for key in indexkeys(word, self.max_dist):
+ bkey = key.encode('utf8')
+ wordset = {word}
+
+ if bkey in self.db:
+ wordset |= bytes2set(self.db[bkey])
+
+ self.db[bkey] = set2bytes(wordset)
+
+ def query(self, word, max_dist=None):
+ """Find all words from the index that are within max_dist of `word`."""
+ if max_dist is None:
+ max_dist = self.max_dist
+ if max_dist > self.max_dist:
+ raise ValueError(
+ f"query max_dist={max_dist} cannot be greater than max_dist={self.max_dist} from the constructor"
+ )
+
+ res = {d: [] for d in range(max_dist + 1)}
+ cands = set()
+
+ for key in indexkeys(word, max_dist):
+ bkey = key.encode('utf8')
+
+ if bkey in self.db:
+ cands.update(bytes2set(self.db[bkey]))
+
+ for cand in cands:
+ dist = editdist(word, cand, max_dist=max_dist)
+ if dist <= max_dist:
+ res[dist].append(cand)
+
+ return res
diff --git a/gensim/similarities/levenshtein.py b/gensim/similarities/levenshtein.py
index ca39e68dd0..51da72c065 100644
--- a/gensim/similarities/levenshtein.py
+++ b/gensim/similarities/levenshtein.py
@@ -5,149 +5,101 @@
# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
"""
-This module provides a namespace for functions that use the Levenshtein distance.
+This module allows fast fuzzy search between strings, using kNN queries with Levenshtein similarity.
"""
-import itertools
import logging
-from math import floor
from gensim.similarities.termsim import TermSimilarityIndex
+from gensim import utils
+try:
+ from gensim.similarities.fastss import FastSS, editdist # noqa:F401
+except ImportError:
+ raise utils.NO_CYTHON
-logger = logging.getLogger(__name__)
-
-
-def levdist(t1, t2, max_distance=float("inf")):
- """Get the Levenshtein distance between two terms.
-
- Return the Levenshtein distance between two terms. The distance is a
- number between <1.0, inf>, higher is less similar.
-
- Parameters
- ----------
- t1 : {bytes, str, unicode}
- The first compared term.
- t2 : {bytes, str, unicode}
- The second compared term.
- max_distance : {int, float}, optional
- If you don't care about distances larger than a known threshold, a more
- efficient code path can be taken. For terms that are clearly "too far
- apart", we will not compute the distance exactly, but we will return
- `max(len(t1), len(t2))` more quickly, meaning "more than
- `max_distance`".
- Default: always compute distance exactly, no threshold clipping.
-
- Returns
- -------
- int
- The Levenshtein distance between `t1` and `t2`.
-
- """
- import Levenshtein
-
- distance = Levenshtein.distance(t1, t2)
- if distance > max_distance:
- return max(len(t1), len(t2))
- return distance
-
-
-def levsim(t1, t2, alpha=1.8, beta=5.0, min_similarity=0.0):
- """Get the Levenshtein similarity between two terms.
-
- Return the Levenshtein similarity between two terms. The similarity is a
- number between <0.0, 1.0>, higher is more similar.
-
- Parameters
- ----------
- t1 : {bytes, str, unicode}
- The first compared term.
- t2 : {bytes, str, unicode}
- The second compared term.
- alpha : float, optional
- The multiplicative factor alpha defined by Charlet and Damnati (2017).
- beta : float, optional
- The exponential factor beta defined by Charlet and Damnati (2017).
- min_similarity : {int, float}, optional
- If you don't care about similarities smaller than a known threshold, a
- more efficient code path can be taken. For terms that are clearly "too
- far apart", we will not compute the distance exactly, but we will
- return zero more quickly, meaning "less than `min_similarity`".
- Default: always compute similarity exactly, no threshold clipping.
-
- Returns
- -------
- float
- The Levenshtein similarity between `t1` and `t2`.
-
- Notes
- -----
- This notion of Levenshtein similarity was first defined in section 2.2 of
- `Delphine Charlet and Geraldine Damnati, "SimBow at SemEval-2017 Task 3:
- Soft-Cosine Semantic Similarity between Questions for Community Question
- Answering", 2017 `_.
-
- """
- assert alpha >= 0
- assert beta >= 0
-
- max_lengths = max(len(t1), len(t2))
- if max_lengths == 0:
- return 1.0
- min_similarity = float(max(min(min_similarity, 1.0), 0.0))
- max_distance = int(floor(max_lengths * (1 - (min_similarity / alpha) ** (1 / beta))))
- distance = levdist(t1, t2, max_distance)
- similarity = alpha * (1 - distance * 1.0 / max_lengths)**beta
- return similarity
+logger = logging.getLogger(__name__)
class LevenshteinSimilarityIndex(TermSimilarityIndex):
- """
- Computes Levenshtein similarities between terms and retrieves most similar
- terms for a given term.
+ r"""
+ Retrieve the most similar terms from a static set of terms ("dictionary")
+ given a query term, using Levenshtein similarity.
- Notes
- -----
- This is a naive implementation that iteratively computes pointwise Levenshtein similarities
- between individual terms. Using this implementation to compute the similarity of all terms in
- real-world dictionaries such as the English Wikipedia will take years.
+ "Levenshtein similarity" is a modification of the Levenshtein (edit) distance,
+ defined in [charletetal17]_.
+
+ This implementation uses the FastSS neighbourhood algorithm
+ for fast kNN nearest-neighbor retrieval.
Parameters
----------
dictionary : :class:`~gensim.corpora.dictionary.Dictionary`
A dictionary that specifies the considered terms.
alpha : float, optional
- The multiplicative factor alpha defined by Charlet and Damnati (2017).
+ Multiplicative factor `alpha` for the Levenshtein similarity. See [charletetal17]_.
beta : float, optional
- The exponential factor beta defined by Charlet and Damnati (2017).
- threshold : float, optional
- Only terms more similar than `threshold` are considered when retrieving
- the most similar terms for a given term.
+ The exponential factor `beta` for the Levenshtein similarity. See [charletetal17]_.
+ max_distance : int, optional
+ Do not consider terms with Levenshtein distance larger than this as
+ "similar". This is done for performance reasons: keep this value below 3
+ for reasonable retrieval performance. Default is 1.
See Also
--------
- :func:`gensim.similarities.levenshtein.levsim`
- The Levenshtein similarity.
+ :class:`~gensim.similarities.termsim.WordEmbeddingSimilarityIndex`
+ Retrieve most similar terms for a given term using the cosine
+ similarity over word embeddings.
:class:`~gensim.similarities.termsim.SparseTermSimilarityMatrix`
Build a term similarity matrix and compute the Soft Cosine Measure.
+ References
+ ----------
+
+ .. [charletetal17] Delphine Charlet and Geraldine Damnati, "SimBow at SemEval-2017 Task 3:
+ Soft-Cosine Semantic Similarity between Questions for Community Question Answering", 2017,
+ https://www.aclweb.org/anthology/S17-2051/.
+
"""
- def __init__(self, dictionary, alpha=1.8, beta=5.0, threshold=0.0):
+ def __init__(self, dictionary, alpha=1.8, beta=5.0, max_distance=2):
self.dictionary = dictionary
self.alpha = alpha
self.beta = beta
- self.threshold = threshold
+ self.max_distance = max_distance
+ logger.info("creating FastSS index from %s", dictionary)
+ self.index = FastSS(words=self.dictionary.values(), max_dist=max_distance)
super(LevenshteinSimilarityIndex, self).__init__()
+ def levsim(self, t1, t2, distance):
+ """Calculate the Levenshtein similarity between two terms given their Levenshtein distance."""
+ max_lengths = max(len(t1), len(t2)) or 1
+ return self.alpha * (1.0 - distance * 1.0 / max_lengths)**self.beta
+
def most_similar(self, t1, topn=10):
- similarities = (
- (levsim(t1, t2, self.alpha, self.beta, self.threshold), t2)
- for t2 in self.dictionary.values()
- if t1 != t2
- )
- most_similar = (
- (t2, similarity)
- for (similarity, t2) in sorted(similarities, reverse=True)
- if similarity > 0
- )
- return itertools.islice(most_similar, int(topn))
+ """kNN fuzzy search: find the `topn` most similar terms from `self.dictionary` to `t1`."""
+ result = {} # map of {dictionary term => its levenshtein similarity to t1}
+ if self.max_distance > 0:
+ effective_topn = topn + 1 if t1 in self.dictionary.token2id else topn
+ effective_topn = min(len(self.dictionary), effective_topn)
+
+ # Implement a "distance backoff" algorithm:
+ # Start with max_distance=1, for performance. And if that doesn't return enough results,
+ # continue with max_distance=2 etc, all the way until self.max_distance which
+ # is a hard cutoff.
+ # At that point stop searching, even if we don't have topn results yet.
+ #
+ # We use the backoff algo to speed up queries for short terms. These return enough results already
+ # with max_distance=1.
+ #
+ # See the discussion at https://github.com/RaRe-Technologies/gensim/pull/3146
+ for distance in range(1, self.max_distance + 1):
+ for t2 in self.index.query(t1, distance).get(distance, []):
+ if t1 == t2:
+ continue
+ similarity = self.levsim(t1, t2, distance)
+ if similarity > 0:
+ result[t2] = similarity
+ if len(result) >= effective_topn:
+ break
+
+ return sorted(result.items(), key=lambda x: (-x[1], x[0]))[:topn]
diff --git a/gensim/similarities/termsim.py b/gensim/similarities/termsim.py
index 8f39e9b36c..d2a3f6728f 100644
--- a/gensim/similarities/termsim.py
+++ b/gensim/similarities/termsim.py
@@ -99,10 +99,30 @@ def most_similar(self, t1, topn=10):
class WordEmbeddingSimilarityIndex(TermSimilarityIndex):
"""
- Use objects of this class to:
+ Computes cosine similarities between word embeddings and retrieves most
+ similar terms for a given term.
- 1) Compute cosine similarities between word embeddings.
- 2) Retrieve the closest word embeddings (by cosine similarity) to a given word embedding.
+ Notes
+ -----
+ By fitting the word embeddings to a vocabulary that you will be using, you
+ can eliminate all out-of-vocabulary (OOV) words that you would otherwise
+ receive from the `most_similar` method. In subword models such as fastText,
+ this procedure will also infer word-vectors for words from your vocabulary
+ that previously had no word-vector.
+
+ >>> from gensim.test.utils import common_texts, datapath
+ >>> from gensim.corpora import Dictionary
+ >>> from gensim.models import FastText
+ >>> from gensim.models.word2vec import LineSentence
+ >>> from gensim.similarities import WordEmbeddingSimilarityIndex
+ >>>
+ >>> model = FastText(common_texts, vector_size=20, min_count=1) # train word-vectors on a corpus
+ >>> different_corpus = LineSentence(datapath('lee_background.cor'))
+ >>> dictionary = Dictionary(different_corpus) # construct a vocabulary on a different corpus
+ >>> words = [word for word, count in dictionary.most_common()]
+ >>> word_vectors = model.wv.vectors_for_all(words) # remove OOV word-vectors and infer word-vectors for new words
+ >>> assert len(dictionary) == len(word_vectors) # all words from our vocabulary received their word-vectors
+ >>> termsim_index = WordEmbeddingSimilarityIndex(word_vectors)
Parameters
----------
@@ -114,13 +134,16 @@ class WordEmbeddingSimilarityIndex(TermSimilarityIndex):
exponent : float, optional
Take the word embedding similarities larger than `threshold` to the power of `exponent`.
kwargs : dict or None
- A dict with keyword arguments that will be passed to the `keyedvectors.most_similar` method
+ A dict with keyword arguments that will be passed to the
+ :meth:`~gensim.models.keyedvectors.KeyedVectors.most_similar` method
when retrieving the word embeddings closest to a given word embedding.
See Also
--------
+ :class:`~gensim.similarities.levenshtein.LevenshteinSimilarityIndex`
+ Retrieve most similar terms for a given term using the Levenshtein distance.
:class:`~gensim.similarities.termsim.SparseTermSimilarityMatrix`
- A sparse term similarity matrix built using a term similarity index.
+ Build a term similarity matrix and compute the Soft Cosine Measure.
"""
def __init__(self, keyedvectors, threshold=0.0, exponent=2.0, kwargs=None):
@@ -195,12 +218,12 @@ def tfidf_sort_key(term_index):
return (-term_idf, term_index)
if tfidf is None:
- logger.info("iterating over columns in dictionary order")
columns = sorted(dictionary.keys())
+ logger.info("iterating over %i columns in dictionary order", len(columns))
else:
assert max(tfidf.idfs) == matrix_order - 1
- logger.info("iterating over columns in tf-idf order")
columns = sorted(tfidf.idfs.keys(), key=tfidf_sort_key)
+ logger.info("iterating over %i columns in tf-idf order", len(columns))
nonzero_counter_dtype = _shortest_uint_dtype(nonzero_limit)
@@ -403,25 +426,29 @@ class SparseTermSimilarityMatrix(SaveLoad):
Examples
--------
- >>> from gensim.test.utils import common_texts
+ >>> from gensim.test.utils import common_texts as corpus, datapath
>>> from gensim.corpora import Dictionary
>>> from gensim.models import Word2Vec
>>> from gensim.similarities import SoftCosineSimilarity, SparseTermSimilarityMatrix, WordEmbeddingSimilarityIndex
>>> from gensim.similarities.index import AnnoyIndexer
- >>> from scikits.sparse.cholmod import cholesky
>>>
- >>> model = Word2Vec(common_texts, vector_size=20, min_count=1) # train word-vectors
- >>> annoy = AnnoyIndexer(model, num_trees=2) # use annoy for faster word similarity lookups
- >>> termsim_index = WordEmbeddingSimilarityIndex(model.wv, kwargs={'indexer': annoy})
- >>> dictionary = Dictionary(common_texts)
- >>> bow_corpus = [dictionary.doc2bow(document) for document in common_texts]
- >>> similarity_matrix = SparseTermSimilarityMatrix(termsim_index, dictionary, symmetric=True, dominant=True)
- >>> docsim_index = SoftCosineSimilarity(bow_corpus, similarity_matrix, num_best=10)
+ >>> model_corpus_file = datapath('lee_background.cor')
+ >>> model = Word2Vec(corpus_file=model_corpus_file, vector_size=20, min_count=1) # train word-vectors
>>>
- >>> query = 'graph trees computer'.split() # make a query
- >>> sims = docsim_index[dictionary.doc2bow(query)] # calculate similarity of query to each doc from bow_corpus
+ >>> dictionary = Dictionary(corpus)
+ >>> tfidf = TfidfModel(dictionary=dictionary)
+ >>> words = [word for word, count in dictionary.most_common()]
+ >>> word_vectors = model.wv.vectors_for_all(words, allow_inference=False) # produce vectors for words in corpus
+ >>>
+ >>> indexer = AnnoyIndexer(word_vectors, num_trees=2) # use Annoy for faster word similarity lookups
+ >>> termsim_index = WordEmbeddingSimilarityIndex(word_vectors, kwargs={'indexer': indexer})
+ >>> similarity_matrix = SparseTermSimilarityMatrix(termsim_index, dictionary, tfidf) # compute word similarities
>>>
- >>> word_embeddings = cholesky(similarity_matrix.matrix).L() # obtain word embeddings from similarity matrix
+ >>> tfidf_corpus = tfidf[[dictionary.doc2bow(document) for document in common_texts]]
+ >>> docsim_index = SoftCosineSimilarity(tfidf_corpus, similarity_matrix, num_best=10) # index tfidf_corpus
+ >>>
+ >>> query = 'graph trees computer'.split() # make a query
+ >>> sims = docsim_index[dictionary.doc2bow(query)] # find the ten closest documents from tfidf_corpus
Check out `the Gallery `_
for more examples.
diff --git a/gensim/test/test_coherencemodel.py b/gensim/test/test_coherencemodel.py
new file mode 100644
index 0000000000..9396fe5ac0
--- /dev/null
+++ b/gensim/test/test_coherencemodel.py
@@ -0,0 +1,300 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Copyright (C) 2010 Radim Rehurek
+# Licensed under the GNU LGPL v2.1 - http://www.gnu.org/licenses/lgpl.html
+
+"""
+Automated tests for checking transformation algorithms (the models package).
+"""
+
+import logging
+import unittest
+import multiprocessing as mp
+from functools import partial
+
+import numpy as np
+from gensim.matutils import argsort
+from gensim.models.coherencemodel import CoherenceModel, BOOLEAN_DOCUMENT_BASED
+from gensim.models.ldamodel import LdaModel
+from gensim.test.utils import get_tmpfile, common_texts, common_dictionary, common_corpus
+
+
+class TestCoherenceModel(unittest.TestCase):
+
+ # set up vars used in testing ("Deerwester" from the web tutorial)
+ texts = common_texts
+ dictionary = common_dictionary
+ corpus = common_corpus
+
+ def setUp(self):
+ # Suppose given below are the topics which two different LdaModels come up with.
+ # `topics1` is clearly better as it has a clear distinction between system-human
+ # interaction and graphs. Hence both the coherence measures for `topics1` should be
+ # greater.
+ self.topics1 = [
+ ['human', 'computer', 'system', 'interface'],
+ ['graph', 'minors', 'trees', 'eps']
+ ]
+ self.topics2 = [
+ ['user', 'graph', 'minors', 'system'],
+ ['time', 'graph', 'survey', 'minors']
+ ]
+ self.topics3 = [
+ ['token', 'computer', 'system', 'interface'],
+ ['graph', 'minors', 'trees', 'eps']
+ ]
+ # using this list the model should be unable to interpret topic
+ # as either a list of tokens or a list of ids
+ self.topics4 = [
+ ['not a token', 'not an id', 'tests using', "this list"],
+ ['should raise', 'an error', 'to pass', 'correctly']
+ ]
+ self.topicIds1 = []
+ for topic in self.topics1:
+ self.topicIds1.append([self.dictionary.token2id[token] for token in topic])
+
+ self.ldamodel = LdaModel(
+ corpus=self.corpus, id2word=self.dictionary, num_topics=2,
+ passes=0, iterations=0
+ )
+
+ def check_coherence_measure(self, coherence):
+ """Check provided topic coherence algorithm on given topics"""
+ if coherence in BOOLEAN_DOCUMENT_BASED:
+ kwargs = dict(corpus=self.corpus, dictionary=self.dictionary, coherence=coherence)
+ else:
+ kwargs = dict(texts=self.texts, dictionary=self.dictionary, coherence=coherence)
+
+ cm1 = CoherenceModel(topics=self.topics1, **kwargs)
+ cm2 = CoherenceModel(topics=self.topics2, **kwargs)
+ cm3 = CoherenceModel(topics=self.topics3, **kwargs)
+ cm4 = CoherenceModel(topics=self.topicIds1, **kwargs)
+ self.assertRaises(ValueError, lambda: CoherenceModel(topics=self.topics4, **kwargs))
+ self.assertEqual(cm1.get_coherence(), cm4.get_coherence())
+ self.assertIsInstance(cm3.get_coherence(), np.double)
+ self.assertGreater(cm1.get_coherence(), cm2.get_coherence())
+
+ def testUMass(self):
+ """Test U_Mass topic coherence algorithm on given topics"""
+ self.check_coherence_measure('u_mass')
+
+ def testCv(self):
+ """Test C_v topic coherence algorithm on given topics"""
+ self.check_coherence_measure('c_v')
+
+ def testCuci(self):
+ """Test C_uci topic coherence algorithm on given topics"""
+ self.check_coherence_measure('c_uci')
+
+ def testCnpmi(self):
+ """Test C_npmi topic coherence algorithm on given topics"""
+ self.check_coherence_measure('c_npmi')
+
+ def testUMassLdaModel(self):
+ """Perform sanity check to see if u_mass coherence works with LDA Model"""
+ # Note that this is just a sanity check because LDA does not guarantee a better coherence
+ # value on the topics if iterations are increased. This can be seen here:
+ # https://gist.github.com/dsquareindia/60fd9ab65b673711c3fa00509287ddde
+ CoherenceModel(model=self.ldamodel, corpus=self.corpus, coherence='u_mass')
+
+ def testCvLdaModel(self):
+ """Perform sanity check to see if c_v coherence works with LDA Model"""
+ CoherenceModel(model=self.ldamodel, texts=self.texts, coherence='c_v')
+
+ def testCw2vLdaModel(self):
+ """Perform sanity check to see if c_w2v coherence works with LDAModel."""
+ CoherenceModel(model=self.ldamodel, texts=self.texts, coherence='c_w2v')
+
+ def testCuciLdaModel(self):
+ """Perform sanity check to see if c_uci coherence works with LDA Model"""
+ CoherenceModel(model=self.ldamodel, texts=self.texts, coherence='c_uci')
+
+ def testCnpmiLdaModel(self):
+ """Perform sanity check to see if c_npmi coherence works with LDA Model"""
+ CoherenceModel(model=self.ldamodel, texts=self.texts, coherence='c_npmi')
+
+ def testErrors(self):
+ """Test if errors are raised on bad input"""
+ # not providing dictionary
+ self.assertRaises(
+ ValueError, CoherenceModel, topics=self.topics1, corpus=self.corpus,
+ coherence='u_mass'
+ )
+ # not providing texts for c_v and instead providing corpus
+ self.assertRaises(
+ ValueError, CoherenceModel, topics=self.topics1, corpus=self.corpus,
+ dictionary=self.dictionary, coherence='c_v'
+ )
+ # not providing corpus or texts for u_mass
+ self.assertRaises(
+ ValueError, CoherenceModel, topics=self.topics1, dictionary=self.dictionary,
+ coherence='u_mass'
+ )
+
+ def testProcesses(self):
+ get_model = partial(CoherenceModel,
+ topics=self.topics1, corpus=self.corpus, dictionary=self.dictionary, coherence='u_mass'
+ )
+
+ model, used_cpus = get_model(), mp.cpu_count() - 1
+ self.assertEqual(model.processes, used_cpus)
+ for p in range(-2, 1):
+ self.assertEqual(get_model(processes=p).processes, used_cpus)
+
+ for p in range(1, 4):
+ self.assertEqual(get_model(processes=p).processes, p)
+
+ def testPersistence(self):
+ fname = get_tmpfile('gensim_models_coherence.tst')
+ model = CoherenceModel(
+ topics=self.topics1, corpus=self.corpus, dictionary=self.dictionary, coherence='u_mass'
+ )
+ model.save(fname)
+ model2 = CoherenceModel.load(fname)
+ self.assertTrue(model.get_coherence() == model2.get_coherence())
+
+ def testPersistenceCompressed(self):
+ fname = get_tmpfile('gensim_models_coherence.tst.gz')
+ model = CoherenceModel(
+ topics=self.topics1, corpus=self.corpus, dictionary=self.dictionary, coherence='u_mass'
+ )
+ model.save(fname)
+ model2 = CoherenceModel.load(fname)
+ self.assertTrue(model.get_coherence() == model2.get_coherence())
+
+ def testPersistenceAfterProbabilityEstimationUsingCorpus(self):
+ fname = get_tmpfile('gensim_similarities.tst.pkl')
+ model = CoherenceModel(
+ topics=self.topics1, corpus=self.corpus, dictionary=self.dictionary, coherence='u_mass'
+ )
+ model.estimate_probabilities()
+ model.save(fname)
+ model2 = CoherenceModel.load(fname)
+ self.assertIsNotNone(model2._accumulator)
+ self.assertTrue(model.get_coherence() == model2.get_coherence())
+
+ def testPersistenceAfterProbabilityEstimationUsingTexts(self):
+ fname = get_tmpfile('gensim_similarities.tst.pkl')
+ model = CoherenceModel(
+ topics=self.topics1, texts=self.texts, dictionary=self.dictionary, coherence='c_v'
+ )
+ model.estimate_probabilities()
+ model.save(fname)
+ model2 = CoherenceModel.load(fname)
+ self.assertIsNotNone(model2._accumulator)
+ self.assertTrue(model.get_coherence() == model2.get_coherence())
+
+ def testAccumulatorCachingSameSizeTopics(self):
+ kwargs = dict(corpus=self.corpus, dictionary=self.dictionary, coherence='u_mass')
+ cm1 = CoherenceModel(topics=self.topics1, **kwargs)
+ cm1.estimate_probabilities()
+ accumulator = cm1._accumulator
+ self.assertIsNotNone(accumulator)
+ cm1.topics = self.topics1
+ self.assertEqual(accumulator, cm1._accumulator)
+ cm1.topics = self.topics2
+ self.assertEqual(None, cm1._accumulator)
+
+ def testAccumulatorCachingTopicSubsets(self):
+ kwargs = dict(corpus=self.corpus, dictionary=self.dictionary, coherence='u_mass')
+ cm1 = CoherenceModel(topics=self.topics1, **kwargs)
+ cm1.estimate_probabilities()
+ accumulator = cm1._accumulator
+ self.assertIsNotNone(accumulator)
+ cm1.topics = [t[:2] for t in self.topics1]
+ self.assertEqual(accumulator, cm1._accumulator)
+ cm1.topics = self.topics1
+ self.assertEqual(accumulator, cm1._accumulator)
+
+ def testAccumulatorCachingWithModelSetting(self):
+ kwargs = dict(corpus=self.corpus, dictionary=self.dictionary, coherence='u_mass')
+ cm1 = CoherenceModel(topics=self.topics1, **kwargs)
+ cm1.estimate_probabilities()
+ self.assertIsNotNone(cm1._accumulator)
+ cm1.model = self.ldamodel
+ topics = []
+ for topic in self.ldamodel.state.get_lambda():
+ bestn = argsort(topic, topn=cm1.topn, reverse=True)
+ topics.append(bestn)
+ self.assertTrue(np.array_equal(topics, cm1.topics))
+ self.assertIsNone(cm1._accumulator)
+
+ def testAccumulatorCachingWithTopnSettingGivenTopics(self):
+ kwargs = dict(corpus=self.corpus, dictionary=self.dictionary, topn=5, coherence='u_mass')
+ cm1 = CoherenceModel(topics=self.topics1, **kwargs)
+ cm1.estimate_probabilities()
+ self.assertIsNotNone(cm1._accumulator)
+
+ accumulator = cm1._accumulator
+ topics_before = cm1._topics
+ cm1.topn = 3
+ self.assertEqual(accumulator, cm1._accumulator)
+ self.assertEqual(3, len(cm1.topics[0]))
+ self.assertEqual(topics_before, cm1._topics)
+
+ # Topics should not have been truncated, so topn settings below 5 should work
+ cm1.topn = 4
+ self.assertEqual(accumulator, cm1._accumulator)
+ self.assertEqual(4, len(cm1.topics[0]))
+ self.assertEqual(topics_before, cm1._topics)
+
+ with self.assertRaises(ValueError):
+ cm1.topn = 6 # can't expand topics any further without model
+
+ def testAccumulatorCachingWithTopnSettingGivenModel(self):
+ kwargs = dict(corpus=self.corpus, dictionary=self.dictionary, topn=5, coherence='u_mass')
+ cm1 = CoherenceModel(model=self.ldamodel, **kwargs)
+ cm1.estimate_probabilities()
+ self.assertIsNotNone(cm1._accumulator)
+
+ accumulator = cm1._accumulator
+ topics_before = cm1._topics
+ cm1.topn = 3
+ self.assertEqual(accumulator, cm1._accumulator)
+ self.assertEqual(3, len(cm1.topics[0]))
+ self.assertEqual(topics_before, cm1._topics)
+
+ cm1.topn = 6 # should be able to expand given the model
+ self.assertEqual(6, len(cm1.topics[0]))
+
+ def testCompareCoherenceForTopics(self):
+ topics = [self.topics1, self.topics2]
+ cm = CoherenceModel.for_topics(
+ topics, dictionary=self.dictionary, texts=self.texts, coherence='c_v')
+ self.assertIsNotNone(cm._accumulator)
+
+ # Accumulator should have all relevant IDs.
+ for topic_list in topics:
+ cm.topics = topic_list
+ self.assertIsNotNone(cm._accumulator)
+
+ (coherence_topics1, coherence1), (coherence_topics2, coherence2) = \
+ cm.compare_model_topics(topics)
+
+ self.assertAlmostEqual(np.mean(coherence_topics1), coherence1, 4)
+ self.assertAlmostEqual(np.mean(coherence_topics2), coherence2, 4)
+ self.assertGreater(coherence1, coherence2)
+
+ def testCompareCoherenceForModels(self):
+ models = [self.ldamodel, self.ldamodel]
+ cm = CoherenceModel.for_models(
+ models, dictionary=self.dictionary, texts=self.texts, coherence='c_v')
+ self.assertIsNotNone(cm._accumulator)
+
+ # Accumulator should have all relevant IDs.
+ for model in models:
+ cm.model = model
+ self.assertIsNotNone(cm._accumulator)
+
+ (coherence_topics1, coherence1), (coherence_topics2, coherence2) = \
+ cm.compare_models(models)
+
+ self.assertAlmostEqual(np.mean(coherence_topics1), coherence1, 4)
+ self.assertAlmostEqual(np.mean(coherence_topics2), coherence2, 4)
+ self.assertAlmostEqual(coherence1, coherence2, places=4)
+
+
+if __name__ == '__main__':
+ logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)
+ unittest.main()
diff --git a/gensim/test/test_corpora_dictionary.py b/gensim/test/test_corpora_dictionary.py
index 7d6659e896..b1f4d4f33f 100644
--- a/gensim/test/test_corpora_dictionary.py
+++ b/gensim/test/test_corpora_dictionary.py
@@ -359,6 +359,18 @@ def test_patch_with_special_tokens(self):
self.assertNotIn((1, 1), d.doc2bow(corpus_with_special_tokens[0]))
self.assertIn((1, 1), d.doc2bow(corpus_with_special_tokens[1]))
+ def test_most_common_with_n(self):
+ texts = [['human', 'human', 'human', 'computer', 'computer', 'interface', 'interface']]
+ d = Dictionary(texts)
+ expected = [('human', 3), ('computer', 2)]
+ assert d.most_common(n=2) == expected
+
+ def test_most_common_without_n(self):
+ texts = [['human', 'human', 'human', 'computer', 'computer', 'interface', 'interface']]
+ d = Dictionary(texts)
+ expected = [('human', 3), ('computer', 2), ('interface', 2)]
+ assert d.most_common(n=None) == expected
+
# endclass TestDictionary
diff --git a/gensim/test/test_data/ensemblelda b/gensim/test/test_data/ensemblelda
new file mode 100644
index 0000000000..cf396af29c
Binary files /dev/null and b/gensim/test/test_data/ensemblelda differ
diff --git a/gensim/test/test_doc2vec.py b/gensim/test/test_doc2vec.py
index 60c9158744..c8b7516c99 100644
--- a/gensim/test/test_doc2vec.py
+++ b/gensim/test/test_doc2vec.py
@@ -589,6 +589,44 @@ def test_dmc_neg_fromfile(self):
)
self.model_sanity(model)
+ def test_dmm_fixedwindowsize(self):
+ """Test DMM doc2vec training with fixed window size."""
+ model = doc2vec.Doc2Vec(
+ list_corpus, vector_size=24,
+ dm=1, dm_mean=1, window=4, shrink_windows=False,
+ hs=0, negative=10, alpha=0.05, min_count=2, epochs=20
+ )
+ self.model_sanity(model)
+
+ def test_dmm_fixedwindowsize_fromfile(self):
+ """Test DMM doc2vec training with fixed window size, from file."""
+ with temporary_file(get_tmpfile('gensim_doc2vec.tst')) as corpus_file:
+ save_lee_corpus_as_line_sentence(corpus_file)
+ model = doc2vec.Doc2Vec(
+ corpus_file=corpus_file, vector_size=24,
+ dm=1, dm_mean=1, window=4, shrink_windows=False,
+ hs=0, negative=10, alpha=0.05, min_count=2, epochs=20
+ )
+ self.model_sanity(model)
+
+ def test_dbow_fixedwindowsize(self):
+ """Test DBOW doc2vec training with fixed window size."""
+ model = doc2vec.Doc2Vec(
+ list_corpus, vector_size=16, shrink_windows=False,
+ dm=0, hs=0, negative=5, min_count=2, epochs=20
+ )
+ self.model_sanity(model)
+
+ def test_dbow_fixedwindowsize_fromfile(self):
+ """Test DBOW doc2vec training with fixed window size, from file."""
+ with temporary_file(get_tmpfile('gensim_doc2vec.tst')) as corpus_file:
+ save_lee_corpus_as_line_sentence(corpus_file)
+ model = doc2vec.Doc2Vec(
+ corpus_file=corpus_file, vector_size=16, shrink_windows=False,
+ dm=0, hs=0, negative=5, min_count=2, epochs=20
+ )
+ self.model_sanity(model)
+
def test_parallel(self):
"""Test doc2vec parallel training with more than default 3 threads."""
# repeat the ~300 doc (~60000 word) Lee corpus to get 6000 docs (~1.2M words)
@@ -718,8 +756,8 @@ def __str__(self):
def epochs(self):
return self.models[0].epochs
- def infer_vector(self, document, alpha=None, min_alpha=None, epochs=None, steps=None):
- return np.concatenate([model.infer_vector(document, alpha, min_alpha, epochs, steps) for model in self.models])
+ def infer_vector(self, document, alpha=None, min_alpha=None, epochs=None):
+ return np.concatenate([model.infer_vector(document, alpha, min_alpha, epochs) for model in self.models])
def train(self, *ignore_args, **ignore_kwargs):
pass # train subcomponents individually
diff --git a/gensim/test/test_ensemblelda.py b/gensim/test/test_ensemblelda.py
new file mode 100644
index 0000000000..ad574108f9
--- /dev/null
+++ b/gensim/test/test_ensemblelda.py
@@ -0,0 +1,448 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+#
+# Author: Tobias B
+
+"""
+Automated tests for checking the EnsembleLda Class
+"""
+
+import os
+import logging
+import unittest
+
+import numpy as np
+from copy import deepcopy
+
+import pytest
+
+from gensim.models import EnsembleLda, LdaMulticore, LdaModel
+from gensim.test.utils import datapath, get_tmpfile, common_corpus, common_dictionary
+
+NUM_TOPICS = 2
+NUM_MODELS = 4
+PASSES = 50
+RANDOM_STATE = 0
+
+# windows tests fail due to the required assertion precision being too high
+RTOL = 1e-04 if os.name == 'nt' else 1e-05
+
+
+class TestEnsembleLda(unittest.TestCase):
+ def get_elda(self):
+ return EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary, num_topics=NUM_TOPICS,
+ passes=PASSES, num_models=NUM_MODELS, random_state=RANDOM_STATE,
+ topic_model_class=LdaModel,
+ )
+
+ def get_elda_mem_unfriendly(self):
+ return EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary, num_topics=NUM_TOPICS,
+ passes=PASSES, num_models=NUM_MODELS, random_state=RANDOM_STATE,
+ memory_friendly_ttda=False, topic_model_class=LdaModel,
+ )
+
+ def assert_ttda_is_valid(self, elda):
+ """Check that ttda has one or more topic and that term probabilities add to one."""
+ assert len(elda.ttda) > 0
+ sum_over_terms = elda.ttda.sum(axis=1)
+ expected_sum_over_terms = np.ones(len(elda.ttda)).astype(np.float32)
+ np.testing.assert_allclose(sum_over_terms, expected_sum_over_terms, rtol=1e-04)
+
+ def test_elda(self):
+ elda = self.get_elda()
+
+ # given that the random_state doesn't change, it should
+ # always be 2 detected topics in this setup.
+ assert elda.stable_topics.shape[1] == len(common_dictionary)
+ assert len(elda.ttda) == NUM_MODELS * NUM_TOPICS
+ self.assert_ttda_is_valid(elda)
+
+ def test_backwards_compatibility_with_persisted_model(self):
+ elda = self.get_elda()
+
+ # compare with a pre-trained reference model
+ loaded_elda = EnsembleLda.load(datapath('ensemblelda'))
+ np.testing.assert_allclose(elda.ttda, loaded_elda.ttda, rtol=RTOL)
+ atol = loaded_elda.asymmetric_distance_matrix.max() * 1e-05
+ np.testing.assert_allclose(
+ elda.asymmetric_distance_matrix,
+ loaded_elda.asymmetric_distance_matrix, atol=atol,
+ )
+
+ def test_recluster(self):
+ # the following test is quite specific to the current implementation and not part of any api,
+ # but it makes improving those sections of the code easier as long as sorted_clusters and the
+ # cluster_model results are supposed to stay the same. Potentially this test will deprecate.
+
+ elda = EnsembleLda.load(datapath('ensemblelda'))
+ loaded_cluster_model_results = deepcopy(elda.cluster_model.results)
+ loaded_valid_clusters = deepcopy(elda.valid_clusters)
+ loaded_stable_topics = deepcopy(elda.get_topics())
+
+ # continue training with the distance matrix of the pretrained reference and see if
+ # the generated clusters match.
+ elda.asymmetric_distance_matrix_outdated = True
+ elda.recluster()
+
+ self.assert_clustering_results_equal(elda.cluster_model.results, loaded_cluster_model_results)
+ assert elda.valid_clusters == loaded_valid_clusters
+ np.testing.assert_allclose(elda.get_topics(), loaded_stable_topics, rtol=RTOL)
+
+ def test_recluster_does_nothing_when_stable_topics_already_found(self):
+ elda = self.get_elda()
+
+ # reclustering shouldn't change anything without
+ # added models or different parameters
+ elda.recluster()
+
+ assert elda.stable_topics.shape[1] == len(common_dictionary)
+ assert len(elda.ttda) == NUM_MODELS * NUM_TOPICS
+ self.assert_ttda_is_valid(elda)
+
+ def test_not_trained_given_zero_passes(self):
+ elda = EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary, num_topics=NUM_TOPICS,
+ passes=0, num_models=NUM_MODELS, random_state=RANDOM_STATE,
+ )
+ assert len(elda.ttda) == 0
+
+ def test_not_trained_given_no_corpus(self):
+ elda = EnsembleLda(
+ id2word=common_dictionary, num_topics=NUM_TOPICS,
+ passes=PASSES, num_models=NUM_MODELS, random_state=RANDOM_STATE,
+ )
+ assert len(elda.ttda) == 0
+
+ def test_not_trained_given_zero_iterations(self):
+ elda = EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary, num_topics=NUM_TOPICS,
+ iterations=0, num_models=NUM_MODELS, random_state=RANDOM_STATE,
+ )
+ assert len(elda.ttda) == 0
+
+ def test_not_trained_given_zero_models(self):
+ elda = EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary, num_topics=NUM_TOPICS,
+ passes=PASSES, num_models=0, random_state=RANDOM_STATE
+ )
+ assert len(elda.ttda) == 0
+
+ def test_mem_unfriendly(self):
+ # elda_mem_unfriendly and self.elda should have topics that are
+ # the same up to floating point variations caused by the two different
+ # implementations
+
+ elda = self.get_elda()
+ elda_mem_unfriendly = self.get_elda_mem_unfriendly()
+
+ assert len(elda_mem_unfriendly.tms) == NUM_MODELS
+ np.testing.assert_allclose(elda.ttda, elda_mem_unfriendly.ttda, rtol=RTOL)
+ np.testing.assert_allclose(elda.get_topics(), elda_mem_unfriendly.get_topics(), rtol=RTOL)
+ self.assert_ttda_is_valid(elda_mem_unfriendly)
+
+ def test_generate_gensim_representation(self):
+ elda = self.get_elda()
+
+ gensim_model = elda.generate_gensim_representation()
+ topics = gensim_model.get_topics()
+ np.testing.assert_allclose(elda.get_topics(), topics, rtol=RTOL)
+
+ def assert_clustering_results_equal(self, clustering_results_1, clustering_results_2):
+ """Assert important attributes of the cluster results"""
+ np.testing.assert_array_equal(
+ [element.label for element in clustering_results_1],
+ [element.label for element in clustering_results_2],
+ )
+ np.testing.assert_array_equal(
+ [element.is_core for element in clustering_results_1],
+ [element.is_core for element in clustering_results_2],
+ )
+
+ def test_persisting(self):
+ elda = self.get_elda()
+ elda_mem_unfriendly = self.get_elda_mem_unfriendly()
+
+ fname = get_tmpfile('gensim_models_ensemblelda')
+ elda.save(fname)
+ loaded_elda = EnsembleLda.load(fname)
+ # storing the ensemble without memory_friendy_ttda
+ elda_mem_unfriendly.save(fname)
+ loaded_elda_mem_unfriendly = EnsembleLda.load(fname)
+
+ # topic_model_class will be lazy loaded and should be None first
+ assert loaded_elda.topic_model_class is None
+
+ # was it stored and loaded correctly?
+ # memory friendly.
+ loaded_elda_representation = loaded_elda.generate_gensim_representation()
+
+ # generating the representation also lazily loads the topic_model_class
+ assert loaded_elda.topic_model_class == LdaModel
+
+ topics = loaded_elda_representation.get_topics()
+ ttda = loaded_elda.ttda
+ amatrix = loaded_elda.asymmetric_distance_matrix
+ np.testing.assert_allclose(elda.get_topics(), topics, rtol=RTOL)
+ np.testing.assert_allclose(elda.ttda, ttda, rtol=RTOL)
+ np.testing.assert_allclose(elda.asymmetric_distance_matrix, amatrix, rtol=RTOL)
+
+ expected_clustering_results = elda.cluster_model.results
+ loaded_clustering_results = loaded_elda.cluster_model.results
+
+ self.assert_clustering_results_equal(expected_clustering_results, loaded_clustering_results)
+
+ # memory unfriendly
+ loaded_elda_mem_unfriendly_representation = loaded_elda_mem_unfriendly.generate_gensim_representation()
+ topics = loaded_elda_mem_unfriendly_representation.get_topics()
+ np.testing.assert_allclose(elda.get_topics(), topics, rtol=RTOL)
+
+ def test_multiprocessing(self):
+ # same configuration
+ random_state = RANDOM_STATE
+
+ # use 3 processes for the ensemble and the distance,
+ # so that the 4 models and 8 topics cannot be distributed
+ # to each worker evenly
+ workers = 3
+
+ # memory friendly. contains List of topic word distributions
+ elda = self.get_elda()
+ elda_multiprocessing = EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary, topic_model_class=LdaModel,
+ num_topics=NUM_TOPICS, passes=PASSES, num_models=NUM_MODELS,
+ random_state=random_state, ensemble_workers=workers, distance_workers=workers,
+ )
+
+ # memory unfriendly. contains List of models
+ elda_mem_unfriendly = self.get_elda_mem_unfriendly()
+ elda_multiprocessing_mem_unfriendly = EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary, topic_model_class=LdaModel,
+ num_topics=NUM_TOPICS, passes=PASSES, num_models=NUM_MODELS,
+ random_state=random_state, ensemble_workers=workers, distance_workers=workers,
+ memory_friendly_ttda=False,
+ )
+
+ np.testing.assert_allclose(
+ elda.get_topics(),
+ elda_multiprocessing.get_topics(),
+ rtol=RTOL
+ )
+ np.testing.assert_allclose(
+ elda_mem_unfriendly.get_topics(),
+ elda_multiprocessing_mem_unfriendly.get_topics(),
+ rtol=RTOL
+ )
+
+ def test_add_models_to_empty(self):
+ elda = self.get_elda()
+
+ ensemble = EnsembleLda(id2word=common_dictionary, num_models=0)
+ ensemble.add_model(elda.ttda[0:1])
+ ensemble.add_model(elda.ttda[1:])
+ ensemble.recluster()
+ np.testing.assert_allclose(ensemble.get_topics(), elda.get_topics(), rtol=RTOL)
+
+ # persisting an ensemble that is entirely built from existing ttdas
+ fname = get_tmpfile('gensim_models_ensemblelda')
+ ensemble.save(fname)
+ loaded_ensemble = EnsembleLda.load(fname)
+ np.testing.assert_allclose(loaded_ensemble.get_topics(), elda.get_topics(), rtol=RTOL)
+ self.test_inference(loaded_ensemble)
+
+ def test_add_models(self):
+ # make sure countings and sizes after adding are correct
+ # create new models and add other models to them.
+
+ # there are a ton of configurations for the first parameter possible,
+ # try them all
+
+ # quickly train something that can be used for counting results
+ num_new_models = 3
+ num_new_topics = 3
+
+ # 1. memory friendly
+ base_elda = self.get_elda()
+ cumulative_elda = EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary,
+ num_topics=num_new_topics, passes=1, num_models=num_new_models,
+ iterations=1, random_state=RANDOM_STATE, topic_model_class=LdaMulticore,
+ workers=3, ensemble_workers=2,
+ )
+
+ # 1.1 ttda
+ num_topics_before_add_model = len(cumulative_elda.ttda)
+ num_models_before_add_model = cumulative_elda.num_models
+ cumulative_elda.add_model(base_elda.ttda)
+ assert len(cumulative_elda.ttda) == num_topics_before_add_model + len(base_elda.ttda)
+ assert cumulative_elda.num_models == num_models_before_add_model + 1 # defaults to 1 for one ttda matrix
+
+ # 1.2 an ensemble
+ num_topics_before_add_model = len(cumulative_elda.ttda)
+ num_models_before_add_model = cumulative_elda.num_models
+ cumulative_elda.add_model(base_elda, 5)
+ assert len(cumulative_elda.ttda) == num_topics_before_add_model + len(base_elda.ttda)
+ assert cumulative_elda.num_models == num_models_before_add_model + 5
+
+ # 1.3 a list of ensembles
+ num_topics_before_add_model = len(cumulative_elda.ttda)
+ num_models_before_add_model = cumulative_elda.num_models
+ # it should be totally legit to add a memory unfriendly object to a memory friendly one
+ base_elda_mem_unfriendly = self.get_elda_mem_unfriendly()
+ cumulative_elda.add_model([base_elda, base_elda_mem_unfriendly])
+ assert len(cumulative_elda.ttda) == num_topics_before_add_model + 2 * len(base_elda.ttda)
+ assert cumulative_elda.num_models == num_models_before_add_model + 2 * NUM_MODELS
+
+ # 1.4 a single gensim model
+ model = base_elda.classic_model_representation
+
+ num_topics_before_add_model = len(cumulative_elda.ttda)
+ num_models_before_add_model = cumulative_elda.num_models
+ cumulative_elda.add_model(model)
+ assert len(cumulative_elda.ttda) == num_topics_before_add_model + len(model.get_topics())
+ assert cumulative_elda.num_models == num_models_before_add_model + 1
+
+ # 1.5 a list gensim models
+ num_topics_before_add_model = len(cumulative_elda.ttda)
+ num_models_before_add_model = cumulative_elda.num_models
+ cumulative_elda.add_model([model, model])
+ assert len(cumulative_elda.ttda) == num_topics_before_add_model + 2 * len(model.get_topics())
+ assert cumulative_elda.num_models == num_models_before_add_model + 2
+
+ self.assert_ttda_is_valid(cumulative_elda)
+
+ # 2. memory unfriendly
+ elda_mem_unfriendly = EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary,
+ num_topics=num_new_topics, passes=1, num_models=num_new_models,
+ iterations=1, random_state=RANDOM_STATE, topic_model_class=LdaMulticore,
+ workers=3, ensemble_workers=2, memory_friendly_ttda=False,
+ )
+
+ # 2.1 a single ensemble
+ num_topics_before_add_model = len(elda_mem_unfriendly.tms)
+ num_models_before_add_model = elda_mem_unfriendly.num_models
+ elda_mem_unfriendly.add_model(base_elda_mem_unfriendly)
+ assert len(elda_mem_unfriendly.tms) == num_topics_before_add_model + NUM_MODELS
+ assert elda_mem_unfriendly.num_models == num_models_before_add_model + NUM_MODELS
+
+ # 2.2 a list of ensembles
+ num_topics_before_add_model = len(elda_mem_unfriendly.tms)
+ num_models_before_add_model = elda_mem_unfriendly.num_models
+ elda_mem_unfriendly.add_model([base_elda_mem_unfriendly, base_elda_mem_unfriendly])
+ assert len(elda_mem_unfriendly.tms) == num_topics_before_add_model + 2 * NUM_MODELS
+ assert elda_mem_unfriendly.num_models == num_models_before_add_model + 2 * NUM_MODELS
+
+ # 2.3 a single gensim model
+ num_topics_before_add_model = len(elda_mem_unfriendly.tms)
+ num_models_before_add_model = elda_mem_unfriendly.num_models
+ elda_mem_unfriendly.add_model(base_elda_mem_unfriendly.tms[0])
+ assert len(elda_mem_unfriendly.tms) == num_topics_before_add_model + 1
+ assert elda_mem_unfriendly.num_models == num_models_before_add_model + 1
+
+ # 2.4 a list of gensim models
+ num_topics_before_add_model = len(elda_mem_unfriendly.tms)
+ num_models_before_add_model = elda_mem_unfriendly.num_models
+ elda_mem_unfriendly.add_model(base_elda_mem_unfriendly.tms)
+ assert len(elda_mem_unfriendly.tms) == num_topics_before_add_model + NUM_MODELS
+ assert elda_mem_unfriendly.num_models == num_models_before_add_model + NUM_MODELS
+
+ # 2.5 topic term distributions should throw errors, because the
+ # actual models are needed for the memory unfriendly ensemble
+ num_topics_before_add_model = len(elda_mem_unfriendly.tms)
+ num_models_before_add_model = elda_mem_unfriendly.num_models
+ with pytest.raises(ValueError):
+ elda_mem_unfriendly.add_model(base_elda_mem_unfriendly.tms[0].get_topics())
+ # remains unchanged
+ assert len(elda_mem_unfriendly.tms) == num_topics_before_add_model
+ assert elda_mem_unfriendly.num_models == num_models_before_add_model
+
+ assert elda_mem_unfriendly.num_models == len(elda_mem_unfriendly.tms)
+ self.assert_ttda_is_valid(elda_mem_unfriendly)
+
+ def test_add_and_recluster(self):
+ # See if after adding a model, the model still makes sense
+ num_new_models = 3
+ num_new_topics = 3
+ random_state = 1
+
+ # train models two sets of models (mem friendly and unfriendly)
+ elda_1 = EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary,
+ num_topics=num_new_topics, passes=10, num_models=num_new_models,
+ iterations=30, random_state=random_state, topic_model_class='lda',
+ distance_workers=4,
+ )
+ elda_mem_unfriendly_1 = EnsembleLda(
+ corpus=common_corpus, id2word=common_dictionary,
+ num_topics=num_new_topics, passes=10, num_models=num_new_models,
+ iterations=30, random_state=random_state, topic_model_class=LdaModel,
+ distance_workers=4, memory_friendly_ttda=False,
+ )
+ elda_2 = self.get_elda()
+ elda_mem_unfriendly_2 = self.get_elda_mem_unfriendly()
+ assert elda_1.random_state != elda_2.random_state
+ assert elda_mem_unfriendly_1.random_state != elda_mem_unfriendly_2.random_state
+
+ # both should be similar
+ np.testing.assert_allclose(elda_1.ttda, elda_mem_unfriendly_1.ttda, rtol=RTOL)
+ np.testing.assert_allclose(elda_1.get_topics(), elda_mem_unfriendly_1.get_topics(), rtol=RTOL)
+ # and every next step applied to both should result in similar results
+
+ # 1. adding to ttda and tms
+ elda_1.add_model(elda_2)
+ elda_mem_unfriendly_1.add_model(elda_mem_unfriendly_2)
+
+ np.testing.assert_allclose(elda_1.ttda, elda_mem_unfriendly_1.ttda, rtol=RTOL)
+ assert len(elda_1.ttda) == len(elda_2.ttda) + num_new_models * num_new_topics
+ assert len(elda_mem_unfriendly_1.ttda) == len(elda_mem_unfriendly_2.ttda) + num_new_models * num_new_topics
+ assert len(elda_mem_unfriendly_1.tms) == NUM_MODELS + num_new_models
+ self.assert_ttda_is_valid(elda_1)
+ self.assert_ttda_is_valid(elda_mem_unfriendly_1)
+
+ # 2. distance matrix
+ elda_1._generate_asymmetric_distance_matrix()
+ elda_mem_unfriendly_1._generate_asymmetric_distance_matrix()
+ np.testing.assert_allclose(
+ elda_1.asymmetric_distance_matrix,
+ elda_mem_unfriendly_1.asymmetric_distance_matrix,
+ )
+
+ # 3. CBDBSCAN results
+ elda_1._generate_topic_clusters()
+ elda_mem_unfriendly_1._generate_topic_clusters()
+ clustering_results = elda_1.cluster_model.results
+ mem_unfriendly_clustering_results = elda_mem_unfriendly_1.cluster_model.results
+ self.assert_clustering_results_equal(clustering_results, mem_unfriendly_clustering_results)
+
+ # 4. finally, the stable topics
+ elda_1._generate_stable_topics()
+ elda_mem_unfriendly_1._generate_stable_topics()
+ np.testing.assert_allclose(
+ elda_1.get_topics(),
+ elda_mem_unfriendly_1.get_topics(),
+ )
+
+ elda_1.generate_gensim_representation()
+ elda_mem_unfriendly_1.generate_gensim_representation()
+
+ # same random state, hence topics should be still similar
+ np.testing.assert_allclose(elda_1.get_topics(), elda_mem_unfriendly_1.get_topics(), rtol=RTOL)
+
+ def test_inference(self, elda=None):
+ if elda is None:
+ elda = self.get_elda()
+
+ # get the most likely token id from topic 0
+ max_id = np.argmax(elda.get_topics()[0, :])
+ assert elda.classic_model_representation.iterations > 0
+ # topic 0 should be dominant in the inference.
+ # the difference between the probabilities should be significant and larger than 0.3
+ inferred = elda[[(max_id, 1)]]
+ assert inferred[0][1] - 0.3 > inferred[1][1]
+
+
+if __name__ == '__main__':
+ logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.WARN)
+ unittest.main()
diff --git a/gensim/test/test_fasttext.py b/gensim/test/test_fasttext.py
index a91a9000a7..efc6a3ca8e 100644
--- a/gensim/test/test_fasttext.py
+++ b/gensim/test/test_fasttext.py
@@ -13,6 +13,7 @@
import sys
import numpy as np
+import pytest
from gensim import utils
from gensim.models.word2vec import LineSentence
@@ -397,144 +398,7 @@ def test_wm_distance(self):
dist = self.test_model.wv.wmdistance(doc, oov_doc)
self.assertNotEqual(float('inf'), dist)
- def test_cbow_hs_training(self):
-
- model_gensim = FT_gensim(
- vector_size=48, sg=0, cbow_mean=1, alpha=0.05, window=5, hs=1, negative=0,
- min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
- sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET)
-
- lee_data = LineSentence(datapath('lee_background.cor'))
- model_gensim.build_vocab(lee_data)
- orig0 = np.copy(model_gensim.wv.vectors[0])
- model_gensim.train(lee_data, total_examples=model_gensim.corpus_count, epochs=model_gensim.epochs)
- self.assertFalse((orig0 == model_gensim.wv.vectors[0]).all()) # vector should vary after training
-
- sims_gensim = model_gensim.wv.most_similar('night', topn=10)
- sims_gensim_words = [word for (word, distance) in sims_gensim] # get similar words
- expected_sims_words = [
- u'night,',
- u'night.',
- u'rights',
- u'kilometres',
- u'in',
- u'eight',
- u'according',
- u'flights',
- u'during',
- u'comes']
- overlaps = set(sims_gensim_words).intersection(expected_sims_words)
- overlap_count = len(overlaps)
- self.assertGreaterEqual(
- overlap_count, 2,
- "only %i overlap in expected %s & actual %s" % (overlap_count, expected_sims_words, sims_gensim_words))
-
- def test_cbow_hs_training_fromfile(self):
- with temporary_file('gensim_fasttext.tst') as corpus_file:
- model_gensim = FT_gensim(
- vector_size=48, sg=0, cbow_mean=1, alpha=0.05, window=5, hs=1, negative=0,
- min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
- sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET * 4)
-
- lee_data = LineSentence(datapath('lee_background.cor'))
- utils.save_as_line_sentence(lee_data, corpus_file)
-
- model_gensim.build_vocab(corpus_file=corpus_file)
- orig0 = np.copy(model_gensim.wv.vectors[0])
- model_gensim.train(corpus_file=corpus_file,
- total_words=model_gensim.corpus_total_words,
- epochs=model_gensim.epochs)
- self.assertFalse((orig0 == model_gensim.wv.vectors[0]).all()) # vector should vary after training
-
- sims_gensim = model_gensim.wv.most_similar('night', topn=10)
- sims_gensim_words = [word for (word, distance) in sims_gensim] # get similar words
- expected_sims_words = [
- u'night,',
- u'night.',
- u'rights',
- u'kilometres',
- u'in',
- u'eight',
- u'according',
- u'flights',
- u'during',
- u'comes']
- overlaps = set(sims_gensim_words).intersection(expected_sims_words)
- overlap_count = len(overlaps)
- self.assertGreaterEqual(
- overlap_count, 2,
- "only %i overlap in expected %s & actual %s" % (overlap_count, expected_sims_words, sims_gensim_words))
-
- def test_sg_hs_training(self):
-
- model_gensim = FT_gensim(
- vector_size=48, sg=1, cbow_mean=1, alpha=0.025, window=5, hs=1, negative=0,
- min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
- sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET)
-
- lee_data = LineSentence(datapath('lee_background.cor'))
- model_gensim.build_vocab(lee_data)
- orig0 = np.copy(model_gensim.wv.vectors[0])
- model_gensim.train(lee_data, total_examples=model_gensim.corpus_count, epochs=model_gensim.epochs)
- self.assertFalse((orig0 == model_gensim.wv.vectors[0]).all()) # vector should vary after training
-
- sims_gensim = model_gensim.wv.most_similar('night', topn=10)
- sims_gensim_words = [word for (word, distance) in sims_gensim] # get similar words
- expected_sims_words = [
- u'night,',
- u'night.',
- u'eight',
- u'nine',
- u'overnight',
- u'crew',
- u'overnight.',
- u'manslaughter',
- u'north',
- u'flight']
- overlaps = set(sims_gensim_words).intersection(expected_sims_words)
- overlap_count = len(overlaps)
- self.assertGreaterEqual(
- overlap_count, 2,
- "only %i overlap in expected %s & actual %s" % (overlap_count, expected_sims_words, sims_gensim_words))
-
- def test_sg_hs_training_fromfile(self):
- with temporary_file('gensim_fasttext.tst') as corpus_file:
- model_gensim = FT_gensim(
- vector_size=48, sg=1, cbow_mean=1, alpha=0.025, window=5, hs=1, negative=0,
- min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
- sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET)
-
- lee_data = LineSentence(datapath('lee_background.cor'))
- utils.save_as_line_sentence(lee_data, corpus_file)
-
- model_gensim.build_vocab(corpus_file=corpus_file)
- orig0 = np.copy(model_gensim.wv.vectors[0])
- model_gensim.train(corpus_file=corpus_file,
- total_words=model_gensim.corpus_total_words,
- epochs=model_gensim.epochs)
- self.assertFalse((orig0 == model_gensim.wv.vectors[0]).all()) # vector should vary after training
-
- sims_gensim = model_gensim.wv.most_similar('night', topn=10)
- sims_gensim_words = [word for (word, distance) in sims_gensim] # get similar words
- expected_sims_words = [
- u'night,',
- u'night.',
- u'eight',
- u'nine',
- u'overnight',
- u'crew',
- u'overnight.',
- u'manslaughter',
- u'north',
- u'flight']
- overlaps = set(sims_gensim_words).intersection(expected_sims_words)
- overlap_count = len(overlaps)
- self.assertGreaterEqual(
- overlap_count, 2,
- "only %i overlap in expected %s & actual %s" % (overlap_count, expected_sims_words, sims_gensim_words))
-
def test_cbow_neg_training(self):
-
model_gensim = FT_gensim(
vector_size=48, sg=0, cbow_mean=1, alpha=0.05, window=5, hs=0, negative=5,
min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
@@ -850,6 +714,194 @@ def obsolete_testLoadOldModel(self):
self.assertEqual(model.wv.vectors_vocab.shape, (12, 100))
self.assertEqual(model.wv.vectors_ngrams.shape, (2000000, 100))
+ def test_vectors_for_all_with_inference(self):
+ """Test vectors_for_all can infer new vectors."""
+ words = [
+ 'responding',
+ 'approached',
+ 'chairman',
+ 'an out-of-vocabulary word',
+ 'another out-of-vocabulary word',
+ ]
+ vectors_for_all = self.test_model.wv.vectors_for_all(words)
+
+ expected = 5
+ predicted = len(vectors_for_all)
+ assert expected == predicted
+
+ expected = self.test_model.wv['responding']
+ predicted = vectors_for_all['responding']
+ assert np.allclose(expected, predicted)
+
+ smaller_distance = np.linalg.norm(
+ vectors_for_all['an out-of-vocabulary word']
+ - vectors_for_all['another out-of-vocabulary word']
+ )
+ greater_distance = np.linalg.norm(
+ vectors_for_all['an out-of-vocabulary word']
+ - vectors_for_all['responding']
+ )
+ assert greater_distance > smaller_distance
+
+ def test_vectors_for_all_without_inference(self):
+ """Test vectors_for_all does not infer new vectors when prohibited."""
+ words = [
+ 'responding',
+ 'approached',
+ 'chairman',
+ 'an out-of-vocabulary word',
+ 'another out-of-vocabulary word',
+ ]
+ vectors_for_all = self.test_model.wv.vectors_for_all(words, allow_inference=False)
+
+ expected = 3
+ predicted = len(vectors_for_all)
+ assert expected == predicted
+
+ expected = self.test_model.wv['responding']
+ predicted = vectors_for_all['responding']
+ assert np.allclose(expected, predicted)
+
+
+@pytest.mark.parametrize('shrink_windows', [True, False])
+def test_cbow_hs_training(shrink_windows):
+ model_gensim = FT_gensim(
+ vector_size=48, sg=0, cbow_mean=1, alpha=0.05, window=5, hs=1, negative=0,
+ min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
+ sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET, shrink_windows=shrink_windows)
+
+ lee_data = LineSentence(datapath('lee_background.cor'))
+ model_gensim.build_vocab(lee_data)
+ orig0 = np.copy(model_gensim.wv.vectors[0])
+ model_gensim.train(lee_data, total_examples=model_gensim.corpus_count, epochs=model_gensim.epochs)
+ assert not (orig0 == model_gensim.wv.vectors[0]).all() # vector should vary after training
+
+ sims_gensim = model_gensim.wv.most_similar('night', topn=10)
+ sims_gensim_words = [word for (word, distance) in sims_gensim] # get similar words
+ expected_sims_words = [
+ u'night,',
+ u'night.',
+ u'rights',
+ u'kilometres',
+ u'in',
+ u'eight',
+ u'according',
+ u'flights',
+ u'during',
+ u'comes']
+ overlaps = set(sims_gensim_words).intersection(expected_sims_words)
+ overlap_count = len(overlaps)
+
+ message = f"only {overlap_count} overlap in expected {expected_sims_words} & actual {sims_gensim_words}"
+ assert overlap_count >= 2, message
+
+
+@pytest.mark.parametrize('shrink_windows', [True, False])
+def test_cbow_hs_training_fromfile(shrink_windows):
+ with temporary_file('gensim_fasttext.tst') as corpus_file:
+ model_gensim = FT_gensim(
+ vector_size=48, sg=0, cbow_mean=1, alpha=0.05, window=5, hs=1, negative=0,
+ min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
+ sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET * 4, shrink_windows=shrink_windows)
+
+ lee_data = LineSentence(datapath('lee_background.cor'))
+ utils.save_as_line_sentence(lee_data, corpus_file)
+
+ model_gensim.build_vocab(corpus_file=corpus_file)
+ orig0 = np.copy(model_gensim.wv.vectors[0])
+ model_gensim.train(corpus_file=corpus_file,
+ total_words=model_gensim.corpus_total_words,
+ epochs=model_gensim.epochs)
+ assert not (orig0 == model_gensim.wv.vectors[0]).all() # vector should vary after training
+
+ sims_gensim = model_gensim.wv.most_similar('night', topn=10)
+ sims_gensim_words = [word for (word, distance) in sims_gensim] # get similar words
+ expected_sims_words = [
+ u'night,',
+ u'night.',
+ u'rights',
+ u'kilometres',
+ u'in',
+ u'eight',
+ u'according',
+ u'flights',
+ u'during',
+ u'comes']
+ overlaps = set(sims_gensim_words).intersection(expected_sims_words)
+ overlap_count = len(overlaps)
+ message = f"only {overlap_count} overlap in expected {expected_sims_words} & actual {sims_gensim_words}"
+ assert overlap_count >= 2, message
+
+
+@pytest.mark.parametrize('shrink_windows', [True, False])
+def test_sg_hs_training(shrink_windows):
+ model_gensim = FT_gensim(
+ vector_size=48, sg=1, cbow_mean=1, alpha=0.025, window=5, hs=1, negative=0,
+ min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
+ sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET, shrink_windows=shrink_windows)
+
+ lee_data = LineSentence(datapath('lee_background.cor'))
+ model_gensim.build_vocab(lee_data)
+ orig0 = np.copy(model_gensim.wv.vectors[0])
+ model_gensim.train(lee_data, total_examples=model_gensim.corpus_count, epochs=model_gensim.epochs)
+ assert not (orig0 == model_gensim.wv.vectors[0]).all() # vector should vary after training
+
+ sims_gensim = model_gensim.wv.most_similar('night', topn=10)
+ sims_gensim_words = [word for (word, distance) in sims_gensim] # get similar words
+ expected_sims_words = [
+ u'night,',
+ u'night.',
+ u'eight',
+ u'nine',
+ u'overnight',
+ u'crew',
+ u'overnight.',
+ u'manslaughter',
+ u'north',
+ u'flight']
+ overlaps = set(sims_gensim_words).intersection(expected_sims_words)
+ overlap_count = len(overlaps)
+
+ message = f"only {overlap_count} overlap in expected {expected_sims_words} & actual {sims_gensim_words}"
+ assert overlap_count >= 2, message
+
+
+@pytest.mark.parametrize('shrink_windows', [True, False])
+def test_sg_hs_training_fromfile(shrink_windows):
+ with temporary_file('gensim_fasttext.tst') as corpus_file:
+ model_gensim = FT_gensim(
+ vector_size=48, sg=1, cbow_mean=1, alpha=0.025, window=5, hs=1, negative=0,
+ min_count=5, epochs=10, batch_words=1000, word_ngrams=1, sample=1e-3, min_n=3, max_n=6,
+ sorted_vocab=1, workers=1, min_alpha=0.0, bucket=BUCKET, shrink_windows=shrink_windows)
+
+ lee_data = LineSentence(datapath('lee_background.cor'))
+ utils.save_as_line_sentence(lee_data, corpus_file)
+
+ model_gensim.build_vocab(corpus_file=corpus_file)
+ orig0 = np.copy(model_gensim.wv.vectors[0])
+ model_gensim.train(corpus_file=corpus_file,
+ total_words=model_gensim.corpus_total_words,
+ epochs=model_gensim.epochs)
+ assert not (orig0 == model_gensim.wv.vectors[0]).all() # vector should vary after training
+
+ sims_gensim = model_gensim.wv.most_similar('night', topn=10)
+ sims_gensim_words = [word for (word, distance) in sims_gensim] # get similar words
+ expected_sims_words = [
+ u'night,',
+ u'night.',
+ u'eight',
+ u'nine',
+ u'overnight',
+ u'crew',
+ u'overnight.',
+ u'manslaughter',
+ u'north',
+ u'flight']
+ overlaps = set(sims_gensim_words).intersection(expected_sims_words)
+ overlap_count = len(overlaps)
+ message = f"only {overlap_count} overlap in expected {expected_sims_words} & actual {sims_gensim_words}"
+ assert overlap_count >= 2, message
+
with open(datapath('toy-data.txt')) as fin:
TOY_SENTENCES = [fin.read().strip().split(' ')]
diff --git a/gensim/test/test_keyedvectors.py b/gensim/test/test_keyedvectors.py
index fd96f9f26f..d5eda547ea 100644
--- a/gensim/test/test_keyedvectors.py
+++ b/gensim/test/test_keyedvectors.py
@@ -9,6 +9,7 @@
Automated tests for checking the poincare module from the models package.
"""
+import functools
import logging
import unittest
@@ -39,6 +40,81 @@ def test_most_similar(self):
predicted = [result[0] for result in self.vectors.most_similar('war', topn=5)]
self.assertEqual(expected, predicted)
+ def test_most_similar_vector(self):
+ """Can we pass vectors to most_similar directly?"""
+ positive = self.vectors.vectors[0:5]
+ most_similar = self.vectors.most_similar(positive=positive)
+ assert most_similar is not None
+
+ def test_most_similar_parameter_types(self):
+ """Are the positive/negative parameter types are getting interpreted correctly?"""
+ partial = functools.partial(self.vectors.most_similar, topn=5)
+
+ position = partial('war', 'peace')
+ position_list = partial(['war'], ['peace'])
+ keyword = partial(positive='war', negative='peace')
+ keyword_list = partial(positive=['war'], negative=['peace'])
+
+ #
+ # The above calls should all yield identical results.
+ #
+ assert position == position_list
+ assert position == keyword
+ assert position == keyword_list
+
+ def test_most_similar_cosmul_parameter_types(self):
+ """Are the positive/negative parameter types are getting interpreted correctly?"""
+ partial = functools.partial(self.vectors.most_similar_cosmul, topn=5)
+
+ position = partial('war', 'peace')
+ position_list = partial(['war'], ['peace'])
+ keyword = partial(positive='war', negative='peace')
+ keyword_list = partial(positive=['war'], negative=['peace'])
+
+ #
+ # The above calls should all yield identical results.
+ #
+ assert position == position_list
+ assert position == keyword
+ assert position == keyword_list
+
+ def test_vectors_for_all_list(self):
+ """Test vectors_for_all returns expected results with a list of keys."""
+ words = [
+ 'conflict',
+ 'administration',
+ 'terrorism',
+ 'an out-of-vocabulary word',
+ 'another out-of-vocabulary word',
+ ]
+ vectors_for_all = self.vectors.vectors_for_all(words)
+
+ expected = 3
+ predicted = len(vectors_for_all)
+ assert expected == predicted
+
+ expected = self.vectors['conflict']
+ predicted = vectors_for_all['conflict']
+ assert np.allclose(expected, predicted)
+
+ def test_vectors_for_all_with_copy_vecattrs(self):
+ """Test vectors_for_all returns can copy vector attributes."""
+ words = ['conflict']
+ vectors_for_all = self.vectors.vectors_for_all(words, copy_vecattrs=True)
+
+ expected = self.vectors.get_vecattr('conflict', 'count')
+ predicted = vectors_for_all.get_vecattr('conflict', 'count')
+ assert expected == predicted
+
+ def test_vectors_for_all_without_copy_vecattrs(self):
+ """Test vectors_for_all returns can copy vector attributes."""
+ words = ['conflict']
+ vectors_for_all = self.vectors.vectors_for_all(words, copy_vecattrs=False)
+
+ not_expected = self.vectors.get_vecattr('conflict', 'count')
+ predicted = vectors_for_all.get_vecattr('conflict', 'count')
+ assert not_expected != predicted
+
def test_most_similar_topn(self):
"""Test most_similar returns correct results when `topn` is specified."""
self.assertEqual(len(self.vectors.most_similar('war', topn=5)), 5)
diff --git a/gensim/test/test_parsing.py b/gensim/test/test_parsing.py
index d61671bd85..f96ad332d2 100644
--- a/gensim/test/test_parsing.py
+++ b/gensim/test/test_parsing.py
@@ -7,11 +7,24 @@
import logging
import unittest
+
+import mock
import numpy as np
-from gensim.parsing.preprocessing import \
- remove_stopwords, strip_punctuation2, strip_tags, strip_short, strip_numeric, strip_non_alphanum, \
- strip_multiple_whitespaces, split_alphanum, stem_text
+from gensim.parsing.preprocessing import (
+ remove_short_tokens,
+ remove_stopword_tokens,
+ remove_stopwords,
+ stem_text,
+ split_alphanum,
+ split_on_space,
+ strip_multiple_whitespaces,
+ strip_non_alphanum,
+ strip_numeric,
+ strip_punctuation,
+ strip_short,
+ strip_tags,
+)
# several documents
doc1 = """C'est un trou de verdure où chante une rivière,
@@ -38,7 +51,7 @@
for many searching purposes, a little fuzziness would help. """
-dataset = [strip_punctuation2(x.lower()) for x in [doc1, doc2, doc3, doc4]]
+dataset = [strip_punctuation(x.lower()) for x in [doc1, doc2, doc3, doc4]]
# doc1 and doc2 have class 0, doc3 and doc4 avec class 1
classes = np.array([[1, 0], [1, 0], [0, 1], [0, 1]])
@@ -67,6 +80,26 @@ def test_split_alphanum(self):
def test_strip_stopwords(self):
self.assertEqual(remove_stopwords("the world is square"), "world square")
+ # confirm redifining the global `STOPWORDS` working
+ with mock.patch('gensim.parsing.preprocessing.STOPWORDS', frozenset(["the"])):
+ self.assertEqual(remove_stopwords("the world is square"), "world is square")
+
+ def test_strip_stopword_tokens(self):
+ self.assertEqual(remove_stopword_tokens(["the", "world", "is", "sphere"]), ["world", "sphere"])
+
+ # confirm redifining the global `STOPWORDS` working
+ with mock.patch('gensim.parsing.preprocessing.STOPWORDS', frozenset(["the"])):
+ self.assertEqual(
+ remove_stopword_tokens(["the", "world", "is", "sphere"]),
+ ["world", "is", "sphere"]
+ )
+
+ def test_strip_short_tokens(self):
+ self.assertEqual(remove_short_tokens(["salut", "les", "amis", "du", "59"], 3), ["salut", "les", "amis"])
+
+ def test_split_on_space(self):
+ self.assertEqual(split_on_space(" salut les amis du 59 "), ["salut", "les", "amis", "du", "59"])
+
def test_stem_text(self):
target = \
"while it is quit us to be abl to search a larg " + \
diff --git a/gensim/test/test_phrases.py b/gensim/test/test_phrases.py
index bbfbfaad40..e8d9567b20 100644
--- a/gensim/test/test_phrases.py
+++ b/gensim/test/test_phrases.py
@@ -305,32 +305,42 @@ def test_pruning(self):
class TestPhrasesPersistence(PhrasesData, unittest.TestCase):
-
def test_save_load_custom_scorer(self):
"""Test saving and loading a Phrases object with a custom scorer."""
+ bigram = Phrases(self.sentences, min_count=1, threshold=.001, scoring=dumb_scorer)
with temporary_file("test.pkl") as fpath:
- bigram = Phrases(self.sentences, min_count=1, threshold=.001, scoring=dumb_scorer)
bigram.save(fpath)
bigram_loaded = Phrases.load(fpath)
- test_sentences = [['graph', 'minors', 'survey', 'human', 'interface', 'system']]
- seen_scores = list(bigram_loaded.find_phrases(test_sentences).values())
- assert all(score == 1 for score in seen_scores)
- assert len(seen_scores) == 3 # 'graph minors' and 'survey human' and 'interface system'
+ test_sentences = [['graph', 'minors', 'survey', 'human', 'interface', 'system']]
+ seen_scores = list(bigram_loaded.find_phrases(test_sentences).values())
+
+ assert all(score == 1 for score in seen_scores)
+ assert len(seen_scores) == 3 # 'graph minors' and 'survey human' and 'interface system'
def test_save_load(self):
"""Test saving and loading a Phrases object."""
+ bigram = Phrases(self.sentences, min_count=1, threshold=1)
+ with temporary_file("test.pkl") as fpath:
+ bigram.save(fpath)
+ bigram_loaded = Phrases.load(fpath)
+
+ test_sentences = [['graph', 'minors', 'survey', 'human', 'interface', 'system']]
+ seen_scores = set(round(score, 3) for score in bigram_loaded.find_phrases(test_sentences).values())
+ assert seen_scores == set([
+ 5.167, # score for graph minors
+ 3.444 # score for human interface
+ ])
+
+ def test_save_load_with_connector_words(self):
+ """Test saving and loading a Phrases object."""
+ connector_words = frozenset({'of'})
+ bigram = Phrases(self.sentences, min_count=1, threshold=1, connector_words=connector_words)
with temporary_file("test.pkl") as fpath:
- bigram = Phrases(self.sentences, min_count=1, threshold=1)
bigram.save(fpath)
bigram_loaded = Phrases.load(fpath)
- test_sentences = [['graph', 'minors', 'survey', 'human', 'interface', 'system']]
- seen_scores = set(round(score, 3) for score in bigram_loaded.find_phrases(test_sentences).values())
- assert seen_scores == set([
- 5.167, # score for graph minors
- 3.444 # score for human interface
- ])
+ assert bigram_loaded.connector_words == connector_words
def test_save_load_string_scoring(self):
"""Test backwards compatibility with a previous version of Phrases with custom scoring."""
@@ -385,6 +395,15 @@ def test_save_load(self):
bigram_loaded[['graph', 'minors', 'survey', 'human', 'interface', 'system']],
['graph_minors', 'survey', 'human_interface', 'system'])
+ def test_save_load_with_connector_words(self):
+ """Test saving and loading a FrozenPhrases object."""
+ connector_words = frozenset({'of'})
+ with temporary_file("test.pkl") as fpath:
+ bigram = FrozenPhrases(Phrases(self.sentences, min_count=1, threshold=1, connector_words=connector_words))
+ bigram.save(fpath)
+ bigram_loaded = FrozenPhrases.load(fpath)
+ self.assertEqual(bigram_loaded.connector_words, connector_words)
+
def test_save_load_string_scoring(self):
"""Test saving and loading a FrozenPhrases object with a string scoring parameter.
This should ensure backwards compatibility with the previous version of FrozenPhrases"""
diff --git a/gensim/test/test_similarities.py b/gensim/test/test_similarities.py
index 4929082c2a..35ddd03397 100644
--- a/gensim/test/test_similarities.py
+++ b/gensim/test/test_similarities.py
@@ -33,7 +33,7 @@
from gensim.similarities import SparseTermSimilarityMatrix
from gensim.similarities import LevenshteinSimilarityIndex
from gensim.similarities.docsim import _nlargest
-from gensim.similarities.levenshtein import levdist, levsim
+from gensim.similarities.fastss import editdist
try:
from pyemd import emd # noqa:F401
@@ -1544,123 +1544,48 @@ def test_inner_product_corpus_corpus_true_true(self):
self.assertTrue(numpy.allclose(expected_result, result.todense()))
-class TestLevenshteinDistance(unittest.TestCase):
- @unittest.skipIf(LevenshteinSimilarityIndex is None, "gensim.similarities.levenshtein is disabled")
- def test_max_distance(self):
- t1 = "holiday"
- t2 = "day"
- max_distance = max(len(t1), len(t2))
-
- self.assertEqual(4, levdist(t1, t2))
- self.assertEqual(4, levdist(t1, t2, 4))
- self.assertEqual(max_distance, levdist(t1, t2, 2))
- self.assertEqual(max_distance, levdist(t1, t2, -2))
-
-
-class TestLevenshteinSimilarity(unittest.TestCase):
- @unittest.skipIf(LevenshteinSimilarityIndex is None, "gensim.similarities.levenshtein is disabled")
- def test_empty_strings(self):
- t1 = ""
- t2 = ""
-
- self.assertEqual(1.0, levsim(t1, t2))
-
- @unittest.skipIf(LevenshteinSimilarityIndex is None, "gensim.similarities.levenshtein is disabled")
- def test_negative_hyperparameters(self):
- t1 = "holiday"
- t2 = "day"
- alpha = 2.0
- beta = 2.0
-
- with self.assertRaises(AssertionError):
- levsim(t1, t2, -alpha, beta)
-
- with self.assertRaises(AssertionError):
- levsim(t1, t2, alpha, -beta)
-
- with self.assertRaises(AssertionError):
- levsim(t1, t2, -alpha, -beta)
-
- @unittest.skipIf(LevenshteinSimilarityIndex is None, "gensim.similarities.levenshtein is disabled")
- def test_min_similarity(self):
- t1 = "holiday"
- t2 = "day"
- alpha = 2.0
- beta = 2.0
- similarity = alpha * (1 - 4.0 / 7)**beta
- assert similarity > 0.1 and similarity < 0.5
-
- self.assertAlmostEqual(similarity, levsim(t1, t2, alpha, beta))
-
- self.assertAlmostEqual(similarity, levsim(t1, t2, alpha, beta, -2))
- self.assertAlmostEqual(similarity, levsim(t1, t2, alpha, beta, -2.0))
-
- self.assertAlmostEqual(similarity, levsim(t1, t2, alpha, beta, 0))
- self.assertAlmostEqual(similarity, levsim(t1, t2, alpha, beta, 0.0))
-
- self.assertEqual(similarity, levsim(t1, t2, alpha, beta, 0.1))
- self.assertEqual(0.0, levsim(t1, t2, alpha, beta, 0.5))
- self.assertEqual(0.0, levsim(t1, t2, alpha, beta, 1.0))
-
- self.assertEqual(0.0, levsim(t1, t2, alpha, beta, 2))
- self.assertEqual(0.0, levsim(t1, t2, alpha, beta, 2.0))
-
-
class TestLevenshteinSimilarityIndex(unittest.TestCase):
def setUp(self):
self.documents = [[u"government", u"denied", u"holiday"], [u"holiday", u"slowing", u"hollingworth"]]
self.dictionary = Dictionary(self.documents)
+ max_distance = max(len(term) for term in self.dictionary.values())
+ self.index = LevenshteinSimilarityIndex(self.dictionary, max_distance=max_distance)
- @unittest.skipIf(LevenshteinSimilarityIndex is None, "gensim.similarities.levenshtein is disabled")
- def test_most_similar(self):
+ def test_most_similar_topn(self):
"""Test most_similar returns expected results."""
- index = LevenshteinSimilarityIndex(self.dictionary)
- results = list(index.most_similar(u"holiday", topn=1))
- self.assertLess(0, len(results))
- self.assertGreaterEqual(1, len(results))
- results = list(index.most_similar(u"holiday", topn=4))
- self.assertLess(1, len(results))
- self.assertGreaterEqual(4, len(results))
+ results = list(self.index.most_similar(u"holiday", topn=0))
+ self.assertEqual(0, len(results))
- # check the order of the results
- results = index.most_similar(u"holiday", topn=4)
- terms, _ = tuple(zip(*results))
- self.assertEqual((u"hollingworth", u"slowing", u"denied", u"government"), terms)
+ results = list(self.index.most_similar(u"holiday", topn=1))
+ self.assertEqual(1, len(results))
- # check that the term itself is not returned
- index = LevenshteinSimilarityIndex(self.dictionary)
- terms = [term for term, similarity in index.most_similar(u"holiday", topn=len(self.dictionary))]
- self.assertFalse(u"holiday" in terms)
+ results = list(self.index.most_similar(u"holiday", topn=4))
+ self.assertEqual(4, len(results))
- # check that the threshold works as expected
- index = LevenshteinSimilarityIndex(self.dictionary, threshold=0.0)
- results = list(index.most_similar(u"holiday", topn=10))
- self.assertLess(0, len(results))
- self.assertGreaterEqual(10, len(results))
+ results = list(self.index.most_similar(u"holiday", topn=len(self.dictionary)))
+ self.assertEqual(len(self.dictionary) - 1, len(results))
+ self.assertNotIn(u"holiday", results)
- index = LevenshteinSimilarityIndex(self.dictionary, threshold=1.0)
- results = list(index.most_similar(u"holiday", topn=10))
- self.assertEqual(0, len(results))
+ def test_most_similar_result_order(self):
+ results = self.index.most_similar(u"holiday", topn=4)
+ terms, _ = zip(*results)
+ expected_terms = (u"hollingworth", u"denied", u"slowing", u"government")
+ self.assertEqual(expected_terms, terms)
- # check that the alpha works as expected
+ def test_most_similar_alpha(self):
index = LevenshteinSimilarityIndex(self.dictionary, alpha=1.0)
first_similarities = numpy.array([similarity for term, similarity in index.most_similar(u"holiday", topn=10)])
index = LevenshteinSimilarityIndex(self.dictionary, alpha=2.0)
second_similarities = numpy.array([similarity for term, similarity in index.most_similar(u"holiday", topn=10)])
self.assertTrue(numpy.allclose(2.0 * first_similarities, second_similarities))
- # check that the beta works as expected
+ def test_most_similar_beta(self):
index = LevenshteinSimilarityIndex(self.dictionary, alpha=1.0, beta=1.0)
first_similarities = numpy.array([similarity for term, similarity in index.most_similar(u"holiday", topn=10)])
index = LevenshteinSimilarityIndex(self.dictionary, alpha=1.0, beta=2.0)
second_similarities = numpy.array([similarity for term, similarity in index.most_similar(u"holiday", topn=10)])
self.assertTrue(numpy.allclose(first_similarities ** 2.0, second_similarities))
- # check proper integration with SparseTermSimilarityMatrix
- index = LevenshteinSimilarityIndex(self.dictionary, alpha=1.0, beta=1.0)
- similarity_matrix = SparseTermSimilarityMatrix(index, DICTIONARY)
- self.assertTrue(scipy.sparse.issparse(similarity_matrix.matrix))
-
class TestWordEmbeddingSimilarityIndex(unittest.TestCase):
def setUp(self):
@@ -1707,6 +1632,32 @@ def test_most_similar(self):
self.assertTrue(numpy.allclose(first_similarities**2.0, second_similarities))
+class TestFastSS(unittest.TestCase):
+ def test_editdist_same_unicode_kind_latin1(self):
+ """Test editdist returns the expected result with two Latin-1 strings."""
+ expected = 2
+ actual = editdist('Zizka', 'siska')
+ assert expected == actual
+
+ def test_editdist_same_unicode_kind_ucs2(self):
+ """Test editdist returns the expected result with two UCS-2 strings."""
+ expected = 2
+ actual = editdist('Žižka', 'šiška')
+ assert expected == actual
+
+ def test_editdist_same_unicode_kind_ucs4(self):
+ """Test editdist returns the expected result with two UCS-4 strings."""
+ expected = 2
+ actual = editdist('Žižka 😀', 'šiška 😀')
+ assert expected == actual
+
+ def test_editdist_different_unicode_kinds(self):
+ """Test editdist returns the expected result with strings of different Unicode kinds."""
+ expected = 2
+ actual = editdist('Žižka', 'siska')
+ assert expected == actual
+
+
if __name__ == '__main__':
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.DEBUG)
unittest.main()
diff --git a/gensim/test/test_word2vec.py b/gensim/test/test_word2vec.py
index d46b2f3e37..43505b0be2 100644
--- a/gensim/test/test_word2vec.py
+++ b/gensim/test/test_word2vec.py
@@ -686,6 +686,38 @@ def test_cbow_neg_fromfile(self):
)
self.model_sanity(model, with_corpus_file=True)
+ def test_sg_fixedwindowsize(self):
+ """Test skipgram with fixed window size. Use NS."""
+ model = word2vec.Word2Vec(
+ sg=1, window=5, shrink_windows=False, hs=0,
+ negative=15, min_count=5, epochs=10, workers=2
+ )
+ self.model_sanity(model)
+
+ def test_sg_fixedwindowsize_fromfile(self):
+ """Test skipgram with fixed window size. Use HS and train from file."""
+ model = word2vec.Word2Vec(
+ sg=1, window=5, shrink_windows=False, hs=1,
+ negative=0, min_count=5, epochs=10, workers=2
+ )
+ self.model_sanity(model, with_corpus_file=True)
+
+ def test_cbow_fixedwindowsize(self, ranks=None):
+ """Test CBOW with fixed window size. Use HS."""
+ model = word2vec.Word2Vec(
+ sg=0, cbow_mean=1, alpha=0.1, window=5, shrink_windows=False,
+ hs=1, negative=0, min_count=5, epochs=10, workers=2
+ )
+ self.model_sanity(model, ranks=ranks)
+
+ def test_cbow_fixedwindowsize_fromfile(self):
+ """Test CBOW with fixed window size. Use NS and train from file."""
+ model = word2vec.Word2Vec(
+ sg=0, cbow_mean=1, alpha=0.1, window=5, shrink_windows=False,
+ hs=0, negative=15, min_count=5, epochs=10, workers=2
+ )
+ self.model_sanity(model, with_corpus_file=True)
+
def test_cosmul(self):
model = word2vec.Word2Vec(sentences, vector_size=2, min_count=1, hs=1, negative=0)
sims = model.wv.most_similar_cosmul('graph', topn=10)
@@ -843,6 +875,16 @@ def test_predict_output_word(self):
model_without_neg = word2vec.Word2Vec(sentences, min_count=1, negative=0)
self.assertRaises(RuntimeError, model_without_neg.predict_output_word, ['system', 'human'])
+ # passing indices instead of words in context
+ str_context = ['system', 'human']
+ mixed_context = [model_with_neg.wv.get_index(str_context[0]), str_context[1]]
+ idx_context = [model_with_neg.wv.get_index(w) for w in str_context]
+ prediction_from_str = model_with_neg.predict_output_word(str_context, topn=5)
+ prediction_from_mixed = model_with_neg.predict_output_word(mixed_context, topn=5)
+ prediction_from_idx = model_with_neg.predict_output_word(idx_context, topn=5)
+ self.assertEqual(prediction_from_str, prediction_from_mixed)
+ self.assertEqual(prediction_from_str, prediction_from_idx)
+
def test_load_old_model(self):
"""Test loading an old word2vec model of indeterminate version"""
diff --git a/gensim/utils.py b/gensim/utils.py
index 6a6d1f0c3a..30b6d85f58 100644
--- a/gensim/utils.py
+++ b/gensim/utils.py
@@ -50,11 +50,10 @@
RE_HTML_ENTITY = re.compile(r'&(#?)([xX]?)(\w{1,8});', re.UNICODE)
NO_CYTHON = RuntimeError(
- "Cython extensions are unavailable. "
- "Without them, this gensim functionality is disabled. "
- "If you've installed from a package, ask the package maintainer to include Cython extensions. "
- "If you're building gensim from source yourself, run `python setup.py build_ext --inplace` "
- "and retry. "
+ "Compiled extensions are unavailable. "
+ "If you've installed from a package, ask the package maintainer to include compiled extensions. "
+ "If you're building Gensim from source yourself, install Cython and a C compiler, and then "
+ "run `python setup.py build_ext --inplace` to retry. "
)
"""An exception that gensim code raises when Cython extensions are unavailable."""
@@ -835,6 +834,9 @@ def __getitem__(self, val):
return str(val)
raise ValueError("internal id out of bounds (%s, expected <0..%s))" % (val, self.num_terms))
+ def __contains__(self, val):
+ return 0 <= val < self.num_terms
+
def iteritems(self):
"""Iterate over all keys and values.
diff --git a/release/annotate_pr.py b/release/annotate_pr.py
new file mode 100644
index 0000000000..1f5d1c8bcb
--- /dev/null
+++ b/release/annotate_pr.py
@@ -0,0 +1,39 @@
+"""Helper script for including change log entries in an open PR.
+
+Automatically constructs the change log entry from the PR title.
+Copies the entry to the window manager clipboard.
+Opens the change log belonging to the specific PR in a browser window.
+All you have to do is paste and click "commit changes".
+"""
+import json
+import sys
+import webbrowser
+
+import smart_open
+
+
+def copy_to_clipboard(text):
+ try:
+ import pyperclip
+ except ImportError:
+ print('pyperclip `_, two Python packages for scientific computing.
You must have them installed prior to installing `gensim`.
-It is also recommended you install a fast BLAS library before installing NumPy. This is optional, but using an optimized BLAS such as `ATLAS `_.
+For alternative modes of installation, see the `documentation `_.
-Gensim is being `continuously tested `_ under Python 3.6, 3.7 and 3.8.
+Gensim is being `continuously tested `_ under all `supported Python versions `_.
Support for Python 2.7 was dropped in gensim 4.0.0 – install gensim 3.8.3 if you must use Python 2.7.
@@ -271,14 +272,13 @@ def run(self):
'mock',
'cython',
'testfixtures',
- 'Morfessor==2.0.2a4',
+ 'Morfessor>=2.0.2a4',
]
if not (sys.platform.lower().startswith("win") and sys.version_info[:2] >= (3, 9)):
core_testenv.extend([
'pyemd',
'nmslib',
- 'python-Levenshtein >= 0.10.2',
])
# Add additional requirements for testing on Linux that are skipped on Windows.
@@ -315,13 +315,13 @@ def run(self):
'pandas',
]
-NUMPY_STR = 'numpy >= 1.11.3'
+NUMPY_STR = 'numpy >= 1.17.0'
#
# We pin the Cython version for reproducibility. We expect our extensions
# to build with any sane version of Cython, so we should update this pin
# periodically.
#
-CYTHON_STR = 'Cython==0.29.21'
+CYTHON_STR = 'Cython==0.29.23'
install_requires = [
NUMPY_STR,
@@ -338,7 +338,7 @@ def run(self):
setup(
name='gensim',
- version='4.0.1',
+ version='4.1.0',
description='Python framework for fast Vector Space Modelling',
long_description=LONG_DESCRIPTION,