FAST_VERSION is -1 when using latest gensim (3.8.1)

[cfd-01]

Problem description

assert gensim.models.doc2vec.FAST_VERSION > -1 fails with gensim 3.8.1, even though it works with 3.7.0.

Steps/code/corpus to reproduce

When I install the latest version of gensim using pip install --upgrade gensim, I observe the following:

>>> import gensim.models.doc2vec
>>> assert gensim.models.doc2vec.FAST_VERSION > -1, "This will be painfully slow otherwise"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AssertionError: This will be painfully slow otherwise

When I roll back to version 3.7, the assertion passes:

>>> import multiprocessing
>>> import gensim.models.doc2vec
>>> assert gensim.models.doc2vec.FAST_VERSION > -1, "This will be painfully slow otherwise"
>>> 

Versions

>>> import platform; print(platform.platform())
Darwin-18.7.0-x86_64-i386-64bit
>>> import sys; print("Python", sys.version)
Python 3.7.6 (default, Dec 30 2019, 19:38:28) 
[Clang 11.0.0 (clang-1100.0.33.16)]
>>> import numpy; print("NumPy", numpy.__version__)
NumPy 1.18.1
>>> import scipy; print("SciPy", scipy.__version__)
SciPy 1.4.1
>>> import gensim; print("gensim", gensim.__version__)
gensim 3.8.1
>>> from gensim.models import word2vec;print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION -1

[piskvorky]

What did the 3.8.1 installation look like? Can you post the full installation output from pip?

That should contain the concrete issue.

[cfd-01]

Collecting gensim
  Using cached gensim-3.8.2-cp37-cp37m-macosx_10_9_x86_64.whl (23.7 MB)
Requirement already satisfied, skipping upgrade: six>=1.5.0 in ./venv/lib/python3.7/site-packages (from gensim) (1.12.0)
Requirement already satisfied, skipping upgrade: scipy>=1.0.0 in ./venv/lib/python3.7/site-packages (from gensim) (1.4.1)
Requirement already satisfied, skipping upgrade: smart-open>=1.8.1 in ./venv/lib/python3.7/site-packages (from gensim) (1.10.0)
Requirement already satisfied, skipping upgrade: numpy>=1.11.3 in ./venv/lib/python3.7/site-packages (from gensim) (1.18.1)
Requirement already satisfied, skipping upgrade: requests in ./venv/lib/python3.7/site-packages (from smart-open>=1.8.1->gensim) (2.23.0)
Requirement already satisfied, skipping upgrade: boto3 in ./venv/lib/python3.7/site-packages (from smart-open>=1.8.1->gensim) (1.12.15)
Requirement already satisfied, skipping upgrade: google-cloud-storage in ./venv/lib/python3.7/site-packages (from smart-open>=1.8.1->gensim) (1.27.0)
Requirement already satisfied, skipping upgrade: idna<3,>=2.5 in ./venv/lib/python3.7/site-packages (from requests->smart-open>=1.8.1->gensim) (2.9)
Requirement already satisfied, skipping upgrade: chardet<4,>=3.0.2 in ./venv/lib/python3.7/site-packages (from requests->smart-open>=1.8.1->gensim) (3.0.4)
Requirement already satisfied, skipping upgrade: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./venv/lib/python3.7/site-packages (from requests->smart-open>=1.8.1->gensim) (1.25.8)
Requirement already satisfied, skipping upgrade: certifi>=2017.4.17 in ./venv/lib/python3.7/site-packages (from requests->smart-open>=1.8.1->gensim) (2019.11.28)
Requirement already satisfied, skipping upgrade: botocore<1.16.0,>=1.15.15 in ./venv/lib/python3.7/site-packages (from boto3->smart-open>=1.8.1->gensim) (1.15.15)
Requirement already satisfied, skipping upgrade: jmespath<1.0.0,>=0.7.1 in ./venv/lib/python3.7/site-packages (from boto3->smart-open>=1.8.1->gensim) (0.9.5)
Requirement already satisfied, skipping upgrade: s3transfer<0.4.0,>=0.3.0 in ./venv/lib/python3.7/site-packages (from boto3->smart-open>=1.8.1->gensim) (0.3.3)
Requirement already satisfied, skipping upgrade: google-cloud-core<2.0dev,>=1.2.0 in ./venv/lib/python3.7/site-packages (from google-cloud-storage->smart-open>=1.8.1->gensim) (1.3.0)
Requirement already satisfied, skipping upgrade: google-auth<2.0dev,>=1.11.0 in ./venv/lib/python3.7/site-packages (from google-cloud-storage->smart-open>=1.8.1->gensim) (1.13.1)
Requirement already satisfied, skipping upgrade: google-resumable-media<0.6dev,>=0.5.0 in ./venv/lib/python3.7/site-packages (from google-cloud-storage->smart-open>=1.8.1->gensim) (0.5.0)
Requirement already satisfied, skipping upgrade: python-dateutil<3.0.0,>=2.1 in ./venv/lib/python3.7/site-packages (from botocore<1.16.0,>=1.15.15->boto3->smart-open>=1.8.1->gensim) (2.8.1)
Requirement already satisfied, skipping upgrade: docutils<0.16,>=0.10 in ./venv/lib/python3.7/site-packages (from botocore<1.16.0,>=1.15.15->boto3->smart-open>=1.8.1->gensim) (0.15.2)
Requirement already satisfied, skipping upgrade: google-api-core<2.0.0dev,>=1.16.0 in ./venv/lib/python3.7/site-packages (from google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage->smart-open>=1.8.1->gensim) (1.16.0)
Requirement already satisfied, skipping upgrade: cachetools<5.0,>=2.0.0 in ./venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (4.1.0)
Requirement already satisfied, skipping upgrade: setuptools>=40.3.0 in ./venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (41.2.0)
Requirement already satisfied, skipping upgrade: pyasn1-modules>=0.2.1 in ./venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (0.2.8)
Requirement already satisfied, skipping upgrade: rsa<4.1,>=3.1.4 in ./venv/lib/python3.7/site-packages (from google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (4.0)
Requirement already satisfied, skipping upgrade: googleapis-common-protos<2.0dev,>=1.6.0 in ./venv/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.16.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage->smart-open>=1.8.1->gensim) (1.51.0)
Requirement already satisfied, skipping upgrade: pytz in ./venv/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.16.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage->smart-open>=1.8.1->gensim) (2019.3)
Requirement already satisfied, skipping upgrade: protobuf>=3.4.0 in ./venv/lib/python3.7/site-packages (from google-api-core<2.0.0dev,>=1.16.0->google-cloud-core<2.0dev,>=1.2.0->google-cloud-storage->smart-open>=1.8.1->gensim) (3.11.3)
Requirement already satisfied, skipping upgrade: pyasn1<0.5.0,>=0.4.6 in ./venv/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2.0dev,>=1.11.0->google-cloud-storage->smart-open>=1.8.1->gensim) (0.4.8)
Installing collected packages: gensim
  Attempting uninstall: gensim
    Found existing installation: gensim 3.7.0
    Uninstalling gensim-3.7.0:
      Successfully uninstalled gensim-3.7.0
Successfully installed gensim-3.8.2

[piskvorky]

Thanks. I installed using the same wheel gensim-3.8.2-cp37-cp37m-macosx_10_9_x86_64.whl in Python3.7 and I can replicate this:

@menshikh-iv @mpenkov were there any changes between 3.7 and 3.8 when it comes to packaging binary wheels?

[piskvorky]

Btw that unable to import 'smart_open.gcs seems out of place. Is it necessary to pollute the gensim import with such scary looking (and irrelevant) printouts?

[menshikh-iv]

were there any changes between 3.7 and 3.8 when it comes to packaging binary wheels?

no (and I guess that this is a problem), at least I have several ideas about "what can be wrong"

@piskvorky can you please check that behavior on gensim==3.7.3, gensim==3.8.0, gensim==3.8.1, gensim==3.8.2 with py37? I need this information for investigation.

[piskvorky]

gensim==3.7.3, gensim==3.8.0, gensim==3.8.1:

from gensim.models import word2vec; print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION 0

gensim==3.8.2:

from gensim.models import word2vec; print("FAST_VERSION", word2vec.FAST_VERSION)
FAST_VERSION -1

The same machine, OS, Python3.7 & virtual env, just changing the gensim version with pip install gensim==x.y.z.

Probably the most critical issue we have right now: Gensim's most used models are unusable.

[menshikh-iv]

Thanks for info, I’ll look into this week @piskvorky

[menshikh-iv]

@piskvorky one more thing, can you do exactly same testing on py35 and py36? I have a feeling that other versions can be affected too?

[piskvorky]

Sure. Python3.6:

3.8.1: FAST_VERSION 0
3.8.2: FAST_VERSION -1

[menshikh-iv]

@piskvorky can you please check this 3 wheels (py 3.5, 3.6 and 3.7). Don't worry about version, this built from a2ec4c3 (current develop)

https://drive.google.com/open?id=1gkjVBuHV4i_zHcOwtiDkDXwqG_XbLjXk
https://drive.google.com/open?id=1PUJrzvxTtcGqLTeeiAU8Dj-KWCMuiYOV
https://drive.google.com/open?id=1_88vr_hJi8BxDMyzWFMvfftyNu9lNiiV

[menshikh-iv]

Related PR MacPython/gensim-wheels#21

[piskvorky]

I don't have access to 3.5 but I confirm the above wheels for 3.7 and 3.6 work!
FAST_VERSION 0 🚀

[gojomo]

What was the problem? Should that file in MacPython/gensim-wheels really name a specific commit, rather than a version? (Or is that just temporary?)

[menshikh-iv]

@gojomo this was temporary, I just need "concrete fresh commit" for my investigations. You can read more about it MacPython/gensim-wheels#21 (comment)

[menshikh-iv]

Fixed by MacPython/gensim-wheels#21