import gensim fails on Windows with numpy 1.19.5

[Bomme]

Problem description

After I upgraded to gensim 4.0.0 from 4.0.0beta, I was no longer able to do import gensim (see numpy error message below). When I upgraded to numpy 1.20.2 the problem was solved.

Steps/code/corpus to reproduce

Python 3.8.6 (tags/v3.8.6:db45529, Sep 23 2020, 15:52:53) [MSC v.1927 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import gensim
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\user\Dev\code\venv\lib\site-packages\gensim\__init__.py", line 11, in <module>
    from gensim import parsing, corpora, matutils, interfaces, models, similarities, utils  # noqa:F401
  File "C:\Users\user\Dev\code\venv\lib\site-packages\gensim\corpora\__init__.py", line 6, in <module>
    from .indexedcorpus import IndexedCorpus  # noqa:F401 must appear before the other classes
  File "C:\Users\user\Dev\code\venv\lib\site-packages\gensim\corpora\indexedcorpus.py", line 14, in <module>
    from gensim import interfaces, utils
  File "C:\Users\user\Dev\code\venv\lib\site-packages\gensim\interfaces.py", line 19, in <module>
    from gensim import utils, matutils
  File "C:\Users\user\Dev\code\venv\lib\site-packages\gensim\matutils.py", line 1024, in <module>
    from gensim._matutils import logsumexp, mean_absolute_difference, dirichlet_expectation
  File "gensim\_matutils.pyx", line 1, in init gensim._matutils
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
>>>

Versions

Windows-10-10.0.19041-SP0
Python 3.8.6 (tags/v3.8.6:db45529, Sep 23 2020, 15:52:53) [MSC v.1927 64 bit (AMD64)] on win32
Bits 64

NumPy 1.19.5
scipy 1.6.0
gensim 4.0.0

[piskvorky]

Thanks for reporting @Bomme . How did you install Gensim 4.0.0?

Probably related to the incompatible C API change in Numpy==1.20.0: numpy/numpy#16938. Previous discussion #3085 (comment) – using oldest-supported-numpy should have fixed this. So I'm not sure what happened.

[Bomme]

I installed it simply through: pip install gensim~=4.0.0

[piskvorky]

Hm. @mpenkov is it possible to tell which numpy was used to build the py3.8 Windows wheels?

If the oldest-supported-numpy trick doesn't work, we're in trouble (numpy is, and we by extension).

[mpenkov]

Yes, we can see from here that 1.20.1 was used.

Successfully installed Cython-0.29.21 gensim-4.0.0 numpy-1.20.1 scipy-1.6.2 smart-open-4.2.0

[piskvorky]

Oh no :( So oldest-supported-numpy failed? We might have to pin the older numpy version manually then, and do a post-release ASAP.

[mpenkov]

I've initiated the 4.0.1 release. Should be ready within an hour or so.

[piskvorky]

Could also be a post release (4.0.0.post0). No functionality changed IIRC, just a packaging issue.

[mpenkov]

I've already made the appropriate commits and started building the wheels. If it's not critical, I'd prefer to release it as 4.0.1 in the next hour or so.

If you have a strong preference for 4.0.0.post0, I'll deal with it tomorrow.

[piskvorky]

No, not strong preference :) Thanks for the prompt re-release!

[mpenkov]

https://pypi.org/project/gensim/4.0.1/

[Bomme]

I can confirm that gensim==4.0.1 and numpy~=1.19.5 seem to work together nicely! 🚀