PDM and Poetry can't install symengine properly

[martinruefenacht]

I am trying to run symengine in a PDM or Poetry project and while I can install it via "pdm add" or "poetry add" and I find it in the virtualenv site-packages, the lib directory is empty compared to my globally installed version with my OS python.

When I run python directly from the venv and import symengine I get:

Python 3.12.1 (main, Dec 18 2023, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import symengine
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/****/****/****/.venv/lib/python3.12/site-packages/symengine/__init__.py", line 12, in <module>
    import symengine.lib.symengine_wrapper as wrapper
ModuleNotFoundError: No module named 'symengine.lib.symengine_wrapper'

Is this a problem with both Poetry and PDM or with symengine not picking something up?

[rikardn]

I tested to create a venv with Python 3.12 and install symengine without using poetry and it works fine on my computer:

$ python3.12 --version
Python 3.12.0
[~/testing] 
$ python3.12 -m venv py312venv
[~/testing] 
$ cd py312venv
[~/testing/py312venv] 
$ . ./bin/activate
(py312venv) [~/testing/py312venv] 
$ pip install symengine
Collecting symengine
  Obtaining dependency information for symengine from https://files.pythonhosted.org/packages/b5/58/d22492431cefd539a3655ec298752ad56a11c349bb32c668d2567172118f/symengine-0.11.0-cp312-cp312-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata
  Downloading symengine-0.11.0-cp312-cp312-manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (1.2 kB)
Downloading symengine-0.11.0-cp312-cp312-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (39.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.3/39.3 MB 19.7 MB/s eta 0:00:00
Installing collected packages: symengine
Successfully installed symengine-0.11.0

[notice] A new release of pip is available: 23.2.1 -> 24.0
[notice] To update, run: pip install --upgrade pip
(py312venv) [~/testing/py312venv] 
$ python
Python 3.12.0 (main, Nov 14 2023, 08:54:25) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import symengine
>>>

The files installed:

[~/testing/py312venv/lib/python3.12/site-packages] 
$ tree symengine*              
symengine
├── functions.py
├── __init__.py
├── lib
│   ├── __init__.py
│   ├── __pycache__
│   │   └── __init__.cpython-312.pyc
│   ├── pywrapper.h
│   ├── symengine.pxd
│   ├── symengine_wrapper.cpython-312-x86_64-linux-gnu.so
│   └── symengine_wrapper.pxd
├── printing.py
├── __pycache__
│   ├── functions.cpython-312.pyc
│   ├── __init__.cpython-312.pyc
│   ├── printing.cpython-312.pyc
│   ├── sympy_compat.cpython-312.pyc
│   ├── test_utilities.cpython-312.pyc
│   └── utilities.cpython-312.pyc
├── sympy_compat.py
├── tests
│   ├── __init__.py
│   ├── __pycache__
│   │   ├── __init__.cpython-312.pyc
│   │   ├── test_arit.cpython-312.pyc
│   │   ├── test_cse.cpython-312.pyc
│   │   ├── test_dict_basic.cpython-312.pyc
│   │   ├── test_eval.cpython-312.pyc
│   │   ├── test_expr.cpython-312.pyc
│   │   ├── test_functions.cpython-312.pyc
│   │   ├── test_lambdify.cpython-312.pyc
│   │   ├── test_logic.cpython-312.pyc
│   │   ├── test_matrices.cpython-312.pyc
│   │   ├── test_ntheory.cpython-312.pyc
│   │   ├── test_number.cpython-312.pyc
│   │   ├── test_pickling.cpython-312.pyc
│   │   ├── test_printing.cpython-312.pyc
│   │   ├── test_sage.cpython-312.pyc
│   │   ├── test_series_expansion.cpython-312.pyc
│   │   ├── test_sets.cpython-312.pyc
│   │   ├── test_solve.cpython-312.pyc
│   │   ├── test_subs.cpython-312.pyc
│   │   ├── test_symbol.cpython-312.pyc
│   │   ├── test_sympify.cpython-312.pyc
│   │   ├── test_sympy_compat.cpython-312.pyc
│   │   ├── test_sympy_conv.cpython-312.pyc
│   │   └── test_var.cpython-312.pyc
│   ├── test_arit.py
│   ├── test_cse.py
│   ├── test_dict_basic.py
│   ├── test_eval.py
│   ├── test_expr.py
│   ├── test_functions.py
│   ├── test_lambdify.py
│   ├── test_logic.py
│   ├── test_matrices.py
│   ├── test_ntheory.py
│   ├── test_number.py
│   ├── test_pickling.py
│   ├── test_printing.py
│   ├── test_sage.py
│   ├── test_series_expansion.py
│   ├── test_sets.py
│   ├── test_solve.py
│   ├── test_subs.py
│   ├── test_symbol.py
│   ├── test_sympify.py
│   ├── test_sympy_compat.py
│   ├── test_sympy_conv.py
│   └── test_var.py
├── test_utilities.py
└── utilities.py
symengine-0.11.0.dist-info
├── AUTHORS
├── INSTALLER
├── LICENSE
├── METADATA
├── RECORD
├── REQUESTED
├── top_level.txt
└── WHEEL
symengine.libs
├── libflint-b75bea04.so.15.0.1
├── libgmp-ba3d8273.so.10.5.0
├── libmpc-5c556745.so.3.3.1
├── libmpfr-7f7b9c9c.so.6.2.1
└── libzstd-52a5ab61.so.1.5.5

[martinruefenacht]

Yes, I got the same with just a normal venv.

What I notice is that several directories in the site-packages are missing in the pdm version:

symengine
├── functions.py
├── __init__.py
├── lib
│   ├── __init__.py
│   └── __pycache__
│       └── __init__.cpython-312.pyc
├── printing.py
├── __pycache__
│   └── __init__.cpython-312.pyc
├── sympy_compat.py
├── tests
│   ├── __init__.py
│   ├── test_arit.py
│   ├── test_cse.py
│   ├── test_dict_basic.py
│   ├── test_eval.py
│   ├── test_expr.py
│   ├── test_functions.py
│   ├── test_lambdify.py
│   ├── test_logic.py
│   ├── test_matrices.py
│   ├── test_ntheory.py
│   ├── test_number.py
│   ├── test_pickling.py
│   ├── test_printing.py
│   ├── test_sage.py
│   ├── test_series_expansion.py
│   ├── test_sets.py
│   ├── test_solve.py
│   ├── test_subs.py
│   ├── test_symbol.py
│   ├── test_sympify.py
│   ├── test_sympy_compat.py
│   ├── test_sympy_conv.py
│   └── test_var.py
├── test_utilities.py
└── utilities.py

The lib/ folder is empty, symengine.libs doesn't get created, and symengine-0.11.

Perhaps this is more a question for pdm and poetry than symengine? They seem to be missing the external libraries.

[rikardn]

Could still be a problem with symengine. Does it work if you use poetry to install some other library that also uses compiled code, like for example numpy?

[martinruefenacht]

Yes, numpy works without issue...

[martinruefenacht]

I just discovered that the lib/, symengine-0.11.0.dist-info, symengine.libs are placed in the lib64 directory.

.venv/lib
└── python3.12
    └── site-packages
        ├── __pycache__
        │   └── _virtualenv.cpython-312.pyc
        ├── symengine
        │   ├── functions.py
        │   ├── __init__.py
        │   ├── lib
        │   │   └── __init__.py
        │   ├── printing.py
        │   ├── sympy_compat.py
        │   ├── tests
        │   │   ├── __init__.py
        │   │   ├── test_arit.py
        │   │   ├── test_cse.py
        │   │   ├── test_dict_basic.py
        │   │   ├── test_eval.py
        │   │   ├── test_expr.py
        │   │   ├── test_functions.py
        │   │   ├── test_lambdify.py
        │   │   ├── test_logic.py
        │   │   ├── test_matrices.py
        │   │   ├── test_ntheory.py
        │   │   ├── test_number.py
        │   │   ├── test_pickling.py
        │   │   ├── test_printing.py
        │   │   ├── test_sage.py
        │   │   ├── test_series_expansion.py
        │   │   ├── test_sets.py
        │   │   ├── test_solve.py
        │   │   ├── test_subs.py
        │   │   ├── test_symbol.py
        │   │   ├── test_sympify.py
        │   │   ├── test_sympy_compat.py
        │   │   ├── test_sympy_conv.py
        │   │   └── test_var.py
        │   ├── test_utilities.py
        │   └── utilities.py
        ├── _virtualenv.pth
        └── _virtualenv.py
.venv/lib64
└── python3.12
    └── site-packages
        ├── symengine
        │   └── lib
        │       ├── pywrapper.h
        │       ├── symengine.pxd
        │       ├── symengine_wrapper.cpython-312-x86_64-linux-gnu.so
        │       └── symengine_wrapper.pxd
        ├── symengine-0.11.0.dist-info
        │   ├── AUTHORS
        │   ├── LICENSE
        │   ├── METADATA
        │   ├── RECORD
        │   ├── top_level.txt
        │   └── WHEEL
        └── symengine.libs
            ├── libflint-b75bea04.so.15.0.1
            ├── libgmp-ba3d8273.so.10.5.0
            ├── libmpc-5c556745.so.3.3.1
            ├── libmpfr-7f7b9c9c.so.6.2.1
            └── libzstd-52a5ab61.so.1.5.5

Whereas numpy gets placed entirely in the lib64 site-packages not spread across.

[martinruefenacht]

And when I move all the files that are placed into the .venv/lib/python3.12/site-packages/ into the lib64 then everything works are expected.

[rikardn]

Good find @martinruefenacht! I think this is the crucial problem. Now we just need to figure out if this is caused by the symengine wheel being composed in a faulty way or if this is the fault of the installation procedure.

[martinruefenacht]

Well I am not familiar with this stuff. I unpacked the numpy and symengine wheels to have a look. What stood out was that the symengine has a .data directory and that contains what I see in the lib/ instead of the lib64/.

symengine
└── lib
    ├── pywrapper.h
    ├── symengine.pxd
    ├── symengine_wrapper.cpython-312-x86_64-linux-gnu.so
    └── symengine_wrapper.pxd
symengine-0.11.0.data
└── purelib
    └── symengine
        ├── functions.py
        ├── __init__.py
        ├── lib
        │   └── __init__.py
        ├── printing.py
        ├── sympy_compat.py
        ├── tests
        │   ├── __init__.py
        │   ├── test_arit.py
        │   ├── test_cse.py
        │   ├── test_dict_basic.py
        │   ├── test_eval.py
        │   ├── test_expr.py
        │   ├── test_functions.py
        │   ├── test_lambdify.py
        │   ├── test_logic.py
        │   ├── test_matrices.py
        │   ├── test_ntheory.py
        │   ├── test_number.py
        │   ├── test_pickling.py
        │   ├── test_printing.py
        │   ├── test_sage.py
        │   ├── test_series_expansion.py
        │   ├── test_sets.py
        │   ├── test_solve.py
        │   ├── test_subs.py
        │   ├── test_symbol.py
        │   ├── test_sympify.py
        │   ├── test_sympy_compat.py
        │   ├── test_sympy_conv.py
        │   └── test_var.py
        ├── test_utilities.py
        └── utilities.py
symengine-0.11.0.dist-info
├── AUTHORS
├── LICENSE
├── METADATA
├── RECORD
├── top_level.txt
└── WHEEL
symengine.libs
├── libflint-b75bea04.so.15.0.1
├── libgmp-ba3d8273.so.10.5.0
├── libmpc-5c556745.so.3.3.1
├── libmpfr-7f7b9c9c.so.6.2.1
└── libzstd-52a5ab61.so.1.5.5

[mrossinek]

I just ran into this myself while symengine is being installed as part of a tox workflow.
I am running Fedora 39. It is odd that everything works fine outside of tox.

I performed a git bisection to find the first bad commit based on the fact that I knew v0.9.2 to be working.
I made sure to always have the correct symengine version as well as compatible Cython installed. I used Python 3.10 since this is supported by all versions between 0.9 and 0.11.

The commit which introduced the failure to build as expected is: bc84086

I am not quite sure what to make of this but hopefully this helps in finding a solution.

[rikardn]

@mrossinek are you also using poetry?

[mrossinek]

No I am using pure pip for the installation. But the case in which it breaks is when this gets executed as part of tox

[rikardn]

Strange. I use symengine myself with pip and tox and that works fine. I wonder if it is the same cause as in the poetry case.

[martinruefenacht]

I am also on Fedora if that helps.

[mrossinek]

I had a colleague point out to me, that it could possibly be related to how the OS (in this case Fedora) handles lib vs lib64 in Python. I did not find much that would be helpful on this, hence I did the bisection.