python>=3.8 dll import fail under windows

[vincefn]

pyobjcryst now installs correctly under windows using pip install . (using the windows branch), but only works using python 3.7, and gets a "DLL load failed while importing _pyobcryst" on python>=3.8

This seems to be due to a change in python 3.8, see https://docs.python.org/3.8/whatsnew/3.8.html#bpo-36085-whatsnew

A workaround should be to insert the adequate paths in the DLL load path using os.add_dll_directory (https://docs.python.org/3.8/library/os.html#os.add_dll_directory), but it is not clear which paths are missing since the error message does not indicate which DLL dependency failed to import (that would be waaay too simple).

A relevant note for conda:
https://conda.io/projects/conda/en/latest/user-guide/troubleshooting.html#numpy-mkl-library-load-failed
.. but it seems using the CONDA_DLL_SEARCH_MODIFICATION_ENABLE does not make any difference

The following link may contain useful info:
https://stackoverflow.com/questions/65334494/python-c-extension-packaging-dll-along-with-pyd

is a package_data missing in the setup_args ?

Yet another possibility:
https://stackoverflow.com/questions/59330863/cant-import-dll-module-in-python/64472088#64472088

Another discussion:
Toblerity/Fiona#851
Toblerity/Fiona#1081

Yet another Relevant Link (Maybe):
https://stackoverflow.com/questions/56797584/why-does-static-version-of-the-boost-python-library-has-a-dependency-to-python-i

Another approach:
https://github.com/PixarAnimationStudios/USD/pull/1511/files

[vincefn]

Interestingly, the new builds (and tests) using pypy rather than cpython actually succeed for python 3.8 and 3.9 :-), despite the build being identical.

Another tool to use : dlltracer
https://pypi.org/project/dlltracer/

dlltracer results (using ctypes.CDLL to minimise what needs to be imported):

with python 3.7 (working):

(pyobjcryst37) C:\WINDOWS\system32>ipython
Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 05:35:01) [MSC v.1916 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 7.33.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys

In [2]: import dlltracer

In [3]: import ctypes

In [4]: with dlltracer.Trace(out=sys.stdout):
   ...:     ctypes.CDLL("C://Users//vince/mambaforge///envs/pyobjcryst37/lib//site-packages//pyobjcryst/_pyobjcryst.cp37-win_amd64.pyd")
   ...:
LoadLibrary \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst37\Lib\site-packages\pyobjcryst\_pyobjcryst.cp37-win_amd64.pyd
LoadLibrary \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst37\vcruntime140_1.dll
LoadLibrary \Device\HarddiskVolume8\Users\vince\mambaforge\pkgs\boost-1.78.0-py37h2cdfcc5_0\Library\bin\boost_python37.dll
LoadLibrary \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst37\msvcp140.dll

and with python 3.9:

(pyobjcryst) C:\Windows\System32>ipython
Python 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:51:29) [MSC v.1929 64 bit (AMD64)]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import sys

In [2]: import dlltracer

In [3]: import ctypes

In [4]: with dlltracer.Trace(out=sys.stdout):
   ...:     ctypes.CDLL("C://Users//vince/mambaforge///envs/pyobjcryst/lib/site-packages/pyobjcryst/_pyobjcryst.cp39-win_amd64.pyd")
   ...:
LoadLibrary \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst\Lib\site-packages\pyobjcryst\_pyobjcryst.cp39-win_amd64.pyd
LoadLibrary \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst\Library\bin\boost_python39.dll
LoadLibrary \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst\vcruntime140_1.dll
LoadLibrary \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst\msvcp140.dll
Failed \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst\Library\bin\boost_python39.dll
Failed \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst\msvcp140.dll
Failed \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst\vcruntime140_1.dll
Failed \Device\HarddiskVolume8\Users\vince\mambaforge\envs\pyobjcryst\Lib\site-packages\pyobjcryst\_pyobjcryst.cp39-win_amd64.pyd
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In [4], line 2
      1 with dlltracer.Trace(out=sys.stdout):
----> 2     ctypes.CDLL("C://Users//vince/mambaforge///envs/pyobjcryst/lib/site-packages/pyobjcryst/_pyobjcryst.cp39-win_amd64.pyd")

File ~\mambaforge\envs\pyobjcryst\lib\ctypes\__init__.py:374, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    371 self._FuncPtr = _FuncPtr
    373 if handle is None:
--> 374     self._handle = _dlopen(self._name, mode)
    375 else:
    376     self._handle = handle

OSError: [WinError 1114] Une routine d’initialisation d’une bibliothèque de liens dynamiques (DLL) a échoué

So it seems thatthe issue is not with the ability to find the required libraries (the paths are found), but rather something goes wrong when actually loading them.

Also note that the path for boost_python39.dll is mambaforge\pkgs\boostXXpy37...\...\boost_python37.dll in the succesfull import, but mambaforge\envs\pyobjcryst\Library\bin\boost_python39.dll in the other. Though there exists a boost_python39.dll in mambaforge\pkgs\boostXXpy39... but it looks identical to the one in site-packages... In fact, when removing the boost in the environment packages, the one in mambaforge\pkgs\boostXXpy39... is found but leads to the same 1114 error...

[vincefn]

What has been tried so far (just the headlines, many tests have been done...):

• using libobjcryst as a .dll or a .lib - no difference
• adding all required paths for the DLL
• using scons or pip to build

In the end, all fail.

What seems clear is that the issue is NOT actually with finding the path to the libraries to load. When done manually with the proper os.add_dll_directory, there is still an error when importing the library.

What is interesting is that using ctypes.CDLL() with winmode=1 or 0x20 'works':

import sys, dlltracer, ctypes
with dlltracer.Trace(out=sys.stdout):
    import ctypes
    r=ctypes.CDLL("C://Users//vince/mambaforge///envs/pyobjcryst311/lib//site-packages//pyobjcryst-2.2.6-py3.11-win-amd64.egg//pyobjcryst//_pyobjcryst.cp311-win_amd64.pyd", winmode=1)

but using winmode=1 then followed by import pyobjcryst._pyobjcryst leads to a crash. Using windmode=0x20 leads to the more regular import error.

So this would indicate that there is an issue with the built pyobjcryst extension - incorrectly exposed symbols ? The actual error may correspond to the exact step as tested before (using py39):

File ~\mambaforge\envs\pyobjcryst\lib\ctypes\__init__.py:374, in CDLL.__init__(self, name, mode, handle, use_errno, use_last_error, winmode)
    371 self._FuncPtr = _FuncPtr
    373 if handle is None:
--> 374     self._handle = _dlopen(self._name, mode)
    375 else:
    376     self._handle = handle

The winmode parameter is explained at: https://learn.microsoft.com/en-us/windows/win32/api/libloaderapi/nf-libloaderapi-loadlibraryexa. Both values (winmode=1 or 0x20) skip some initialisation steps during the import.

[vincefn]

Creating a new minimal project with just helpers.cpp, python_streambuf.cpp, general_ext.cpp, registerconverters.cpp it seems that the following line is triggering the DLL import error:

PyObject* pyobjcryst_ObjCrystException =
    PyErr_NewException((char*)"pyobjcryst.ObjCrystException", 0, 0);

...this requires commenting other lines using pyobjcryst_ObjCrystException but uncommenting that line only is sufficient to trigger the import error..

...keeping the PyObject* pyobjcryst_ObjCrystException at the same place and moving:

pyobjcryst_ObjCrystException =  PyErr_NewException((char*)"pyobjcryst.ObjCrystException", 0, 0);

into the wrap_registerconverters() function seems to solve the issue...