Import tests fail on dynamic builds of packages that depend on external libraries on linux

[couteau]

I am building an application (QGIS) that depends on python packages that, in turn, rely on external c/c++ libraries. It uses a custom triplet that builds shared libraries for most dependencies. I have successfully built the application on MacOS, but when I try to build on linux, the import test for these packages fails because python/the package can't find the external libraries on which it depends. If I instead build using static linking of dependencies, the tests work fine.

To give a specific example where I am encountering this issue, the py-numpy package depends on OpenBlas. Building an application that depends on py-numpy successfully builds all the dependencies (python, openblas, lapack, etc) and successfully builds the numpy package, but vcpkg it attempts to run vcpkg_python_test_import, the import fails with the message

ImportError: libopenblas.so.0: cannot open shared object file: No such file or directory

The problem appears to be that libopenblas.so.* are installed to the CURRENT_INSTALLED_DIR/lib directory, but that directory is not on the LD_LIBRARY_PATH when the test runs. vcpkg_python_test_import adds the CURRENT_PACKAGES_DIR to LD_LIBRARY_PATH, but that directory only includes files built by the package being tested, not its dependencies from other packages. If I run the import_test.py for numpy from the command line after setting LD_LIBRARY_PATH to the directory corresponding to CURRENT_INSTALLED_DIR, the import works. And if I make a local copy of the vcpkg-python-scripts port in my overlay-ports folder and add CURRENT_INSTALLED_DIR/lib to the LD_LIBRARY_PATH in vcpkg_python_test_import, the import works.

For reference, the triplet for dynamic linking of dependencies I'm using has the following contents:

set(VCPKG_TARGET_ARCHITECTURE x64)
set(VCPKG_CRT_LINKAGE dynamic)
set(VCPKG_LIBRARY_LINKAGE dynamic)

set(VCPKG_CMAKE_SYSTEM_NAME Linux)
set(VCPKG_BUILD_TYPE release)

set(VCPKG_FIXUP_ELF_RPATH ON)

I don't know if this should be considered a bug in vcpkg_python_test_import or if there is some way I can work around this issue with my cmake or vcpkg setup.

[m-kuhn]

@couteau thanks for reporting this.

If you manually add ${VCPKG_INSTALLED_DIR}/lib to LD_LIBRARY_PATH does it work?

You can temporarily override ports if you copy them from the ports folder here to QGIS/vcpkg/ports to test.

The things in here were tested with windows and osx

[couteau]

I'm not sure what you mean by manually, but yes, that's where I created a new version of thevcpkg-python-scripts port and added ${VCPKG_INSTALLED_DIR}/lib to LD_LIBRARY_PATH there in the vcpkg_python_test_import function, and yes, that did solve the issue. From looking at the original code for vcpkg_python_test_import, it doesn't look like modifying LD_LIBRARY_PATH in the environment before running cmake or in the py-numpy portfile would work because vcpkg_python_test_import replaces and then restores whatever LD_LIBRARY_PATH is set to when it runs. So it looks like modifying the vcpkg-python-scripts port is the only way to fix it, and that's working for me at the moment.

[m-kuhn]

So it looks like modifying the vcpkg-python-scripts port is the only way to fix it, and that's working for me at the moment.

And that's the proper fix. Would you mind opening a pull request?