Question about correct usage inside a conda env

[rjanvier]

Hi,
I have a scikit-build-based / nanobind project .

When I try to build it into an activated conda env where the libstd++ conda env lib is different from the one of my system, the build system takes into account my system libstd++ at link stage. As a result it could crash at import time if my system libstdc++ ABI is > to the one in the conda env (../lib/libstdc++.so.6: version 'GLIBCXX_3.4.30' not found (required by....). One option is to repair the wheel, but it's not optimal. I noticed that Setuptools based builds seems to be aware of running into a conda env by setting correct -L and -rpath flags. So my question is pretty simple : am I missing something in sk-build-core configuration or is it a design choice and in this case should I rely on "pure" cmake mecanisms to detect and handle the compilation into an activated conda env?

thanks in advance!

[drprojects]

Hello,

Would it be possible to have an update on this, please ?

Thanks in advance !

[henryiii]

Sorry, can try to look into it. I expect we should be reading more info from sysconfig and setting CMake values for it. How easy is it to reproduce? Can I just use a modern ubuntu docker container? What conda packages should I grab?

[drprojects]

Thanks a lot @henryiii for the quick reply !

I think @rjanvier will explain better than I, he is the mastermind behind our project's build 😇

Still, as per reproducibility, I think any Ubuntu docker container should do the trick, with miniconda installed. Below is a MRE to reproduce the error described above by @rjanvier.

Create a conda environment with python = ">=3.8,<3.13":

conda create -n pgeof_env python=3.9
conda activate pgeof_env

Within the environment, run:

pip install git+https://github.com/drprojects/point_geometric_features.git

Here is what the error looks like on my end:

Collecting git+https://github.com/drprojects/point_geometric_features.git
  Cloning https://github.com/drprojects/point_geometric_features.git to /tmp/pip-req-build-zz3pgtat
  Running command git clone --filter=blob:none --quiet https://github.com/drprojects/point_geometric_features.git /tmp/pip-req-build-zz3pgtat
  Resolved https://github.com/drprojects/point_geometric_features.git to commit a8823a9a6fda33a8ff1282ccddab1a0a9b2716a2
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
  Preparing metadata (pyproject.toml) ... done
Collecting numpy<2,>=1.7 (from pgeof==0.3.0)
  Downloading numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Downloading numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 18.2/18.2 MB 95.6 MB/s eta 0:00:00
Building wheels for collected packages: pgeof
  Building wheel for pgeof (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for pgeof (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [54 lines of output]
      *** scikit-build-core 0.10.7 using CMake 3.30.5 (wheel)
      *** Configuring CMake...
      loading initial cache file build/cp39-cp39-linux_x86_64/CMakeInit.txt
      -- The C compiler identification is GNU 11.4.0
      -- The CXX compiler identification is GNU 11.4.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/gcc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/g++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- Found Python: /home/damien/miniconda3/envs/pgeof_env/bin/python (found suitable version "3.9.20", minimum required is "3.8") found components: Interpreter Development.Module
      -- Configuring done (0.3s)
      -- Generating done (0.0s)
      -- Build files have been written to: /tmp/pip-req-build-zz3pgtat/build/cp39-cp39-linux_x86_64
      *** Building project with Ninja...
      [1/14] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/src/trampoline.cpp.o
      [2/14] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/src/nb_internals.cpp.o
      [3/14] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/src/implicit.cpp.o
      [4/14] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/src/nb_static_property.cpp.o
      [5/14] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/src/nb_ndarray.cpp.o
      [6/14] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/src/common.cpp.o
      [7/14] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/src/error.cpp.o
      [8/14] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/src/nb_func.cpp.o
      [9/14] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/src/nb_enum.cpp.o
      [10/14] Building CXX object CMakeFiles/nanobind-static.dir/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/src/nb_type.cpp.o
      [11/14] Linking CXX static library libnanobind-static.a
      [12/14] Building CXX object CMakeFiles/pgeof_ext.dir/src/pgeof_ext.cpp.o
      [13/14] Linking CXX shared module pgeof_ext.cpython-39-x86_64-linux-gnu.so
      [14/14] Generating py.typed, pgeof_ext.pyi
      FAILED: py.typed pgeof_ext.pyi /tmp/pip-req-build-zz3pgtat/build/cp39-cp39-linux_x86_64/py.typed /tmp/pip-req-build-zz3pgtat/build/cp39-cp39-linux_x86_64/pgeof_ext.pyi
      cd /tmp/pip-req-build-zz3pgtat/build/cp39-cp39-linux_x86_64 && /home/damien/miniconda3/envs/pgeof_env/bin/python /tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/stubgen.py -q -i /tmp/pip-req-build-zz3pgtat/build/cp39-cp39-linux_x86_64 -M py.typed -m pgeof_ext -o pgeof_ext.pyi
      Traceback (most recent call last):
        File "/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/stubgen.py", line 1411, in <module>
          main()
        File "/tmp/pip-build-env-t1c5j02y/overlay/lib/python3.9/site-packages/nanobind/stubgen.py", line 1344, in main
          mod_imported = importlib.import_module(mod)
        File "/home/damien/miniconda3/envs/pgeof_env/lib/python3.9/importlib/__init__.py", line 127, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
        File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
        File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 666, in _load_unlocked
        File "<frozen importlib._bootstrap>", line 565, in module_from_spec
        File "<frozen importlib._bootstrap_external>", line 1173, in create_module
        File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
      ImportError: /home/damien/miniconda3/envs/pgeof_env/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /tmp/pip-req-build-zz3pgtat/build/cp39-cp39-linux_x86_64/pgeof_ext.cpython-39-x86_64-linux-gnu.so)
      ninja: build stopped: subcommand failed.
      
      *** CMake build failed
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pgeof
Failed to build pgeof
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (pgeof)

[henryiii]

Where are you installing the conda compilers?

[drprojects]

In my specific case, I am on a linux machine and miniconda3 is located as set by default by the installer: in my home directory (i.e. /home/user/). Does that answer your question ?

[LecrisUT]

Hmm, something is strange:

Check for working C compiler: /usr/bin/gcc - skipped
Check for working CXX compiler: /usr/bin/g++ - skipped
ImportError: /home/damien/miniconda3/envs/pgeof_env/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.30' not found

For some reason the build is using the system's compiler, but the python interpreter was built with the one from miniconda.

[rjanvier]

@drprojects yes but maybe you have to depends on a c++ compiler (https://anaconda.org/conda-forge/cxx-compiler). It makes sense for me but what bother me is that in the same context, if I remember correctly, setuptools based extensions produce "correct" packages but not sk-build-core. it's not a criticism about sk-build-core, I just want to understand what is wrong with the setup of our package.

[henryiii]

You didn't install the conda compilers. Conda has its own compiling stack. You only have the system compiler, so I don't see how setuptools could have passed; what is it setting the paths to?

(In general, conda wasn't designed to have people build code against it, it was designed to only install pre-built conda recipes or maybe build your own conda recipes)