Latest changes to statically link `libpython` breaks usage of packages like `python-mscl`

[harshil21]

Hello!

I noticed that after upgrading uv to 0.7.6 and reinstalling python (uv python install --reinstall), the import of a package like python-mscl breaks. Here's a traceback:

Python 3.13.3 (main, May 17 2025, 13:34:03) [GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from python_mscl import mscl
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    from python_mscl import mscl
  File "/home/pi/AirbrakesV2/.venv/lib/python3.13/site-packages/python_mscl/__init__.py", line 1, in <module>
    from . import mscl
  File "/home/pi/AirbrakesV2/.venv/lib/python3.13/site-packages/python_mscl/mscl.py", line 13, in <module>
    from . import _mscl
ImportError: libpython3.13.so.1.0: cannot open shared object file: No such file or directory

---

Since I'm also the maintainer of python-mscl, I can give some more insight on how that package works:

My package is just a wrapper around https://github.com/LORD-MicroStrain/MSCL, aiming to simplify the installation procedure of the library, which is a little convoluted and not easy to setup on new machines - link.

So I am essentially just bundling the .so and .py files they provide into a package for every python version and architecture. And this just works on import, since it can find the .so and .py files in the same directory.

---

So I'm not sure if there's something wrong with how I'm packaging this library, and I have to correct it somehow, or if this was a problem with how my python was built. Since this happened immediately after reinstalling my python, and I saw the release note which mentioned this change, I thought I should open an issue to seek some clarification.

[zanieb]

Thank you for the report! cc @geofft who is the expert here.

[geofft]

Thanks, can take a look in a bit - my initial instinct is that this is probably something that should be tweaked in your build (per Python upstream guidance), but I am curious if you think this is breaking badly enough that we should consider rolling this back at least for a bit.

[harshil21]

should be tweaked in your build (per Python upstream guidance)

I'm not sure on how or what I can change to make this work (I don't have much experience working with extension modules). Would I have to rebuild from source and do something differently?

but I am curious if you think this is breaking badly enough that we should consider rolling this back at least for a bit.

I'm not fully qualified to answer that, but I would mention that most python distributions allow dynamic loading of extension modules at runtime, so this might be a surprise to some users.

I'm also curious on the cited performance gain from statically linking libpython. Were there any benchmarks run? What kind of improvements do we see?

[zanieb]

Were there any benchmarks run? What kind of improvements do we see?

We did some benchmarks prior to working on that change. It's like 15-25%, though it's platform dependent.

[harshil21]

Here's an insight - This issue does not occur with x86_64 builds. That import error seems to only happen on the raspberry pi, which is on aarch64.

So this might be something specific only to that architecture.

It's like 15-25%, though it's platform dependent.

I can also confirm that there is definitely some improvement, I'm seeing about 5-10% improvement on my specific workflow.

[vyasr]

We just ran into this as well in rapidsai/dask-upstream-testing#56.

I am curious if you think this is breaking badly enough that we should consider rolling this back at least for a bit.

@geofft I think it would probably be helpful to give a bit of lead time to projects to adapt to this. I'm guessing that many projects have been linking to libpython without recognizing the pitfalls of that. I am not entirely sure how we are hitting this in our case since we run auditwheel on our wheels, so I'll have to take another look later today and see if there are edge cases that auditwheel is letting through that might be exacerbating the frequency of this problem (or if we're simply misusing it somehow). In general, including libpython linkage seems like an easy mistake to make. For projects using CMake to build against Python, though, it will require a bit of work since you will have to move away from a typical target_link_libraries(... Python3::Python) to instead extract include dirs and only using those. That will be a nontrivial change for most people who are not regular CMake practitioners, so some grace period would be appreciate. I imagine Meson or Bazel builds will have similar issues.

That being said, I do think it is a mistake. To respond to the first half of your comment:

Thanks, can take a look in a bit - my initial instinct is that this is probably something that should be tweaked in your build (per Python upstream guidance)

Yes, I agree that other projects should likely be fixing this. libpython linkage is explicitly forbidden by the manylinux standard to support exactly this kind of use case if I understand correctly.

I'm not sure on how or what I can change to make this work (I don't have much experience working with extension modules). Would I have to rebuild from source and do something differently?

@harshil21 Yes, the correct solution is to avoid linking to libpython altogether. The GNU linker allows undefined symbols by default when building shared libraries, so simply not including libpython on the link line when building should be sufficient. The Python interpreter will load up all the missing symbols at runtime so your package should work without that linkage. In your case, it sounds like you are repackaging pre-built binaries. If that is the case, you can modify them with patchelf. Something like patchelf --remove-needed libpython3.13.so.1.0 ${YOUR_LIBRARY_HERE} should do the trick.

[geofft]

Thanks all! To put it in slightly more words, it looks like @harshil21 is maintaining a project that builds a wheel but the actual extension module in that wheel is built by another project, somewhere around https://github.com/LORD-MicroStrain/MSCL/blob/v67.1.0/MSCL/CMakeLists.txt . I suspect this is "easy" to fix directly in that CMake code if they will merge and do a release, or yeah you can run patchelf after the fact as outlined above, but yes, maybe we should do a more controlled rollout on the python-build-standalone side. Let me think a bit and check with a few others.

(Also, I stated this on your issue but for anyone else - if you want to discuss in a slightly more real-time forum, I'm in the Astral and PyPI Discords, feel free to at-mention me.)

[vyasr]

It looks like auditwheel is currently set up to ignore libpython linkage when checking for external dependencies. That's a problem for us here because it means that projects that have been relying on auditwheel to enforce manylinux compliance have been getting a free pass on this particular violation. I opened pypa/auditwheel#589 to discuss whether auditwheel's behavior should be changed.

In the interim, my suggestion is that python-build-standalone roll back this change and give a bit of lead time to projects to update before bringing back the static linkage. My guess is that there are a large number of projects relying on auditwheel that assume they are doing the right thing, and some of them may not be.

I'm also not sure how this will play on other platforms where there has never been a standard like manylinux to codify this information since there were de facto realities imposed by the OS. Is it possible to have an OS X distribution without Python available? If not, and if pyenv and similar tools always produce environments with libpython.so, then prior to python-build-standalone there may have been no real environments where this was a problem.

[geofft]

A workaround if you're broken by this in the short term is

uvx uv@0.7.5 python install --reinstall

which will grab the versions of python-build-standalone that were current in the previous uv release, i.e., the most recent versions with a dynamic libpython, and unpack those.

If this starts causing significantly more breakage, we can revert (we can pretty quickly put out a new release of uv that reverts the python-build-standalone versions, so we go back to known-good builds). I am working through some options to let us roll forward, i.e., deliver an interpreter that statically links libpython but is still compatible with these sorts of existing extensions.

In particular, I think I know what's going on with Debian/Ubuntu and conda-forge, which ship Pythons that are (like ours) statically linking libpython, and why the two libraries mentioned so far in this bug are fine in practice on those Pythons but not ours, which I'll expand on in the next comment (actually I am not sure I have the details 100% right so I'm going to work on some code first :) ).