Description
Description
Currently the wheels in which we are shipping native libraries have a conditional load logic that first searches for a library on the system and loads that before loading one in the wheel. As I've discussed in other channels, there is no way to support this kind of switching safely in general. If two different copies of a library exist and different packages make different assumptions when loading, the first one loaded wins and will be used in all cases. I'm working on more general solutions to this problem, but in the meantime we at minimum need to reconsider our default behavior, which is causing users confusion now. If we assume that for rapids native libraries the only consumers of those wheels are other rapids wheels that we control and are therefore consistently behaved, we can make some more sane default choices for the user. In general, we should assume that our packages are being installed by end users unfamiliar with packaging. The default behavior should support this, meaning that by default we probably want the cudf pip wheel to use libcudf from a wheel even if a system libcudf library exists by default. Conversely, more advanced users may install cudf into an environment (like a container) where the library already exists and should be reused. We should enable that use case using an environment variable. A global configuration is another option, but I'd like to reserve that for when I have a more general solution available.
We should also switch all our library loads to use RTLD_LOCAL instead of RTLD_GLOBAL to be safe. Global loads can easily lead to symbol clashes.
Benefits of this work
- adds flexibility for different preferred methods of library-loading
- makes library-loading safer (i.e. less likely to cause conflicts)
Acceptance Criteria
For all relevant RAPIDS libraries:
- preference for system vs. wheel shared library is configurable via an environment
- shared library loading defaults to preferring the library from the wheel itself
- all library loads are done with
RTLD_LOCAL
Approach
N/A
Notes
N/A
Task Lists
NOTE: individual-repo devcontainers changes are only needed for the ucx wheels, see rapidsai/devcontainers#421 (comment).