CUDA location requirements and documentation

[cmd-ntrf]

For CUDA-enabled software packages, our software environment relies on having driver libraries installed in the path /usr/lib64/nvidia as documented on Compute Canada Wiki.

The minor versions of the symbolic links we create in /usr/lib64/nvidia appear to no longer be needed by the software the install. The removal of libnvidia-fatbinaryloader.so.$version in recent CUDA releases is what made us question the need for minor versions.

On the presence of minor version @mboisson had this to say:

We do rely on /usr/lib64/nvidia/libcuda.so.$cuda_version to figure out which version of cuda drivers is installed, to hide or show specific versions of the cuda modules. But this is home made code and could be adjusted, i.e. function get_installed_cuda_driver_version at line 178 of /cvmfs/soft.computecanada.ca/config/lmod/SitePackage.lua

And the following suggestion

If the minor versions of the shared objects are no longer needed, we could just have a directory of symlinks distributed on cvmfs that would include only the major libraries

Some serious testing is needed, but it would simplify documentation as the current script we provide on Compute Canada wiki is out of date, and keeping it up-to-date is an ever boring moving target.

[mboisson]

As a counter argument to the symlink strategy, colleagues in the EESSI project noted that the location of the shared objects is different on different distributions. They could be in :

• /usr/lib64 (on CentOS7-8)
• /usr/lib64/nvidia (on older CentOS)
• /usr/lib/nvidia-xxx (on Ubuntu)

The symlink strategy would therefore need to include some layers, and possibly not work at all on Ubuntu (assuming those xxx are driver versions.

[mboisson]

From our meeting on 20210312:

• We could have a mix of symlinks (for libraries in /usr/lib64) + LD_LIBRARY_PATH (for when it is installed in different folders)
• Need to document minimum driver version when the dependency on libnvidia-fatbinaryloader.so.$version was dropped.
• Need to test OpenGL/EGL libraries
• It might change the search order (need to verify for name clashes)

[mboisson]

Issue #79 (once tested and implemented) provides a solution for this one.