Skip to content

Commit

Permalink
Switch to symlink
Browse files Browse the repository at this point in the history
  • Loading branch information
@mc-nv
mc-nv committed Oct 11, 2022
1 parent 19ec104 commit d6151d2
Showing 1 changed file with 3 additions and 3 deletions.
6 changes: 3 additions & 3 deletions build.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1101,9 +1101,9 @@ def dockerfile_prepare_container_linux(argmap, backends, enable_gpu,
COPY --from=min_container /usr/local/cuda/lib64/stubs/libcublasLt.so /usr/local/cuda/lib64/stubs/libcublasLt.so.11
RUN mkdir -p /usr/local/cuda/targets/{cuda_arch}-linux/lib
COPY --from=min_container /usr/local/cuda-11.8/targets/{cuda_arch}-linux/lib/libcudart.so.11.0 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
COPY --from=min_container /usr/local/cuda-11.8/targets/{cuda_arch}-linux/lib/libcupti.so.11.8 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
COPY --from=min_container /usr/local/cuda-11.8/targets/{cuda_arch}-linux/lib/libnvToolsExt.so.1 /usr/local/cuda/targets/{cuda_arch}-linux/lib/.
COPY --from=min_container /usr/local/cuda/targets/{cuda_arch}-linux/lib/libcudart.so.* /usr/local/cuda/targets/{cuda_arch}-linux/lib/.

This comment has been minimized.

Sign in to view

Copy link
@rmccorm4

rmccorm4 Oct 11, 2022 •
edited
Loading

Collaborator

I think this will be OK as long as the symlinks transfer properly as well.

I was going to suggest we could simply use the highest level indirection like libcudart.so which ultimately points to the specific version, but I believe sometimes applications look specifically for the versioned library file (ex: libcudart.so.11.0), so it may be better to copy them all.

example from 22.09-py3-min:

root@rmccormick-dt:/# ls -ls /usr/local/cuda/targets/x86_64-linux/lib/libcudart*
   0 lrwxrwxrwx 1 root root      17 Sep 21 18:46 /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so -> libcudart.so.11.0
   0 lrwxrwxrwx 1 root root      20 Sep 21 18:46 /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.11.0 -> libcudart.so.11.8.89
 664 -rw-r--r-- 1 root root  679264 Sep 21 18:46 /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.11.8.89
1172 -rw-r--r-- 1 root root 1198880 Sep 21 18:46 /usr/local/cuda/targets/x86_64-linux/lib/libcudart_static.a

See above that libcudart.so -> libcudart.so.11.0 -> libcudart.so.11.8.89 - which just goes up one step of symlink indirection from what we had previously pointing to libcudart.so.11.0.

This comment has been minimized.

Sign in to view

Copy link
@mc-nv

mc-nv Oct 11, 2022

Author Contributor

To be honest I don't see any reason to have this step, as those libraries are shipped with a CUDA container, and this container is our base image, also CUDA base container uses alternatives to handle proper location for the libraries...
I would vote to remove that statement

% docker run --rm -it cuda:11.8-devel-ubuntu20.04--22.10 bash -c "ls -l  /usr/local/cuda/lib64/libcudar*"

lrwxrwxrwx 1 root root      17 Sep 21 18:45 /usr/local/cuda/lib64/libcudart.so -> libcudart.so.11.0
lrwxrwxrwx 1 root root      20 Sep 21 18:45 /usr/local/cuda/lib64/libcudart.so.11.0 -> libcudart.so.11.8.89
-rw-r--r-- 1 root root  670888 Sep 21 18:45 /usr/local/cuda/lib64/libcudart.so.11.8.89
-rw-r--r-- 1 root root 1058730 Sep 21 18:45 /usr/local/cuda/lib64/libcudart_static.a

This comment has been minimized.

Sign in to view

Copy link
@mc-nv

mc-nv Oct 12, 2022

Author Contributor

According to my findings the size of the container won't be impacted in fact.

ls -lih  /usr/local/cuda/lib64/libcudart.so* /usr/local/cuda/lib64/libcupti.so.*  /usr/local/cuda/lib64/libnvToolsExt.so.*
31337036 lrwxrwxrwx 1 root root   17 Sep 21 18:45 /usr/local/cuda/lib64/libcudart.so -> libcudart.so.11.0
31337037 lrwxrwxrwx 1 root root   20 Sep 21 18:45 /usr/local/cuda/lib64/libcudart.so.11.0 -> libcudart.so.11.8.89
31337038 -rw-r--r-- 1 root root 656K Sep 21 18:45 /usr/local/cuda/lib64/libcudart.so.11.8.89
31345510 lrwxrwxrwx 1 root root   20 Aug 27 04:14 /usr/local/cuda/lib64/libcupti.so.11.8 -> libcupti.so.2022.3.0
31345511 -rw-r--r-- 1 root root 6.1M Aug 27 04:14 /usr/local/cuda/lib64/libcupti.so.2022.3.0
31337042 lrwxrwxrwx 1 root root   22 Aug 25 23:26 /usr/local/cuda/lib64/libnvToolsExt.so.1 -> libnvToolsExt.so.1.0.0
31337043 -rw-r--r-- 1 root root  44K Aug 25 23:26 /usr/local/cuda/lib64/libnvToolsExt.so.1.0.0

CC: @jbkyang-nvi
COPY --from=min_container /usr/local/cuda/targets/{cuda_arch}-linux/lib/libcupti.so.* /usr/local/cuda/targets/{cuda_arch}-linux/lib/.

This comment has been minimized.

Sign in to view

Copy link
@rmccorm4

rmccorm4 Oct 11, 2022

Collaborator

Same as above:

root@rmccormick-dt:/# ls -ls /usr/local/cuda/targets/x86_64-linux/lib/libcupti* 
   0 lrwxrwxrwx 1 root root      16 Aug 27 00:48 /usr/local/cuda/targets/x86_64-linux/lib/libcupti.so -> libcupti.so.11.8
   0 lrwxrwxrwx 1 root root      20 Aug 27 00:48 /usr/local/cuda/targets/x86_64-linux/lib/libcupti.so.11.8 -> libcupti.so.2022.3.0
7032 -rw-r--r-- 1 root root 7199856 Aug 27 00:48 /usr/local/cuda/targets/x86_64-linux/lib/libcupti.so.2022.3.0
COPY --from=min_container /usr/local/cuda/targets/{cuda_arch}-linux/lib/libnvToolsExt.so.* /usr/local/cuda/targets/{cuda_arch}-linux/lib/.

This comment has been minimized.

Sign in to view

Copy link
@rmccorm4

rmccorm4 Oct 11, 2022

Collaborator

Same as above:

root@rmccormick-dt:/# ls -ls /usr/local/cuda/targets/x86_64-linux/lib/libnvToolsExt*
 0 lrwxrwxrwx 1 root root    18 Aug 25 23:25 /usr/local/cuda/targets/x86_64-linux/lib/libnvToolsExt.so -> libnvToolsExt.so.1
 0 lrwxrwxrwx 1 root root    22 Aug 25 23:25 /usr/local/cuda/targets/x86_64-linux/lib/libnvToolsExt.so.1 -> libnvToolsExt.so.1.0.0
40 -rw-r--r-- 1 root root 40136 Aug 25 23:25 /usr/local/cuda/targets/x86_64-linux/lib/libnvToolsExt.so.1.0.0
COPY --from=min_container /usr/lib/{libs_arch}-linux-gnu/libcudnn.so.8 /usr/lib/{libs_arch}-linux-gnu/libcudnn.so.8
Expand Down

0 comments on commit d6151d2

Please sign in to comment.