Build issues with local CUDA installation

[adamjstewart]

Description

When building jaxlib with an externally installed copy of CUDA (something required by all package managers and HPC systems), I see the following error:

gcc: error: unrecognized command-line option '--cuda-path=external/cuda_nvcc'

It's possible I'm passing the wrong flags somewhere. I'm using:

> python3 build/build.py --enable_cuda --cuda_compute_capabilities=8.0 --bazel_options=--repo_env=LOCAL_CUDA_PATH=... --bazel_options=--repo_env=LOCAL_CUDNN_PATH=... --bazel_options=--repo_env=LOCAL_NCCL_PATH=...

(of course, with ... replaced by the actual paths)

System info (python version, jaxlib version, accelerator, etc.)

• Python: 3.11.9
• Jaxlib: 0.4.32
• GCC: 11.4.0

Build log

[ybaturina]

Hi @adamjstewart GCC compiler is not officially supported by JAX. I recommend using Clang. You can pass the clang path in --clang_path option.

[ybaturina]

If you absolutely need to use GCC, we have an experimental support that can be enabled like this:

--bazel_options=--action_env=CUDA_NVCC="1" --bazel_options=--@local_config_cuda//:cuda_compiler=nvcc

[adamjstewart]

I tried adding these flags but I still see the exact same error:

gcc: error: unrecognized command-line option '--cuda-path=external/cuda_nvcc'

[ybaturina]

Would you paste the full stack trace here please? I'd like to make sure that CUDA_NVCC value is recognized by Bazel.

[adamjstewart]

Here you go:

• Build log

[ybaturina]

Hmm, one more suggestion: try this
```--bazel_options=--action_env=TF_NVCC_CLANG="1" --bazel_options=--@local_config_cuda//:cuda_compiler=nvcc`

The reason why your build fails is that GCC is unable to compile CUDA dependencies, it should be done with NVCC compiler.

[adamjstewart]

Still the same issue:

gcc: error: unrecognized command-line option ‘--cuda-path=external/cuda_nvcc’

[ybaturina]

This is what I've tried:

python3.10 build/build.py --enable_cuda --use_clang=false --bazel_options=--repo_env=CC="/dt9/usr/bin/gcc" --bazel_options=--repo_env=TF_SYSROOT="/dt9" --bazel_options=--action_env=CUDA_NVCC="1" --bazel_options=--@local_config_cuda//:cuda_compiler=nvcc

The subcommand I got:

SUBCOMMAND: # //jaxlib:cpu_feature_guard.so [action 'Compiling jaxlib/cpu_feature_guard.c', configuration: 988f5a730e2bd9c88c71efcc9c7f0d36ad2ec3c5f71c922aabaf7614ff994b0f, execution platform: @local_execution_config_platform//:platform]
(cd /home/ybaturina/.cache/bazel/_bazel_ybaturina/ead9107e8e47a1c42911a02736d63d03/execroot/__main__ && \
  exec env - \
    CUDA_NVCC=1 \
    PATH=/home/kbuilder/.local/bin:/usr/local/bin/python3.10:/home/ybaturina/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin \
    PWD=/proc/self/cwd \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/k8-opt/bin/jaxlib/_objs/cpu_feature_guard.so/cpu_feature_guard.pic.d '-frandom-seed=bazel-out/k8-opt/bin/jaxlib/_objs/cpu_feature_guard.so/cpu_feature_guard.pic.o' '-DBAZEL_CURRENT_REPOSITORY=""' -iquote . -iquote bazel-out/k8-opt/bin -iquote external/python_x86_64-unknown-linux-gnu -iquote bazel-out/k8-opt/bin/external/python_x86_64-unknown-linux-gnu -isystem external/python_x86_64-unknown-linux-gnu/include -isystem bazel-out/k8-opt/bin/external/python_x86_64-unknown-linux-gnu/include -isystem external/python_x86_64-unknown-linux-gnu/include/python3.10 -isystem bazel-out/k8-opt/bin/external/python_x86_64-unknown-linux-gnu/include/python3.10 -isystem external/python_x86_64-unknown-linux-gnu/include/python3.10m -isystem bazel-out/k8-opt/bin/external/python_x86_64-unknown-linux-gnu/include/python3.10m -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIC -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '-fvisibility=hidden' -Wno-sign-compare -Wno-unknown-warning-option -Wno-stringop-truncation -Wno-array-parameter '-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir.' -mavx -fno-strict-aliasing -fexceptions '-fvisibility=hidden' '--sysroot=/dt9' -c jaxlib/cpu_feature_guard.c -o bazel-out/k8-opt/bin/jaxlib/_objs/cpu_feature_guard.so/cpu_feature_guard.pic.o)

I didn't get the --cuda_path option passed to the NVCC compiler.

I assume that something in the environment variables on your machine messes up the subcommand configuration.
Since JAX doesn't support GCC compilation officially, I strongly recommend using clang for the compilation.

[daskol]

There is the --cuda-path issue with GCC for me as well.

Alternatively, I tried to build it with Clang and local CUDA, CUDNN, and NCCL but other issues occure.

In file included from external/xla/xla/tsl/cuda/cudnn_stub.cc:16:
In file included from external/com_google_absl/absl/container/flat_hash_map.h:38:
In file included from external/com_google_absl/absl/algorithm/container.h:43:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/algorithm:61:
In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/bits/stl_algo.h:71:
/usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/cstdlib:79:15: fatal error: 'stdlib.h' file not found
   79 | #include_next <stdlib.h>
      |               ^~~~~~~~~~
1 error generated.

Specifically, I run bazel directly as follows.

build/bazel-6.5.0-linux-x86_64 run --verbose_failures=true \
    --repo_env=LOCAL_CUDA_PATH=/opt/cuda \
    --repo_env=LOCAL_CUDNN_PATH=/usr \
    --repo_env=LOCAL_NCCL_PATH=/usr \
    //jaxlib/tools:build_wheel -- \
    --output_path=$PWD/dist --cpu=x86_64 \
    --jaxlib_git_hash=78ade74d695407306461718a6d73cfed89b4d972

Also, I add the following .bazelrc.user to the repository root.

# .bazelrc.user
build --strategy=Genrule=standalone
build --action_env CLANG_COMPILER_PATH="/usr/bin/clang-18"
build --repo_env CC="/usr/bin/clang-18"
build --repo_env BAZEL_COMPILER="/usr/bin/clang-18"
build --copt=-Wno-error=unused-command-line-argument
build --copt=-Wno-gnu-offsetof-extensions
build --config=avx_posix
build --config=mkl_open_source_only
build --config=cuda
build --config=nvcc_clang
build --action_env=CLANG_CUDA_COMPILER_PATH=/usr/bin/clang-18
build --repo_env HERMETIC_PYTHON_VERSION="3.12"

Dependency versions follow.

$ pacman -Qs '(cuda|cudnn|clang)'
local/clang 18.1.8-4
    C language family frontend for LLVM
local/compiler-rt 18.1.8-1
    Compiler runtime libraries for clang
local/cuda 12.6.2-2
    NVIDIA's GPU programming toolkit
local/cudnn 9.2.1.18-1
    NVIDIA CUDA Deep Neural Network library

[ybaturina]

This looks like a problem with GCC installation.
If you run clang -v, then you'll see smth like this:
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-linux-gnu/14

Looking at the error above, I suggest running this command:
find /usr/bin/../include -name "stdlib.h
If the file is not found in GCC v14, that means you'll need to install missing headers and run sudo apt install g++-14

[daskol]

I reproduce the issue for jaxlib from 0.4.32, 0.4.33, and 0.4.34 with clang-14 and clang-18 (depends on gcc and gcc-libs 14.2.1+r134+gab884fffe3fc-1). Also, cuda package depends on gcc-13 (it's Arch).

$ /usr/lib/llvm14/bin/clang-14 -v
clang version 14.0.6
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/lib/llvm14/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/13.3.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/14.2.1
Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/13.3.0
Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/14.2.1
Selected GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/14.2.1
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64

Also I have appended -v option to failed command crosstool_wrapper_driver_is_not_gcc. It displays system search which have stdlib.h.

$ (cd ... && .../crosstool_wrapper_driver_is_not_gcc ... -v)
Selected GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/14.2.1
 ...
 /usr/lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1
 /usr/lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/x86_64-pc-linux-gnu
 /usr/lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/backward
 /usr/lib/llvm14/lib/clang/14.0.6/include
 /usr/local/include
 /usr/include
End of search list.
external/xla/xla/tsl/cuda/cupti_stub.cc:16:10: fatal error: 'third_party/gpus/cuda/extras/CUPTI/include/cupti.h' file not found
#include "third_party/gpus/cuda/extras/CUPTI/include/cupti.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
$ ls -l /usr/lib64/gcc/x86_64-pc-linux-gnu/14.2.1/../../../../include/c++/14.2.1/stdlib.h 
-rw-r--r-- 1 root root 2.3K Sep 10 13:07 /usr/lib64/gcc/x86_64-pc-linux- gnu/14.2.1/../../../../include/c++/14.2.1/stdlib.h

But it is a bit odd that now third_party/gpus/cuda/extras/CUPTI/include/cupti.h is not found. 🤯

[daskol]

There is indeed no directory third_party/gpus. I didn't find third_party/gpus/cuda/extras/CUPTI with the command below. 🤯

find -L bazel-jax-jax-v0.4.34 -name 'CUPTI'

UPD Is it upstream issue (XLA)?

[ybaturina]

Would you check if your local CUDA installation has CUPTI headers please? Specifically, the following headers should be present:
https://github.com/openxla/xla/blob/main/third_party/tsl/third_party/gpus/cuda/hermetic/cuda_cupti.BUILD.tpl#L21-L58

Also please check that the structure of the local CUDA/CUDNN/NCCL dirs is exactly the same as described here.