cuda/cuda_config.h missing when compiling custom ops with nvcc

[pronobis]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  Yes, see below.
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Ubuntu 16.04
• TensorFlow installed from (source or binary):
  Source
• TensorFlow version (use command below):
  v1.3.0-0-g9e76bf324 1.3.0
• Python version:
  3.5.2
• Bazel version (if compiling from source):
  0.5.4
• CUDA/cuDNN version:
  8.0.44 / 5.1.5
• GPU model and memory:
  Any.
• Exact command to reproduce:
  See below.

Describe the problem

When compiling a custom op using nvcc, which includes tensorflow/core/util/cuda_kernel_helper.h, I get the following error:

/usr/local/cuda-8.0/bin/nvcc -c -o ~/Code/libspn/build/ops/gather_columns_functor_gpu.cu.cc.o ~/Code/libspn/libspn/ops/gather_columns_functor_gpu.cu.cc -std=c++11 -x=cu -Xcompiler -fPIC -DGOOGLE_CUDA=1 --expt-relaxed-constexpr -I ~/.local/lib/python3.5/site-packages/tensorflow/include -gencode=arch=compute_35,"code=sm_35,compute_35" -gencode=arch=compute_52,"code=sm_52,compute_52" -gencode=arch=compute_61,"code=sm_61,compute_61"
In file included from ~/.local/lib/python3.5/site-packages/tensorflow/include/tensorflow/core/platform/default/stream_executor.h:26:0,
                 from ~/.local/lib/python3.5/site-packages/tensorflow/include/tensorflow/core/platform/stream_executor.h:24,
                 from ~/.local/lib/python3.5/site-packages/tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h:26,
                 from ~/Code/libspn/libspn/ops/gather_columns_functor_gpu.cu.h:11,
                 from ~/Code/libspn/libspn/ops/gather_columns_functor_gpu.cu.cc:5:
~/.local/lib/python3.5/site-packages/tensorflow/include/tensorflow/stream_executor/dso_loader.h:32:30: fatal error: cuda/cuda_config.h: No such file or directory
compilation terminated.

Copying cuda_config.h to /site-packages/tensorflow/include/tensorflow/stream_executor/cuda solves the problem.

The same issue has been observed by several other users in #6602 (see the comments added after the issue was closed).

[yangyu12]

the same problem, I want to know if there is any better way to solve this problem

[aselle]

@allenlavoie, have you looked at this as part of your library cleanup?

[allenlavoie]

I haven't. I may have run into it when I was following our custom op documentation. It sounds like worst case we could add a copy in https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/pip_package/build_pip_package.sh ? Or maybe a genrule before that.

[jmlago]

I have the same problem while compiling with nvcc. Also is because tensorflow/core/util/cuda_kernel_helper.h was included.

But when I copy the file, i get the following error:

../site-packages/tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h:28:80: fatal error: cuda/include/cuda.h: No such file or directory
compilation terminated.

[PatWie]

The commit 2c598e8 destroys my makefile, as well. This commit consistly changes in several places the includes

-#include "third_party/gpus/cuda/include/cuda.h"
+#include "cuda/include/cuda.h"

Which gives me the error message:

../site-packages/tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h:28:80: fatal error: cuda/include/cuda.h: No such file or directory
compilation terminated.

compiling user-ops with GPU does not work since 1.3.1.

Where lives that guy, tensorflower-gardener? He is the one who usually introduces those breaking changes!

[HelloSeeing]

@PatWie +1
I encounter this problem when build android tensorflow_demo, giving the following error message:

tensorflow/core/kernels/lrn_op.cc:34:10: fatal error: 'cuda/include/cuda.h' file not found

[pronobis]

With TF 1.3, I only find cuda_config.h missing, but when compiling against current master 1.4-rc0/1, I also additionally get:

tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h:24:31: fatal error: cuda/include/cuda.h: No such file or directory
compilation terminated.

[bigbigda]

I also encounter the same problem in TF 1.3.0. This error is introduced with #include "third_party/eigen3/unsupported/Eigen/CXX11/Tensor"
/usr/local/lib/python3.5/dist-packages/tensorflow/include/tensorflow/stream_executor/dso_loader.h:32:30: fatal error: cuda/cuda_config.h: No such file or directory

[Leo-Mrl]

I also encounter this problem in TF 1.4.0, any clue ?

[kkk324]

Also stucking on this problem in TF 1.4.0...