Compilation error: image_ops_gpu.cu.pic.o was not created.

[zuxfoucault]

The issue is similar to https://stackoverflow.com/questions/44116381/error-when-install-tensorflow-from-source
/cc @drpngx

1 error detected in the compilation of "/tmp/tmpxft_00004364_00000000-7_image_ops_gpu.cu.cpp1.ii".
 ERROR:PATH/tensorflow_cuda75/tensorflow/contrib/image/BUILD:20:1: output 'tensorflow/contrib/image/_objs/python/ops/_image_ops_gpu/tensorflow/contrib/image/kernels/image_ops_gpu.cu.pic.o' was not created.

ERROR: PATH/tensorflow_cuda75/tensorflow/contrib/image/BUILD:20:1: 
not all outputs were created or valid.`

Selective logs before error:

 ./tensorflow/contrib/image/kernels/image_ops.h(69): error: Within a __device__/__global__ function, only __shared__ variables may be marked "static"
          detected during:                                                                                                                                                 instantiation of "Eigen::TensorEvaluator<const Eigen::TensorGeneratorOp<Generator, ArgType>, Device>::CoeffReturnType Eigen::TensorEvaluator<const Eigen::TensorGeneratorOp<Generator, ArgType>, Device>::coeff(Eigen::TensorEvaluator<const Eigen::TensorGeneratorOp<Generator, ArgType>, Device>::Index) const [with Generator=tensorflow::generator::ProjectiveGenerator<tensorflow::functor::GPUDevice, tensorflow::uint8>, ArgType=const Eigen::TensorMap<Eigen::Tensor<cons
t tensorflow::uint8, 4, 1, Eigen::DenseIndex>, 16, Eigen::MakePointer>, Device=Eigen::GpuDevice]"
external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorAssign.h(136): here                                                                                        instantiation of "void Eigen::TensorEvaluator<const Eigen::TensorAssignOp<LeftArgType, RightArgType>, Device>::evalScalar(Eigen::TensorEvaluator<const Eigen::TensorAssignOp<LeftArgType, RightArgType>, Device>::Index) [with LeftArgType=Eigen::TensorMap<Eigen::Tensor<tensorflow::uint8, 4, 1, Eigen::DenseInd
ex>, 16, Eigen::MakePointer>, RightArgType=const Eigen::TensorGeneratorOp<tensorflow::generator::ProjectiveGenerator<tensorflow::functor::GPUDevice, tensorflow
::uint8>, const Eigen::TensorMap<Eigen::Tensor<const tensorflow::uint8, 4, 1, Eigen::DenseIndex>, 16, Eigen::MakePointer>>, Device=Eigen::GpuDevice]"
external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorExecutor.h(210): here


...... omit....

           instantiation of "void Eigen::internal::EigenMetaKernelEval<Evaluator, Index, Vectorizable>::run(Evaluator &, Index, Index, Index) [with Evaluator=Eigen::TensorEvaluator<const Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<double, 4, 1, Eigen::DenseIndex>, 16, Eigen::MakePointer>, const Eigen::TensorGeneratorOp<tensorflow::generator::ProjectiveGenerator<tensorflow::functor::GPUDevice, double>, c
onst Eigen::TensorMap<Eigen::Tensor<const double, 4, 1, Eigen::DenseIndex>, 16, Eigen::MakePointer>>>, E
igen::GpuDevice>, Index=Eigen::DenseIndex, Vectorizable=false]"                                         external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorExecutor.h(243): here                               instantiation of "void Eigen::internal::EigenMetaKernel(Evaluator, Index) [with Evaluator=Eigen::TensorEvaluator<const Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<double, 4, 1, Eigen::DenseIndex>, 16, Eigen::MakePointer>, const Eigen::TensorGeneratorOp<tensorflow::generator::ProjectiveGenerator<tensorflow::functor::GPUDevice, double>, const Eigen::TensorMap<Eigen::Tensor<const double, 4, 1, E
igen::DenseIndex>, 16, Eigen::MakePointer>>>, Eigen::GpuDevice>, Index=Eigen::DenseIndex]"
external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorExecutor.h(260): here                               instantiation of "void Eigen::internal::TensorExecutor<Expression, Eigen::GpuDevice, Vectorizable>::run(const Expression &, const Eigen::GpuDevice &) [with Expression=const Eigen::TensorAssignOp<Eigen::TensorMap<Eigen::Tensor<double, 4, 1, Eigen::DenseIndex>, 16, Eigen::MakePointer>, const Eigen::Te
nsorGeneratorOp<tensorflow::generator::ProjectiveGenerator<tensorflow::functor::GPUDevice, double>, cons
t Eigen::TensorMap<Eigen::Tensor<const double, 4, 1, Eigen::DenseIndex>, 16, Eigen::MakePointer>>>, Vect
orizable=false]"
external/eigen_archive/unsupported/Eigen/CXX11/src/Tensor/TensorDevice.h(35): here                                  instantiation of "Eigen::TensorDevice<ExpressionType, DeviceType> &Eigen::TensorDevice<ExpressionType, DeviceType>::operator=(const OtherDerived &) [with ExpressionType=Eigen::TensorMap<Eigen::Ten
sor<double, 4, 1, Eigen::DenseIndex>, 16, Eigen::MakePointer>, DeviceType=tensorflow::functor::GPUDevice, OtherDerived=Eigen::TensorGeneratorOp<tensorflow::generator::ProjectiveGenerator<tensorflow::functor::
GPUDevice, double>, const Eigen::TensorMap<Eigen::Tensor<const double, 4, 1, Eigen::DenseIndex>, 16, Eig
en::MakePointer>>]"
(156): here
            instantiation of "void tensorflow::functor::FillProjectiveTransform<Device, T>::operator()(c
onst Device &, tensorflow::functor::FillProjectiveTransform<Device, T>::OutputType *, const tensorflow::
functor::FillProjectiveTransform<Device, T>::InputType &, const tensorflow::functor::FillProjectiveTrans
form<Device, T>::TransformsType &) const [with Device=tensorflow::functor::GPUDevice, T=double]"
tensorflow/contrib/image/kernels/image_ops_gpu.cu.cc(36): here

Environment: centos 6.8 devtoolset-3 cuda 7.5 cudnn v4. gcc-4.9 bazel 4.5 TF r1.2

Any advice and suggestions will be appreciated!

[hzy46]

same issue..

[gunan]

It is possible the build with cuda 7.5 is now broken.
You can try upgrading to cuda 8.0 and rebuilding.

[zuxfoucault]

Thanks for the prompt response!
TF r.1.2 can successfully be built with cuda 8.0. But, on the workstation, I don't have the privilege to upgrade the current cuda driver 352.93, which didn't support cuda 8.0.

PS. Commit a4b352b works fine with cuda 7.5 and also included the updating RNNCell objects changes.

[gunan]

As we moved our support to cuda 8.0, the support for 7.5 is best effort.
I cannot promise we will get to this in a timely manner, so I am marking this contributions welcome.
I am happy to accept a PR to fix this on CUDA 7.5 and also does not break CUDA 8.0

[jpuigcerver]

Hi @gunan, I created a PR that fixes this problem.

[lapolonio]

Hello,

I see that the PR was merged into master. Which release and when will the patch be merged into? Has it already been merged into a release?

[lapolonio]

I found the commit on r1.3

@zuxfoucault @jpuigcerver Did you try building from source with the PR merged? It's still not working for me.

[gunan]

I would recommend upgrading to cuda 8. CUDA 9 is out as an RC now, and once that is released we will swiftly move to CUDA 9, which will make CUDA 7.5 even more obsolete.

[lapolonio]

@gunan I have cuda driver 352.93 installed for the geforce titan x with cuda toolkit 7.5

Will the hardware work with cuda tookit 8?

[gunan]

The driver will be a problem in your case, too.
It is possible a new issue was introduced.
As we do not run continuous tests using older CUDA versions, we would not have caught it.
Again, we will welcome any contributions to fix any issues.