Skip to content
This repository has been archived by the owner on Mar 12, 2021. It is now read-only.
This repository has been archived by the owner on Mar 12, 2021. It is now read-only.

import CuArrays always fails with CUDA 10.2.89 (but works fine with CUDA 10.0.130 and 10.1.105) #601

Closed
Closed
import CuArrays always fails with CUDA 10.2.89 (but works fine with CUDA 10.0.130 and 10.1.105)#601

Description

@DilumAluthge

DilumAluthge
openedon Feb 24, 2020

Summary

I am unable to run import CuArrays with CUDA 10.2.89. However, I am able to successfully run import CuArrays with either CUDA 10.0.130 or CUDA 10.1.105 on the same cluster. (This is an HPC cluster with multiple different versions of CUDA available.)

The error I get looks like this:

┌ Error: CuArrays.jl failed to initialize
│   exception =
│    could not load library "libcublas"
│    libcublas.so: cannot open shared object file: No such file or directory
│    Stacktrace:
│     [1] dlopen(::String, ::UInt32; throw_error::Bool) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
│     [2] dlopen at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
│     [3] (::CuArrays.CUBLAS.var"#509#lookup_fptr#28")() at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:29
│     [4] macro expansion at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:37 [inlined]
│     [5] macro expansion at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/error.jl:65 [inlined]
│     [6] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/libcublas.jl:27 [inlined]
│     [7] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:38 [inlined]
│     [8] version() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:42
│     [9] __init__() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:98
│     [10] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
│     [11] _require_from_serialized(::String) at ./loading.jl:748
│     [12] _require(::Base.PkgId) at ./loading.jl:1039
│     [13] require(::Base.PkgId) at ./loading.jl:927
│     [14] require(::Module, ::Symbol) at ./loading.jl:922
│     [15] eval(::Module, ::Any) at ./boot.jl:331
│     [16] eval_user_input(::Any, ::REPL.REPLBackend) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:118
│     [17] macro expansion at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:150 [inlined]
│     [18] (::REPL.var"#31#32"{REPL.REPLBackend})() at ./task.jl:358
└ @ CuArrays ~/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:141

How to reproduce

First run these commands in Bash:

export JULIA_CUDA_VERBOSE="true"
export JULIA_DEBUG="all"
rm -rf ~/.julia

Then open Julia and run the following:

julia> versioninfo(verbose = true)

julia> import Pkg

julia> Pkg.add("CuArrays")

julia> import CuArrays

Full output

CUDA 10.2.89: (fails)

Click to expand 
$ which nvcc
/gpfs/runtime/opt/cuda/10.2/cuda/bin/nvcc

$ which nvdisasm
/gpfs/runtime/opt/cuda/10.2/cuda/bin/nvdisasm

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89

$ nvdisasm --version
nvdisasm: NVIDIA (R) CUDA disassembler
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:25:30_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.0-DEV.274 (2020-02-15)
 _/ |\__'_|_|_|\__'_|  |  Commit 8eb0f9fefb (8 days old master)
|__/                   |

julia> versioninfo(verbose = true)
Julia Version 1.5.0-DEV.274
Commit 8eb0f9fefb (2020-02-15 12:41 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      "Red Hat Enterprise Linux Server release 7.3 (Maipo)"
  uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64
  CPU: Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz:
              speed         user         nice          sys         idle          irq
       #1  3601 MHz  260888663 s        209 s   79076485 s  314462620 s          0 s
       #2  3601 MHz  225435476 s        556 s   66634601 s  362178032 s          0 s
       #3  3601 MHz  153758996 s        331 s   47119890 s  453453880 s          0 s
       #4  3601 MHz  133428383 s        355 s   44200083 s  476426058 s          0 s
       #5  3601 MHz  231463336 s        286 s   61205976 s  362261076 s          0 s
       #6  3601 MHz  154414432 s        604 s   45161270 s  455708660 s          0 s
       #7  3601 MHz  104946172 s        268 s   34510105 s  515709887 s          0 s
       #8  3601 MHz   94884882 s        496 s   31751569 s  528198402 s          0 s

  Memory: 93.04103088378906 GB (80644.640625 MB free)
  Uptime: 6.578586e6 sec
  Load Avg:  1.201171875  1.01953125  0.62353515625
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_DEBUG = all
  JULIA_CUDA_VERBOSE = true
  CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
  MANPATH = /gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1:
  TERM = xterm-256color
  LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
  CUDA_HOME = /gpfs/runtime/opt/cuda/10.2/cuda
  LD_LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64
  CPATH = /gpfs/runtime/opt/cuda/10.2/cuda/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include
  NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US
  PATH = /gpfs/runtime/opt/cuda/10.2/cuda/bin:/gpfs/runtime/opt/gcc/8.3/bin:/users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin
  C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
  LD_RUN_PATH = /gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
  JAVA_HOME = /gpfs/runtime/opt/java/8u111
  MODULEPATH = /gpfs/runtime/modulefiles
  HOME = /users/daluthge
  IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp
  MODULEHOME = /gpfs/runtime/pymodules
  PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig
  QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins

julia> import Pkg

julia> Pkg.add("CuArrays")
    Cloning default registries into `~/.julia`
######################################################################## 100.0%
      Added registry `General` to `~/.julia/registries/General`
  Resolving package versions...
  Installed Requires ─────────── v1.0.1
  Installed Adapt ────────────── v1.0.1
  Installed TimerOutputs ─────── v0.5.3
  Installed CUDAapi ──────────── v3.1.0
  Installed AbstractFFTs ─────── v0.5.0
  Installed GPUArrays ────────── v2.0.1
  Installed CuArrays ─────────── v1.7.2
  Installed CUDAnative ───────── v2.10.2
  Installed CEnum ────────────── v0.2.0
  Installed OrderedCollections ─ v1.1.0
  Installed DataStructures ───── v0.17.9
  Installed MacroTools ───────── v0.5.4
  Installed BinaryProvider ───── v0.5.8
  Installed NNlib ────────────── v0.6.4
  Installed CUDAdrv ──────────── v6.0.0
  Installed LLVM ─────────────── v1.3.3
   Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Project.toml`
  [3a865a2d] + CuArrays v1.7.2
   Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Manifest.toml`
  [621f4979] + AbstractFFTs v0.5.0
  [79e6a3ab] + Adapt v1.0.1
  [b99e7846] + BinaryProvider v0.5.8
  [fa961155] + CEnum v0.2.0
  [3895d2a7] + CUDAapi v3.1.0
  [c5f51814] + CUDAdrv v6.0.0
  [be33ccc6] + CUDAnative v2.10.2
  [3a865a2d] + CuArrays v1.7.2
  [864edb3b] + DataStructures v0.17.9
  [0c68f7d7] + GPUArrays v2.0.1
  [929cbde3] + LLVM v1.3.3
  [1914dd2f] + MacroTools v0.5.4
  [872c559c] + NNlib v0.6.4
  [bac558e1] + OrderedCollections v1.1.0
  [ae029012] + Requires v1.0.1
  [a759f4b9] + TimerOutputs v0.5.3
  [2a0f44e3] + Base64
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [de0858da] + Printf
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
   Building NNlib → `~/.julia/packages/NNlib/3krvM/deps/build.log`

julia> import CuArrays
[ Info: Precompiling CuArrays [3a865a2d-5b23-5a0f-bc46-62713ec82fae]
┌ Debug: Precompiling CUDAapi [3895d2a7-ec45-59b8-82bb-cfc6a382f9b3]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAdrv [c5f51814-7f29-56b8-a69c-e4d8f6be1fde]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17]
└ @ Base loading.jl:1276
┌ Debug: Precompiling LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:1276
┌ Debug: Found LLVM v9.0.1 at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/bin/../lib/libLLVM-9.so with support for AArch64, AMDGPU, ARC, ARM, AVR, BPF, Hexagon, Lanai, MSP430, Mips, NVPTX, PowerPC, RISCV, Sparc, SystemZ, WebAssembly, X86, XCore
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:47
┌ Debug: Using LLVM.jl wrapper for LLVM v9.0
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:75
┌ Debug: Precompiling Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:1276
┌ Debug: Precompiling TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:1276
┌ Debug: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
└ @ Base loading.jl:1276
┌ Debug: Precompiling OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
└ @ Base loading.jl:1276
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for binary nvdisasm
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for binary nvdisasm
│   locations =
│    20-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/bin"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/bin"
│     "/gpfs/runtime/opt/gcc/8.3/bin"
│     "/users/daluthge/bin"
│     "/gpfs/runtime/opt/python/3.7.4/bin"
│     "/gpfs/runtime/opt/git/2.20.2/bin"
│     "/gpfs/runtime/opt/binutils/2.31/bin"
│     ⋮
│     "/usr/bin"
│     "/usr/local/sbin"
│     "/usr/sbin"
│     "/usr/lpp/mmfs/bin"
│     "/usr/lpp/mmfs/sbin"
│     "/opt/ibutils/bin"
│     "/gpfs/runtime/bin"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found binary nvdisasm at /gpfs/runtime/opt/cuda/10.2/cuda/bin
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141
┌ Debug: CUDA toolkit identified as 10.2.89
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297
┌ Debug: Request to look for libdevice
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Look for libdevice
│   locations =
│    2-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/nvvm/libdevice"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found unified device library at /gpfs/runtime/opt/cuda/10.2/cuda/nvvm/libdevice/libdevice.10.bc
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327
┌ Debug: Request to look for libcudadevrt
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│   locations =
│    3-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found CUDA device runtime library libcudadevrt.a at /gpfs/runtime/opt/cuda/10.2/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379
┌ Debug: Request to look for library nvToolsExt
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0
│   locations =
│    4-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libnvToolsExt.so at /gpfs/runtime/opt/cuda/10.2/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Request to look for library cupti
│   locations =
│    2-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.2
│   locations =
│    8-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.2/cuda"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/lib64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/libx64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib64"
│     "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libcupti.so at /gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Using LLVM v9.0.1
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:170
┌ Debug: LLVM supports capabilities 2.0, 2.1, 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:191
┌ Debug: Using CUDA driver v10.2.0 and toolkit v10.2.0
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:196
┌ Debug: CUDA driver supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:213
┌ Debug: CUDA toolkit supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:214
┌ Debug: CUDAnative supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:174
┌ Debug: Precompiling GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:1276
┌ Debug: Precompiling AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:1276
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:1276
┌ Debug: Precompiling MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
└ @ Base loading.jl:1276
┌ Debug: Precompiling NNlib [872c559c-99b0-510c-b3b7-b6c96a88d5cd]
└ @ Base loading.jl:1276
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
┌ Error: CuArrays.jl failed to initialize
│   exception =
│    could not load library "libcublas"
│    libcublas.so: cannot open shared object file: No such file or directory
│    Stacktrace:
│     [1] dlopen(::String, ::UInt32; throw_error::Bool) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
│     [2] dlopen at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
│     [3] (::CuArrays.CUBLAS.var"#509#lookup_fptr#28")() at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:29
│     [4] macro expansion at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:37 [inlined]
│     [5] macro expansion at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/error.jl:65 [inlined]
│     [6] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/libcublas.jl:27 [inlined]
│     [7] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:38 [inlined]
│     [8] version() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:42
│     [9] __init__() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:98
│     [10] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
│     [11] _require_from_serialized(::String) at ./loading.jl:748
│     [12] _require(::Base.PkgId) at ./loading.jl:1039
│     [13] require(::Base.PkgId) at ./loading.jl:927
│     [14] require(::Module, ::Symbol) at ./loading.jl:922
│     [15] eval(::Module, ::Any) at ./boot.jl:331
│     [16] eval_user_input(::Any, ::REPL.REPLBackend) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:118
│     [17] macro expansion at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:150 [inlined]
│     [18] (::REPL.var"#31#32"{REPL.REPLBackend})() at ./task.jl:358
└ @ CuArrays ~/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:141

CUDA 10.1.105: (works fine)

Click to expand 
$ which nvcc
/gpfs/runtime/opt/cuda/10.1.105/cuda/bin/nvcc

$ which nvdisasm
/gpfs/runtime/opt/cuda/10.1.105/cuda/bin/nvdisasm

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105

$ nvdisasm --version
nvdisasm: NVIDIA (R) CUDA disassembler
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:51_PST_2019
Cuda compilation tools, release 10.1, V10.1.105
bash-4.2$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.0-DEV.274 (2020-02-15)
 _/ |\__'_|_|_|\__'_|  |  Commit 8eb0f9fefb (8 days old master)
|__/                   |

julia> versioninfo(verbose = true)
Julia Version 1.5.0-DEV.274
Commit 8eb0f9fefb (2020-02-15 12:41 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      "Red Hat Enterprise Linux Server release 7.3 (Maipo)"
  uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64
  CPU: Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz:
              speed         user         nice          sys         idle          irq
       #1  3601 MHz  260888239 s        209 s   79076296 s  314453175 s          0 s
       #2  3601 MHz  225435401 s        556 s   66634561 s  362168060 s          0 s
       #3  3601 MHz  153758683 s        331 s   47119620 s  453444427 s          0 s
       #4  3601 MHz  133428321 s        355 s   44199667 s  476416450 s          0 s
       #5  3601 MHz  231457860 s        286 s   61205722 s  362256718 s          0 s
       #6  3601 MHz  154411110 s        604 s   45161031 s  455702131 s          0 s
       #7  3601 MHz  104945705 s        268 s   34510058 s  515700335 s          0 s
       #8  3601 MHz   94884872 s        496 s   31751407 s  528188490 s          0 s

  Memory: 93.04103088378906 GB (80663.54296875 MB free)
  Uptime: 6.578485e6 sec
  Load Avg:  1.19580078125  0.96240234375  0.56298828125
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_DEBUG = all
  JULIA_CUDA_VERBOSE = true
  CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
  MANPATH = /gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1:
  TERM = xterm-256color
  LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64:/gpfs/runtime/opt/cuda/10.1.105/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
  CUDA_HOME = /gpfs/runtime/opt/cuda/10.1.105/cuda
  LD_LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64:/gpfs/runtime/opt/cuda/10.1.105/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64
  CPATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/include:/gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include
  NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US
  PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/bin:/users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/gcc/8.3/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin
  C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
  LD_RUN_PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64:/gpfs/runtime/opt/cuda/10.1.105/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
  JAVA_HOME = /gpfs/runtime/opt/java/8u111
  MODULEPATH = /gpfs/runtime/modulefiles
  HOME = /users/daluthge
  IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp
  MODULEHOME = /gpfs/runtime/pymodules
  PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig
  QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins

julia> import Pkg

julia> Pkg.add("CuArrays")
    Cloning default registries into `~/.julia`
######################################################################## 100.0%
      Added registry `General` to `~/.julia/registries/General`
  Resolving package versions...
  Installed Requires ─────────── v1.0.1
  Installed Adapt ────────────── v1.0.1
  Installed TimerOutputs ─────── v0.5.3
  Installed AbstractFFTs ─────── v0.5.0
  Installed CUDAapi ──────────── v3.1.0
  Installed GPUArrays ────────── v2.0.1
  Installed CuArrays ─────────── v1.7.2
  Installed CUDAnative ───────── v2.10.2
  Installed CEnum ────────────── v0.2.0
  Installed OrderedCollections ─ v1.1.0
  Installed DataStructures ───── v0.17.9
  Installed MacroTools ───────── v0.5.4
  Installed BinaryProvider ───── v0.5.8
  Installed NNlib ────────────── v0.6.4
  Installed CUDAdrv ──────────── v6.0.0
  Installed LLVM ─────────────── v1.3.3
   Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Project.toml`
  [3a865a2d] + CuArrays v1.7.2
   Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Manifest.toml`
  [621f4979] + AbstractFFTs v0.5.0
  [79e6a3ab] + Adapt v1.0.1
  [b99e7846] + BinaryProvider v0.5.8
  [fa961155] + CEnum v0.2.0
  [3895d2a7] + CUDAapi v3.1.0
  [c5f51814] + CUDAdrv v6.0.0
  [be33ccc6] + CUDAnative v2.10.2
  [3a865a2d] + CuArrays v1.7.2
  [864edb3b] + DataStructures v0.17.9
  [0c68f7d7] + GPUArrays v2.0.1
  [929cbde3] + LLVM v1.3.3
  [1914dd2f] + MacroTools v0.5.4
  [872c559c] + NNlib v0.6.4
  [bac558e1] + OrderedCollections v1.1.0
  [ae029012] + Requires v1.0.1
  [a759f4b9] + TimerOutputs v0.5.3
  [2a0f44e3] + Base64
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [de0858da] + Printf
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
   Building NNlib → `~/.julia/packages/NNlib/3krvM/deps/build.log`

julia> import CuArrays
[ Info: Precompiling CuArrays [3a865a2d-5b23-5a0f-bc46-62713ec82fae]
┌ Debug: Precompiling CUDAapi [3895d2a7-ec45-59b8-82bb-cfc6a382f9b3]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAdrv [c5f51814-7f29-56b8-a69c-e4d8f6be1fde]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17]
└ @ Base loading.jl:1276
┌ Debug: Precompiling LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:1276
┌ Debug: Found LLVM v9.0.1 at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/bin/../lib/libLLVM-9.so with support for AArch64, AMDGPU, ARC, ARM, AVR, BPF, Hexagon, Lanai, MSP430, Mips, NVPTX, PowerPC, RISCV, Sparc, SystemZ, WebAssembly, X86, XCore
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:47
┌ Debug: Using LLVM.jl wrapper for LLVM v9.0
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:75
┌ Debug: Precompiling Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:1276
┌ Debug: Precompiling TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:1276
┌ Debug: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
└ @ Base loading.jl:1276
┌ Debug: Precompiling OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
└ @ Base loading.jl:1276
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for binary nvdisasm
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for binary nvdisasm
│   locations =
│    20-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/bin"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/bin"
│     "/users/daluthge/bin"
│     "/gpfs/runtime/opt/python/3.7.4/bin"
│     "/gpfs/runtime/opt/git/2.20.2/bin"
│     "/gpfs/runtime/opt/gcc/8.3/bin"
│     "/gpfs/runtime/opt/binutils/2.31/bin"
│     ⋮
│     "/usr/bin"
│     "/usr/local/sbin"
│     "/usr/sbin"
│     "/usr/lpp/mmfs/bin"
│     "/usr/lpp/mmfs/sbin"
│     "/opt/ibutils/bin"
│     "/gpfs/runtime/bin"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found binary nvdisasm at /gpfs/runtime/opt/cuda/10.1.105/cuda/bin
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141
┌ Debug: CUDA toolkit identified as 10.1.105
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297
┌ Debug: Request to look for libdevice
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Look for libdevice
│   locations =
│    2-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/nvvm/libdevice"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found unified device library at /gpfs/runtime/opt/cuda/10.1.105/cuda/nvvm/libdevice/libdevice.10.bc
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327
┌ Debug: Request to look for libcudadevrt
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│   locations =
│    3-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found CUDA device runtime library libcudadevrt.a at /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379
┌ Debug: Request to look for library nvToolsExt
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0
│   locations =
│    4-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib64"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libnvToolsExt.so at /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Request to look for library cupti
│   locations =
│    2-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.1
│   locations =
│    8-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib64"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/libx64"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/lib"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/lib64"
│     "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libcupti.so at /gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Using LLVM v9.0.1
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:170
┌ Debug: LLVM supports capabilities 2.0, 2.1, 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:191
┌ Debug: Using CUDA driver v10.2.0 and toolkit v10.1.0
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:196
┌ Debug: CUDA driver supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:213
┌ Debug: CUDA toolkit supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:214
┌ Debug: CUDAnative supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:174
┌ Debug: Precompiling GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:1276
┌ Debug: Precompiling AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:1276
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:1276
┌ Debug: Precompiling MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
└ @ Base loading.jl:1276
┌ Debug: Precompiling NNlib [872c559c-99b0-510c-b3b7-b6c96a88d5cd]
└ @ Base loading.jl:1276
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114

CUDA 10.0.130: (works fine)

Click to expand 
$ which nvcc
/gpfs/runtime/opt/cuda/10.0.130/cuda/bin/nvcc

$ which nvdisasm
/gpfs/runtime/opt/cuda/10.0.130/cuda/bin/nvdisasm

$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

$ nvdisasm --version
nvdisasm: NVIDIA (R) CUDA disassembler
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:11_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
bash-4.2$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.0-DEV.274 (2020-02-15)
 _/ |\__'_|_|_|\__'_|  |  Commit 8eb0f9fefb (8 days old master)
|__/                   |

julia> versioninfo(verbose = true)
Julia Version 1.5.0-DEV.274
Commit 8eb0f9fefb (2020-02-15 12:41 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
      "Red Hat Enterprise Linux Server release 7.3 (Maipo)"
  uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64
  CPU: Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz:
              speed         user         nice          sys         idle          irq
       #1  3601 MHz  260887968 s        209 s   79076153 s  314446557 s          0 s
       #2  3601 MHz  225435375 s        556 s   66634534 s  362161063 s          0 s
       #3  3601 MHz  153758324 s        331 s   47119535 s  453437851 s          0 s
       #4  3601 MHz  133428293 s        355 s   44199325 s  476409768 s          0 s
       #5  3601 MHz  231454591 s        286 s   61205506 s  362253151 s          0 s
       #6  3601 MHz  154408009 s        604 s   45160867 s  455698340 s          0 s
       #7  3601 MHz  104945700 s        268 s   34510038 s  515693325 s          0 s
       #8  3601 MHz   94884863 s        496 s   31751307 s  528181552 s          0 s

  Memory: 93.04103088378906 GB (80678.2890625 MB free)
  Uptime: 6.578414e6 sec
  Load Avg:  1.1494140625  0.908203125  0.515625
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
  JULIA_DEBUG = all
  JULIA_CUDA_VERBOSE = true
  CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
  MANPATH = /gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1:
  TERM = xterm-256color
  LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64:/gpfs/runtime/opt/cuda/10.0.130/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
  CUDA_HOME = /gpfs/runtime/opt/cuda/10.0.130/cuda
  LD_LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64:/gpfs/runtime/opt/cuda/10.0.130/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64
  CPATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/include:/gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include
  NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US
  PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/bin:/users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/gcc/8.3/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin
  C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
  LD_RUN_PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64:/gpfs/runtime/opt/cuda/10.0.130/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
  JAVA_HOME = /gpfs/runtime/opt/java/8u111
  MODULEPATH = /gpfs/runtime/modulefiles
  HOME = /users/daluthge
  IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp
  MODULEHOME = /gpfs/runtime/pymodules
  PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig
  QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins

julia> import Pkg

julia> Pkg.add("CuArrays")
    Cloning default registries into `~/.julia`
######################################################################## 100.0%
      Added registry `General` to `~/.julia/registries/General`
  Resolving package versions...
  Installed TimerOutputs ─────── v0.5.3
  Installed Adapt ────────────── v1.0.1
  Installed Requires ─────────── v1.0.1
  Installed AbstractFFTs ─────── v0.5.0
  Installed CUDAapi ──────────── v3.1.0
  Installed GPUArrays ────────── v2.0.1
  Installed CEnum ────────────── v0.2.0
  Installed OrderedCollections ─ v1.1.0
  Installed DataStructures ───── v0.17.9
  Installed CuArrays ─────────── v1.7.2
  Installed MacroTools ───────── v0.5.4
  Installed BinaryProvider ───── v0.5.8
  Installed CUDAnative ───────── v2.10.2
  Installed NNlib ────────────── v0.6.4
  Installed CUDAdrv ──────────── v6.0.0
  Installed LLVM ─────────────── v1.3.3
   Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Project.toml`
  [3a865a2d] + CuArrays v1.7.2
   Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Manifest.toml`
  [621f4979] + AbstractFFTs v0.5.0
  [79e6a3ab] + Adapt v1.0.1
  [b99e7846] + BinaryProvider v0.5.8
  [fa961155] + CEnum v0.2.0
  [3895d2a7] + CUDAapi v3.1.0
  [c5f51814] + CUDAdrv v6.0.0
  [be33ccc6] + CUDAnative v2.10.2
  [3a865a2d] + CuArrays v1.7.2
  [864edb3b] + DataStructures v0.17.9
  [0c68f7d7] + GPUArrays v2.0.1
  [929cbde3] + LLVM v1.3.3
  [1914dd2f] + MacroTools v0.5.4
  [872c559c] + NNlib v0.6.4
  [bac558e1] + OrderedCollections v1.1.0
  [ae029012] + Requires v1.0.1
  [a759f4b9] + TimerOutputs v0.5.3
  [2a0f44e3] + Base64
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [de0858da] + Printf
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
   Building NNlib → `~/.julia/packages/NNlib/3krvM/deps/build.log`

julia> import CuArrays
[ Info: Precompiling CuArrays [3a865a2d-5b23-5a0f-bc46-62713ec82fae]
┌ Debug: Precompiling CUDAapi [3895d2a7-ec45-59b8-82bb-cfc6a382f9b3]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAdrv [c5f51814-7f29-56b8-a69c-e4d8f6be1fde]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17]
└ @ Base loading.jl:1276
┌ Debug: Precompiling LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:1276
┌ Debug: Found LLVM v9.0.1 at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/bin/../lib/libLLVM-9.so with support for AArch64, AMDGPU, ARC, ARM, AVR, BPF, Hexagon, Lanai, MSP430, Mips, NVPTX, PowerPC, RISCV, Sparc, SystemZ, WebAssembly, X86, XCore
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:47
┌ Debug: Using LLVM.jl wrapper for LLVM v9.0
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:75
┌ Debug: Precompiling Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:1276
┌ Debug: Precompiling TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:1276
┌ Debug: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
└ @ Base loading.jl:1276
┌ Debug: Precompiling OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
└ @ Base loading.jl:1276
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for binary nvdisasm
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for binary nvdisasm
│   locations =
│    20-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/bin"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/bin"
│     "/users/daluthge/bin"
│     "/gpfs/runtime/opt/python/3.7.4/bin"
│     "/gpfs/runtime/opt/git/2.20.2/bin"
│     "/gpfs/runtime/opt/gcc/8.3/bin"
│     "/gpfs/runtime/opt/binutils/2.31/bin"
│     ⋮
│     "/usr/bin"
│     "/usr/local/sbin"
│     "/usr/sbin"
│     "/usr/lpp/mmfs/bin"
│     "/usr/lpp/mmfs/sbin"
│     "/opt/ibutils/bin"
│     "/gpfs/runtime/bin"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found binary nvdisasm at /gpfs/runtime/opt/cuda/10.0.130/cuda/bin
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141
┌ Debug: CUDA toolkit identified as 10.0.130
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297
┌ Debug: Request to look for libdevice
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Look for libdevice
│   locations =
│    2-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/nvvm/libdevice"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found unified device library at /gpfs/runtime/opt/cuda/10.0.130/cuda/nvvm/libdevice/libdevice.10.bc
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327
┌ Debug: Request to look for libcudadevrt
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│   locations =
│    3-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found CUDA device runtime library libcudadevrt.a at /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379
┌ Debug: Request to look for library nvToolsExt
│   locations =
│    1-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0
│   locations =
│    4-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libnvToolsExt.so at /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Request to look for library cupti
│   locations =
│    2-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.0
│   locations =
│    8-element Array{String,1}:
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/libx64"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/lib"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/lib64"
│     "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libcupti.so at /gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Using LLVM v9.0.1
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:170
┌ Debug: LLVM supports capabilities 2.0, 2.1, 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:191
┌ Debug: Using CUDA driver v10.2.0 and toolkit v10.0.0
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:196
┌ Debug: CUDA driver supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:213
┌ Debug: CUDA toolkit supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2 and 6.3
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:214
┌ Debug: CUDAnative supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1 and 6.3
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:174
┌ Debug: Precompiling GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:1276
┌ Debug: Precompiling AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:1276
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:1276
┌ Debug: Precompiling MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
└ @ Base loading.jl:1276
┌ Debug: Precompiling NNlib [872c559c-99b0-510c-b3b7-b6c96a88d5cd]
└ @ Base loading.jl:1276
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    bug

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions