This repository has been archived by the owner on Mar 12, 2021. It is now read-only.
This repository has been archived by the owner on Mar 12, 2021. It is now read-only.
import CuArrays
always fails with CUDA 10.2.89 (but works fine with CUDA 10.0.130 and 10.1.105) #601
Closed
Description
Summary
I am unable to run import CuArrays
with CUDA 10.2.89. However, I am able to successfully run import CuArrays
with either CUDA 10.0.130 or CUDA 10.1.105 on the same cluster. (This is an HPC cluster with multiple different versions of CUDA available.)
The error I get looks like this:
┌ Error: CuArrays.jl failed to initialize
│ exception =
│ could not load library "libcublas"
│ libcublas.so: cannot open shared object file: No such file or directory
│ Stacktrace:
│ [1] dlopen(::String, ::UInt32; throw_error::Bool) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
│ [2] dlopen at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
│ [3] (::CuArrays.CUBLAS.var"#509#lookup_fptr#28")() at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:29
│ [4] macro expansion at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:37 [inlined]
│ [5] macro expansion at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/error.jl:65 [inlined]
│ [6] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/libcublas.jl:27 [inlined]
│ [7] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:38 [inlined]
│ [8] version() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:42
│ [9] __init__() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:98
│ [10] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
│ [11] _require_from_serialized(::String) at ./loading.jl:748
│ [12] _require(::Base.PkgId) at ./loading.jl:1039
│ [13] require(::Base.PkgId) at ./loading.jl:927
│ [14] require(::Module, ::Symbol) at ./loading.jl:922
│ [15] eval(::Module, ::Any) at ./boot.jl:331
│ [16] eval_user_input(::Any, ::REPL.REPLBackend) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:118
│ [17] macro expansion at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:150 [inlined]
│ [18] (::REPL.var"#31#32"{REPL.REPLBackend})() at ./task.jl:358
└ @ CuArrays ~/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:141
How to reproduce
First run these commands in Bash:
export JULIA_CUDA_VERBOSE="true"
export JULIA_DEBUG="all"
rm -rf ~/.julia
Then open Julia and run the following:
julia> versioninfo(verbose = true)
julia> import Pkg
julia> Pkg.add("CuArrays")
julia> import CuArrays
Full output
CUDA 10.2.89: (fails)
Click to expand
$ which nvcc
/gpfs/runtime/opt/cuda/10.2/cuda/bin/nvcc
$ which nvdisasm
/gpfs/runtime/opt/cuda/10.2/cuda/bin/nvdisasm
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:24:38_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89
$ nvdisasm --version
nvdisasm: NVIDIA (R) CUDA disassembler
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Wed_Oct_23_19:25:30_PDT_2019
Cuda compilation tools, release 10.2, V10.2.89
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.5.0-DEV.274 (2020-02-15)
_/ |\__'_|_|_|\__'_| | Commit 8eb0f9fefb (8 days old master)
|__/ |
julia> versioninfo(verbose = true)
Julia Version 1.5.0-DEV.274
Commit 8eb0f9fefb (2020-02-15 12:41 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
"Red Hat Enterprise Linux Server release 7.3 (Maipo)"
uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64
CPU: Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz:
speed user nice sys idle irq
#1 3601 MHz 260888663 s 209 s 79076485 s 314462620 s 0 s
#2 3601 MHz 225435476 s 556 s 66634601 s 362178032 s 0 s
#3 3601 MHz 153758996 s 331 s 47119890 s 453453880 s 0 s
#4 3601 MHz 133428383 s 355 s 44200083 s 476426058 s 0 s
#5 3601 MHz 231463336 s 286 s 61205976 s 362261076 s 0 s
#6 3601 MHz 154414432 s 604 s 45161270 s 455708660 s 0 s
#7 3601 MHz 104946172 s 268 s 34510105 s 515709887 s 0 s
#8 3601 MHz 94884882 s 496 s 31751569 s 528198402 s 0 s
Memory: 93.04103088378906 GB (80644.640625 MB free)
Uptime: 6.578586e6 sec
Load Avg: 1.201171875 1.01953125 0.62353515625
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
JULIA_DEBUG = all
JULIA_CUDA_VERBOSE = true
CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
MANPATH = /gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1:
TERM = xterm-256color
LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
CUDA_HOME = /gpfs/runtime/opt/cuda/10.2/cuda
LD_LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64
CPATH = /gpfs/runtime/opt/cuda/10.2/cuda/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include
NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US
PATH = /gpfs/runtime/opt/cuda/10.2/cuda/bin:/gpfs/runtime/opt/gcc/8.3/bin:/users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin
C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
LD_RUN_PATH = /gpfs/runtime/opt/cuda/10.2/cuda/lib64:/gpfs/runtime/opt/cuda/10.2/cuda/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
JAVA_HOME = /gpfs/runtime/opt/java/8u111
MODULEPATH = /gpfs/runtime/modulefiles
HOME = /users/daluthge
IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp
MODULEHOME = /gpfs/runtime/pymodules
PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig
QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
julia> import Pkg
julia> Pkg.add("CuArrays")
Cloning default registries into `~/.julia`
######################################################################## 100.0%
Added registry `General` to `~/.julia/registries/General`
Resolving package versions...
Installed Requires ─────────── v1.0.1
Installed Adapt ────────────── v1.0.1
Installed TimerOutputs ─────── v0.5.3
Installed CUDAapi ──────────── v3.1.0
Installed AbstractFFTs ─────── v0.5.0
Installed GPUArrays ────────── v2.0.1
Installed CuArrays ─────────── v1.7.2
Installed CUDAnative ───────── v2.10.2
Installed CEnum ────────────── v0.2.0
Installed OrderedCollections ─ v1.1.0
Installed DataStructures ───── v0.17.9
Installed MacroTools ───────── v0.5.4
Installed BinaryProvider ───── v0.5.8
Installed NNlib ────────────── v0.6.4
Installed CUDAdrv ──────────── v6.0.0
Installed LLVM ─────────────── v1.3.3
Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Project.toml`
[3a865a2d] + CuArrays v1.7.2
Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Manifest.toml`
[621f4979] + AbstractFFTs v0.5.0
[79e6a3ab] + Adapt v1.0.1
[b99e7846] + BinaryProvider v0.5.8
[fa961155] + CEnum v0.2.0
[3895d2a7] + CUDAapi v3.1.0
[c5f51814] + CUDAdrv v6.0.0
[be33ccc6] + CUDAnative v2.10.2
[3a865a2d] + CuArrays v1.7.2
[864edb3b] + DataStructures v0.17.9
[0c68f7d7] + GPUArrays v2.0.1
[929cbde3] + LLVM v1.3.3
[1914dd2f] + MacroTools v0.5.4
[872c559c] + NNlib v0.6.4
[bac558e1] + OrderedCollections v1.1.0
[ae029012] + Requires v1.0.1
[a759f4b9] + TimerOutputs v0.5.3
[2a0f44e3] + Base64
[8ba89e20] + Distributed
[b77e0a4c] + InteractiveUtils
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[de0858da] + Printf
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[2f01184e] + SparseArrays
[10745b16] + Statistics
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
Building NNlib → `~/.julia/packages/NNlib/3krvM/deps/build.log`
julia> import CuArrays
[ Info: Precompiling CuArrays [3a865a2d-5b23-5a0f-bc46-62713ec82fae]
┌ Debug: Precompiling CUDAapi [3895d2a7-ec45-59b8-82bb-cfc6a382f9b3]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAdrv [c5f51814-7f29-56b8-a69c-e4d8f6be1fde]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17]
└ @ Base loading.jl:1276
┌ Debug: Precompiling LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:1276
┌ Debug: Found LLVM v9.0.1 at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/bin/../lib/libLLVM-9.so with support for AArch64, AMDGPU, ARC, ARM, AVR, BPF, Hexagon, Lanai, MSP430, Mips, NVPTX, PowerPC, RISCV, Sparc, SystemZ, WebAssembly, X86, XCore
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:47
┌ Debug: Using LLVM.jl wrapper for LLVM v9.0
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:75
┌ Debug: Precompiling Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:1276
┌ Debug: Precompiling TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:1276
┌ Debug: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
└ @ Base loading.jl:1276
┌ Debug: Precompiling OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
└ @ Base loading.jl:1276
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for binary nvdisasm
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for binary nvdisasm
│ locations =
│ 20-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.2/cuda"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/bin"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/bin"
│ "/gpfs/runtime/opt/gcc/8.3/bin"
│ "/users/daluthge/bin"
│ "/gpfs/runtime/opt/python/3.7.4/bin"
│ "/gpfs/runtime/opt/git/2.20.2/bin"
│ "/gpfs/runtime/opt/binutils/2.31/bin"
│ ⋮
│ "/usr/bin"
│ "/usr/local/sbin"
│ "/usr/sbin"
│ "/usr/lpp/mmfs/bin"
│ "/usr/lpp/mmfs/sbin"
│ "/opt/ibutils/bin"
│ "/gpfs/runtime/bin"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found binary nvdisasm at /gpfs/runtime/opt/cuda/10.2/cuda/bin
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141
┌ Debug: CUDA toolkit identified as 10.2.89
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297
┌ Debug: Request to look for libdevice
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Look for libdevice
│ locations =
│ 2-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.2/cuda"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/nvvm/libdevice"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found unified device library at /gpfs/runtime/opt/cuda/10.2/cuda/nvvm/libdevice/libdevice.10.bc
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327
┌ Debug: Request to look for libcudadevrt
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│ locations =
│ 3-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.2/cuda"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/lib"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/lib64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found CUDA device runtime library libcudadevrt.a at /gpfs/runtime/opt/cuda/10.2/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379
┌ Debug: Request to look for library nvToolsExt
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.2/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0
│ locations =
│ 4-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.2/cuda"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/lib"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/lib64"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libnvToolsExt.so at /gpfs/runtime/opt/cuda/10.2/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Request to look for library cupti
│ locations =
│ 2-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.2/cuda"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.2
│ locations =
│ 8-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.2/cuda"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/lib"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/lib64"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/libx64"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib64"
│ "/gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libcupti.so at /gpfs/runtime/opt/cuda/10.2/cuda/extras/CUPTI/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Using LLVM v9.0.1
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:170
┌ Debug: LLVM supports capabilities 2.0, 2.1, 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:191
┌ Debug: Using CUDA driver v10.2.0 and toolkit v10.2.0
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:196
┌ Debug: CUDA driver supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:213
┌ Debug: CUDA toolkit supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:214
┌ Debug: CUDAnative supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:174
┌ Debug: Precompiling GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:1276
┌ Debug: Precompiling AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:1276
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:1276
┌ Debug: Precompiling MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
└ @ Base loading.jl:1276
┌ Debug: Precompiling NNlib [872c559c-99b0-510c-b3b7-b6c96a88d5cd]
└ @ Base loading.jl:1276
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
┌ Error: CuArrays.jl failed to initialize
│ exception =
│ could not load library "libcublas"
│ libcublas.so: cannot open shared object file: No such file or directory
│ Stacktrace:
│ [1] dlopen(::String, ::UInt32; throw_error::Bool) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109
│ [2] dlopen at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/Libdl/src/Libdl.jl:109 [inlined] (repeats 2 times)
│ [3] (::CuArrays.CUBLAS.var"#509#lookup_fptr#28")() at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:29
│ [4] macro expansion at /users/daluthge/.julia/packages/CUDAapi/wYUAO/src/call.jl:37 [inlined]
│ [5] macro expansion at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/error.jl:65 [inlined]
│ [6] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/libcublas.jl:27 [inlined]
│ [7] cublasGetProperty at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:38 [inlined]
│ [8] version() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/blas/wrappers.jl:42
│ [9] __init__() at /users/daluthge/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:98
│ [10] _include_from_serialized(::String, ::Array{Any,1}) at ./loading.jl:697
│ [11] _require_from_serialized(::String) at ./loading.jl:748
│ [12] _require(::Base.PkgId) at ./loading.jl:1039
│ [13] require(::Base.PkgId) at ./loading.jl:927
│ [14] require(::Module, ::Symbol) at ./loading.jl:922
│ [15] eval(::Module, ::Any) at ./boot.jl:331
│ [16] eval_user_input(::Any, ::REPL.REPLBackend) at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:118
│ [17] macro expansion at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/share/julia/stdlib/v1.5/REPL/src/REPL.jl:150 [inlined]
│ [18] (::REPL.var"#31#32"{REPL.REPLBackend})() at ./task.jl:358
└ @ CuArrays ~/.julia/packages/CuArrays/HE8G6/src/CuArrays.jl:141
CUDA 10.1.105: (works fine)
Click to expand
$ which nvcc
/gpfs/runtime/opt/cuda/10.1.105/cuda/bin/nvcc
$ which nvdisasm
/gpfs/runtime/opt/cuda/10.1.105/cuda/bin/nvdisasm
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:17_PST_2019
Cuda compilation tools, release 10.1, V10.1.105
$ nvdisasm --version
nvdisasm: NVIDIA (R) CUDA disassembler
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Fri_Feb__8_19:08:51_PST_2019
Cuda compilation tools, release 10.1, V10.1.105
bash-4.2$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.5.0-DEV.274 (2020-02-15)
_/ |\__'_|_|_|\__'_| | Commit 8eb0f9fefb (8 days old master)
|__/ |
julia> versioninfo(verbose = true)
Julia Version 1.5.0-DEV.274
Commit 8eb0f9fefb (2020-02-15 12:41 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
"Red Hat Enterprise Linux Server release 7.3 (Maipo)"
uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64
CPU: Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz:
speed user nice sys idle irq
#1 3601 MHz 260888239 s 209 s 79076296 s 314453175 s 0 s
#2 3601 MHz 225435401 s 556 s 66634561 s 362168060 s 0 s
#3 3601 MHz 153758683 s 331 s 47119620 s 453444427 s 0 s
#4 3601 MHz 133428321 s 355 s 44199667 s 476416450 s 0 s
#5 3601 MHz 231457860 s 286 s 61205722 s 362256718 s 0 s
#6 3601 MHz 154411110 s 604 s 45161031 s 455702131 s 0 s
#7 3601 MHz 104945705 s 268 s 34510058 s 515700335 s 0 s
#8 3601 MHz 94884872 s 496 s 31751407 s 528188490 s 0 s
Memory: 93.04103088378906 GB (80663.54296875 MB free)
Uptime: 6.578485e6 sec
Load Avg: 1.19580078125 0.96240234375 0.56298828125
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
JULIA_DEBUG = all
JULIA_CUDA_VERBOSE = true
CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
MANPATH = /gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1:
TERM = xterm-256color
LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64:/gpfs/runtime/opt/cuda/10.1.105/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
CUDA_HOME = /gpfs/runtime/opt/cuda/10.1.105/cuda
LD_LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64:/gpfs/runtime/opt/cuda/10.1.105/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64
CPATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/include:/gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include
NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US
PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/bin:/users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/gcc/8.3/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin
C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
LD_RUN_PATH = /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64:/gpfs/runtime/opt/cuda/10.1.105/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
JAVA_HOME = /gpfs/runtime/opt/java/8u111
MODULEPATH = /gpfs/runtime/modulefiles
HOME = /users/daluthge
IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp
MODULEHOME = /gpfs/runtime/pymodules
PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig
QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
julia> import Pkg
julia> Pkg.add("CuArrays")
Cloning default registries into `~/.julia`
######################################################################## 100.0%
Added registry `General` to `~/.julia/registries/General`
Resolving package versions...
Installed Requires ─────────── v1.0.1
Installed Adapt ────────────── v1.0.1
Installed TimerOutputs ─────── v0.5.3
Installed AbstractFFTs ─────── v0.5.0
Installed CUDAapi ──────────── v3.1.0
Installed GPUArrays ────────── v2.0.1
Installed CuArrays ─────────── v1.7.2
Installed CUDAnative ───────── v2.10.2
Installed CEnum ────────────── v0.2.0
Installed OrderedCollections ─ v1.1.0
Installed DataStructures ───── v0.17.9
Installed MacroTools ───────── v0.5.4
Installed BinaryProvider ───── v0.5.8
Installed NNlib ────────────── v0.6.4
Installed CUDAdrv ──────────── v6.0.0
Installed LLVM ─────────────── v1.3.3
Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Project.toml`
[3a865a2d] + CuArrays v1.7.2
Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Manifest.toml`
[621f4979] + AbstractFFTs v0.5.0
[79e6a3ab] + Adapt v1.0.1
[b99e7846] + BinaryProvider v0.5.8
[fa961155] + CEnum v0.2.0
[3895d2a7] + CUDAapi v3.1.0
[c5f51814] + CUDAdrv v6.0.0
[be33ccc6] + CUDAnative v2.10.2
[3a865a2d] + CuArrays v1.7.2
[864edb3b] + DataStructures v0.17.9
[0c68f7d7] + GPUArrays v2.0.1
[929cbde3] + LLVM v1.3.3
[1914dd2f] + MacroTools v0.5.4
[872c559c] + NNlib v0.6.4
[bac558e1] + OrderedCollections v1.1.0
[ae029012] + Requires v1.0.1
[a759f4b9] + TimerOutputs v0.5.3
[2a0f44e3] + Base64
[8ba89e20] + Distributed
[b77e0a4c] + InteractiveUtils
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[de0858da] + Printf
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[2f01184e] + SparseArrays
[10745b16] + Statistics
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
Building NNlib → `~/.julia/packages/NNlib/3krvM/deps/build.log`
julia> import CuArrays
[ Info: Precompiling CuArrays [3a865a2d-5b23-5a0f-bc46-62713ec82fae]
┌ Debug: Precompiling CUDAapi [3895d2a7-ec45-59b8-82bb-cfc6a382f9b3]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAdrv [c5f51814-7f29-56b8-a69c-e4d8f6be1fde]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17]
└ @ Base loading.jl:1276
┌ Debug: Precompiling LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:1276
┌ Debug: Found LLVM v9.0.1 at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/bin/../lib/libLLVM-9.so with support for AArch64, AMDGPU, ARC, ARM, AVR, BPF, Hexagon, Lanai, MSP430, Mips, NVPTX, PowerPC, RISCV, Sparc, SystemZ, WebAssembly, X86, XCore
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:47
┌ Debug: Using LLVM.jl wrapper for LLVM v9.0
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:75
┌ Debug: Precompiling Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:1276
┌ Debug: Precompiling TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:1276
┌ Debug: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
└ @ Base loading.jl:1276
┌ Debug: Precompiling OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
└ @ Base loading.jl:1276
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for binary nvdisasm
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for binary nvdisasm
│ locations =
│ 20-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/bin"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/bin"
│ "/users/daluthge/bin"
│ "/gpfs/runtime/opt/python/3.7.4/bin"
│ "/gpfs/runtime/opt/git/2.20.2/bin"
│ "/gpfs/runtime/opt/gcc/8.3/bin"
│ "/gpfs/runtime/opt/binutils/2.31/bin"
│ ⋮
│ "/usr/bin"
│ "/usr/local/sbin"
│ "/usr/sbin"
│ "/usr/lpp/mmfs/bin"
│ "/usr/lpp/mmfs/sbin"
│ "/opt/ibutils/bin"
│ "/gpfs/runtime/bin"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found binary nvdisasm at /gpfs/runtime/opt/cuda/10.1.105/cuda/bin
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141
┌ Debug: CUDA toolkit identified as 10.1.105
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297
┌ Debug: Request to look for libdevice
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Look for libdevice
│ locations =
│ 2-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/nvvm/libdevice"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found unified device library at /gpfs/runtime/opt/cuda/10.1.105/cuda/nvvm/libdevice/libdevice.10.bc
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327
┌ Debug: Request to look for libcudadevrt
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│ locations =
│ 3-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found CUDA device runtime library libcudadevrt.a at /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379
┌ Debug: Request to look for library nvToolsExt
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0
│ locations =
│ 4-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib64"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libnvToolsExt.so at /gpfs/runtime/opt/cuda/10.1.105/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Request to look for library cupti
│ locations =
│ 2-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.1
│ locations =
│ 8-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/lib64"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/libx64"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/lib"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/lib64"
│ "/gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libcupti.so at /gpfs/runtime/opt/cuda/10.1.105/cuda/extras/CUPTI/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Using LLVM v9.0.1
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:170
┌ Debug: LLVM supports capabilities 2.0, 2.1, 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:191
┌ Debug: Using CUDA driver v10.2.0 and toolkit v10.1.0
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:196
┌ Debug: CUDA driver supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:213
┌ Debug: CUDA toolkit supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:214
┌ Debug: CUDAnative supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:174
┌ Debug: Precompiling GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:1276
┌ Debug: Precompiling AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:1276
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:1276
┌ Debug: Precompiling MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
└ @ Base loading.jl:1276
┌ Debug: Precompiling NNlib [872c559c-99b0-510c-b3b7-b6c96a88d5cd]
└ @ Base loading.jl:1276
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
CUDA 10.0.130: (works fine)
Click to expand
$ which nvcc
/gpfs/runtime/opt/cuda/10.0.130/cuda/bin/nvcc
$ which nvdisasm
/gpfs/runtime/opt/cuda/10.0.130/cuda/bin/nvdisasm
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
$ nvdisasm --version
nvdisasm: NVIDIA (R) CUDA disassembler
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:11_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130
bash-4.2$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.5.0-DEV.274 (2020-02-15)
_/ |\__'_|_|_|\__'_| | Commit 8eb0f9fefb (8 days old master)
|__/ |
julia> versioninfo(verbose = true)
Julia Version 1.5.0-DEV.274
Commit 8eb0f9fefb (2020-02-15 12:41 UTC)
Platform Info:
OS: Linux (x86_64-pc-linux-gnu)
"Red Hat Enterprise Linux Server release 7.3 (Maipo)"
uname: Linux 3.10.0-957.5.1.el7.x86_64 #1 SMP Wed Dec 19 10:46:58 EST 2018 x86_64 x86_64
CPU: Intel(R) Xeon(R) Gold 5122 CPU @ 3.60GHz:
speed user nice sys idle irq
#1 3601 MHz 260887968 s 209 s 79076153 s 314446557 s 0 s
#2 3601 MHz 225435375 s 556 s 66634534 s 362161063 s 0 s
#3 3601 MHz 153758324 s 331 s 47119535 s 453437851 s 0 s
#4 3601 MHz 133428293 s 355 s 44199325 s 476409768 s 0 s
#5 3601 MHz 231454591 s 286 s 61205506 s 362253151 s 0 s
#6 3601 MHz 154408009 s 604 s 45160867 s 455698340 s 0 s
#7 3601 MHz 104945700 s 268 s 34510038 s 515693325 s 0 s
#8 3601 MHz 94884863 s 496 s 31751307 s 528181552 s 0 s
Memory: 93.04103088378906 GB (80678.2890625 MB free)
Uptime: 6.578414e6 sec
Load Avg: 1.1494140625 0.908203125 0.515625
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-9.0.1 (ORCJIT, skylake)
Environment:
JULIA_DEBUG = all
JULIA_CUDA_VERBOSE = true
CPLUS_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
MANPATH = /gpfs/runtime/opt/python/3.7.4/share/man:/gpfs/runtime/opt/git/2.20.2/share/man:/gpfs/runtime/opt/gcc/8.3/share/man:/gpfs/runtime/opt/binutils/2.31/share/man:/gpfs/runtime/opt/intel/2017.0/man/common/man1:
TERM = xterm-256color
LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64:/gpfs/runtime/opt/cuda/10.0.130/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
CUDA_HOME = /gpfs/runtime/opt/cuda/10.0.130/cuda
LD_LIBRARY_PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64:/gpfs/runtime/opt/cuda/10.0.130/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64:/gpfs/runtime/opt/java/8u111/jre/lib/amd64
CPATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/include:/gpfs/runtime/opt/python/3.7.4/include:/gpfs/runtime/opt/gcc/8.3/include:/gpfs/runtime/opt/binutils/2.31/include:/gpfs/runtime/opt/intel/2017.0/mkl/include
NLSPATH = /gpfs/runtime/opt/intel/2017.0/lib/intel64/locale/en_US:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64/locale/en_US
PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/bin:/users/daluthge/bin:/gpfs/runtime/opt/python/3.7.4/bin:/gpfs/runtime/opt/git/2.20.2/bin:/gpfs/runtime/opt/gcc/8.3/bin:/gpfs/runtime/opt/binutils/2.31/bin:/gpfs/runtime/opt/intel/2017.0/bin:/gpfs/runtime/opt/matlab/R2017b/bin:/gpfs/runtime/opt/java/8u111/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/usr/lpp/mmfs/sbin:/opt/ibutils/bin:/gpfs/runtime/bin
C_INCLUDE_PATH = /gpfs/runtime/opt/gcc/8.3/include
LD_RUN_PATH = /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64:/gpfs/runtime/opt/cuda/10.0.130/cuda/lib:/gpfs/runtime/opt/python/3.7.4/lib:/gpfs/runtime/opt/gcc/8.3/lib64:/gpfs/runtime/opt/binutils/2.31/lib:/gpfs/runtime/opt/intel/2017.0/lib/intel64:/gpfs/runtime/opt/intel/2017.0/mkl/lib/intel64
JAVA_HOME = /gpfs/runtime/opt/java/8u111
MODULEPATH = /gpfs/runtime/modulefiles
HOME = /users/daluthge
IPP_PATH = /gpfs/runtime/opt/intel/2017.0/ipp
MODULEHOME = /gpfs/runtime/pymodules
PKG_CONFIG_PATH = /gpfs/runtime/opt/python/3.7.4/lib/pkgconfig
QT_PLUGIN_PATH = /usr/lib64/kde4/plugins:/usr/lib/kde4/plugins
julia> import Pkg
julia> Pkg.add("CuArrays")
Cloning default registries into `~/.julia`
######################################################################## 100.0%
Added registry `General` to `~/.julia/registries/General`
Resolving package versions...
Installed TimerOutputs ─────── v0.5.3
Installed Adapt ────────────── v1.0.1
Installed Requires ─────────── v1.0.1
Installed AbstractFFTs ─────── v0.5.0
Installed CUDAapi ──────────── v3.1.0
Installed GPUArrays ────────── v2.0.1
Installed CEnum ────────────── v0.2.0
Installed OrderedCollections ─ v1.1.0
Installed DataStructures ───── v0.17.9
Installed CuArrays ─────────── v1.7.2
Installed MacroTools ───────── v0.5.4
Installed BinaryProvider ───── v0.5.8
Installed CUDAnative ───────── v2.10.2
Installed NNlib ────────────── v0.6.4
Installed CUDAdrv ──────────── v6.0.0
Installed LLVM ─────────────── v1.3.3
Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Project.toml`
[3a865a2d] + CuArrays v1.7.2
Updating `/gpfs_home/daluthge/.julia/environments/v1.5/Manifest.toml`
[621f4979] + AbstractFFTs v0.5.0
[79e6a3ab] + Adapt v1.0.1
[b99e7846] + BinaryProvider v0.5.8
[fa961155] + CEnum v0.2.0
[3895d2a7] + CUDAapi v3.1.0
[c5f51814] + CUDAdrv v6.0.0
[be33ccc6] + CUDAnative v2.10.2
[3a865a2d] + CuArrays v1.7.2
[864edb3b] + DataStructures v0.17.9
[0c68f7d7] + GPUArrays v2.0.1
[929cbde3] + LLVM v1.3.3
[1914dd2f] + MacroTools v0.5.4
[872c559c] + NNlib v0.6.4
[bac558e1] + OrderedCollections v1.1.0
[ae029012] + Requires v1.0.1
[a759f4b9] + TimerOutputs v0.5.3
[2a0f44e3] + Base64
[8ba89e20] + Distributed
[b77e0a4c] + InteractiveUtils
[8f399da3] + Libdl
[37e2e46d] + LinearAlgebra
[56ddb016] + Logging
[d6f4376e] + Markdown
[de0858da] + Printf
[9a3f8284] + Random
[ea8e919c] + SHA
[9e88b42a] + Serialization
[6462fe0b] + Sockets
[2f01184e] + SparseArrays
[10745b16] + Statistics
[8dfed614] + Test
[cf7118a7] + UUIDs
[4ec0a83e] + Unicode
Building NNlib → `~/.julia/packages/NNlib/3krvM/deps/build.log`
julia> import CuArrays
[ Info: Precompiling CuArrays [3a865a2d-5b23-5a0f-bc46-62713ec82fae]
┌ Debug: Precompiling CUDAapi [3895d2a7-ec45-59b8-82bb-cfc6a382f9b3]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAdrv [c5f51814-7f29-56b8-a69c-e4d8f6be1fde]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CEnum [fa961155-64e5-5f13-b03f-caf6b980ea82]
└ @ Base loading.jl:1276
┌ Debug: Precompiling CUDAnative [be33ccc6-a3ff-5ff2-a52e-74243cff1e17]
└ @ Base loading.jl:1276
┌ Debug: Precompiling LLVM [929cbde3-209d-540e-8aea-75f648917ca0]
└ @ Base loading.jl:1276
┌ Debug: Found LLVM v9.0.1 at /gpfs_home/daluthge/dev/JuliaLang/julia/usr/bin/../lib/libLLVM-9.so with support for AArch64, AMDGPU, ARC, ARM, AVR, BPF, Hexagon, Lanai, MSP430, Mips, NVPTX, PowerPC, RISCV, Sparc, SystemZ, WebAssembly, X86, XCore
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:47
┌ Debug: Using LLVM.jl wrapper for LLVM v9.0
└ @ LLVM ~/.julia/packages/LLVM/DAnFH/src/LLVM.jl:75
┌ Debug: Precompiling Adapt [79e6a3ab-5dfb-504d-930d-738a2a938a0e]
└ @ Base loading.jl:1276
┌ Debug: Precompiling TimerOutputs [a759f4b9-e2f1-59dc-863e-4aeb61b1ea8f]
└ @ Base loading.jl:1276
┌ Debug: Precompiling DataStructures [864edb3b-99cc-5e75-8d2d-829cb0a9cfe8]
└ @ Base loading.jl:1276
┌ Debug: Precompiling OrderedCollections [bac558e1-5e72-5ebc-8fee-abe8a469f55d]
└ @ Base loading.jl:1276
┌ Debug: Looking for CUDA toolkit via environment variables CUDA_HOME
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Request to look for binary nvdisasm
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for binary nvdisasm
│ locations =
│ 20-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/bin"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/bin"
│ "/users/daluthge/bin"
│ "/gpfs/runtime/opt/python/3.7.4/bin"
│ "/gpfs/runtime/opt/git/2.20.2/bin"
│ "/gpfs/runtime/opt/gcc/8.3/bin"
│ "/gpfs/runtime/opt/binutils/2.31/bin"
│ ⋮
│ "/usr/bin"
│ "/usr/local/sbin"
│ "/usr/sbin"
│ "/usr/lpp/mmfs/bin"
│ "/usr/lpp/mmfs/sbin"
│ "/opt/ibutils/bin"
│ "/gpfs/runtime/bin"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found binary nvdisasm at /gpfs/runtime/opt/cuda/10.0.130/cuda/bin
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:141
┌ Debug: CUDA toolkit identified as 10.0.130
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:297
┌ Debug: Request to look for libdevice
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Look for libdevice
│ locations =
│ 2-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/nvvm/libdevice"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found unified device library at /gpfs/runtime/opt/cuda/10.0.130/cuda/nvvm/libdevice/libdevice.10.bc
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:327
┌ Debug: Request to look for libcudadevrt
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for CUDA device runtime library libcudadevrt.a
│ locations =
│ 3-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found CUDA device runtime library libcudadevrt.a at /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:379
┌ Debug: Request to look for library nvToolsExt
│ locations =
│ 1-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libnvToolsExt.so, libnvToolsExt.so.1, libnvToolsExt.so.1.0
│ locations =
│ 4-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libnvToolsExt.so at /gpfs/runtime/opt/cuda/10.0.130/cuda/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Request to look for library cupti
│ locations =
│ 2-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Looking for library libcupti.so, libcupti.so.10, libcupti.so.10.0
│ locations =
│ 8-element Array{String,1}:
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/lib64"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/libx64"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/lib"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/lib64"
│ "/gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/libx64"
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/CUDAapi.jl:8
┌ Debug: Found library libcupti.so at /gpfs/runtime/opt/cuda/10.0.130/cuda/extras/CUPTI/lib64
└ @ CUDAapi ~/.julia/packages/CUDAapi/wYUAO/src/discovery.jl:90
┌ Debug: Using LLVM v9.0.1
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:170
┌ Debug: LLVM supports capabilities 2.0, 2.1, 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.3 and 6.4
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:191
┌ Debug: Using CUDA driver v10.2.0 and toolkit v10.0.0
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:196
┌ Debug: CUDA driver supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2, 6.3, 6.4 and 6.5
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:213
┌ Debug: CUDA toolkit supports capabilities 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5 with PTX 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 2.0, 2.1, 2.2, 2.3, 3.0, 3.1, 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1, 6.2 and 6.3
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/compatibility.jl:214
┌ Debug: CUDAnative supports devices 3.0, 3.2, 3.5, 3.7, 5.0, 5.2, 5.3, 6.0, 6.1, 6.2, 7.0, 7.2 and 7.5; PTX 3.2, 4.0, 4.1, 4.2, 4.3, 5.0, 6.0, 6.1 and 6.3
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:174
┌ Debug: Precompiling GPUArrays [0c68f7d7-f131-5f86-a1c3-88cf8149b2d7]
└ @ Base loading.jl:1276
┌ Debug: Precompiling AbstractFFTs [621f4979-c628-5d54-868e-fcf4e3e8185c]
└ @ Base loading.jl:1276
┌ Debug: Precompiling Requires [ae029012-a4dd-5104-9daa-d747884805df]
└ @ Base loading.jl:1276
┌ Debug: Precompiling MacroTools [1914dd2f-81c6-5fcd-8719-6d5c9610ff09]
└ @ Base loading.jl:1276
┌ Debug: Precompiling NNlib [872c559c-99b0-510c-b3b7-b6c96a88d5cd]
└ @ Base loading.jl:1276
┌ Warning: Incompatibility detected between CUDA and LLVM 8.0+; disabling debug info emission for CUDA kernels
└ @ CUDAnative ~/.julia/packages/CUDAnative/hfulr/src/CUDAnative.jl:114
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment