Skip to content
This repository has been archived by the owner on Mar 12, 2021. It is now read-only.
This repository has been archived by the owner on Mar 12, 2021. It is now read-only.

Test failures on Tesla K20c/CUDA 9.0 #594

Closed

Description

I get test failures when I run the tests locally:

julia> ENV["JULIA_DEBUG"]="CUDAapi"
"CUDAapi"

julia> include("runtests.jl")
┌ Debug: CUDA toolkit identified as 9.0.176
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:297
┌ Debug: Request to look for library cublas
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcublas.so, libcublas.so.9, libcublas.so.9.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Found library libcublas.so at /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:90
┌ Debug: Request to look for library cusparse
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcusparse.so, libcusparse.so.9, libcusparse.so.9.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Found library libcusparse.so at /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:90
┌ Debug: Request to look for library cusolver
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcusolver.so, libcusolver.so.9, libcusolver.so.9.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Found library libcusolver.so at /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:90
┌ Debug: Request to look for library cufft
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcufft.so, libcufft.so.9, libcufft.so.9.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Found library libcufft.so at /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:90
┌ Debug: Request to look for library curand
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcurand.so, libcurand.so.9, libcurand.so.9.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Found library libcurand.so at /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/discovery.jl:90
┌ Debug: Request to look for library cudnn
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcudnn.so, libcudnn.so.7, libcudnn.so.7.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Request to look for library cutensor
│   locations =
│    1-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
┌ Debug: Looking for library libcutensor.so, libcutensor.so.1, libcutensor.so.1.0
│   locations =
│    4-element Array{String,1}:
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c"       
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib"   
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib64" 
│     "/home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/libx64"
└ @ CUDAapi ~/.julia/dev/CUDAapi/src/CUDAapi.jl:8
[ Info: Testing using device Tesla K20c (compute capability 3.5.0, 4.873 GiB available memory) on CUDA driver 9.0.0 and toolkit 9.0.176
┌ Warning: Performing scalar operations on GPU arrays: This is very slow, consider disallowing these operations with `allowscalar(false)`
└ @ GPUArrays ~/.julia/packages/GPUArrays/1wgPO/src/indexing.jl:16
Array: Test Failed at /home/tim/.julia/dev/CuArrays/test/base.jl:44
  Expression: cu([1]) isa AbstractArray{Int}
   Evaluated: Float32[1.0] isa AbstractArray{Int64,N} where N
Stacktrace:
 [1] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:44
 [2] top-level scope at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
 [3] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:34
SubArray: Error During Test at /home/tim/.julia/dev/CuArrays/test/base.jl:173
  Got exception outside of a @test
  ArgumentError: invalid index: 1.0f0 of type Float32
  Stacktrace:
   [1] to_index(::Float32) at ./indices.jl:273
   [2] to_index(::Array{Float32,1}, ::Float32) at ./indices.jl:250
   [3] to_indices at ./indices.jl:301 [inlined]
   [4] to_indices at ./indices.jl:298 [inlined]
   [5] getindex at ./abstractarray.jl:981 [inlined]
   [6] getindex at ./subarray.jl:230 [inlined]
   [7] isassigned(::SubArray{Float32,1,Array{Float32,1},Tuple{Array{Float32,1}},false}, ::Int64) at ./abstractarray.jl:405
   [8] show_delim_array(::IOContext{Base.DevNull}, ::SubArray{Float32,1,Array{Float32,1},Tuple{Array{Float32,1}},false}, ::Char, ::String, ::Char, ::Bool, ::Int64, ::Int64) at ./show.jl:661
   [9] show_delim_array at ./show.jl:654 [inlined]
   [10] show_vector(::Base.DevNull, ::SubArray{Float32,1,Array{Float32,1},Tuple{Array{Float32,1}},false}, ::Char, ::Char) at ./arrayshow.jl:443
   [11] show_vector(::Base.DevNull, ::SubArray{Float32,1,Array{Float32,1},Tuple{Array{Float32,1}},false}) at ./arrayshow.jl:432
   [12] show_vector(::Base.DevNull, ::SubArray{Float32,1,CuArray{Float32,1,Nothing},Tuple{CuArray{Float32,1,Nothing}},false}) at /home/tim/.julia/packages/GPUArrays/1wgPO/src/abstractarray.jl:71
   [13] show(::Base.DevNull, ::SubArray{Float32,1,CuArray{Float32,1,Nothing},Tuple{CuArray{Float32,1,Nothing}},false}) at ./arrayshow.jl:418
   [14] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:237
   [15] top-level scope at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [16] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:174
   [17] include at ./boot.jl:328 [inlined]
   [18] include_relative(::Module, ::String) at ./loading.jl:1105
   [19] include(::Module, ::String) at ./Base.jl:31
   [20] include(::String) at ./client.jl:424
   [21] top-level scope at /home/tim/.julia/dev/CuArrays/test/runtests.jl:46
   [22] top-level scope at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [23] top-level scope at /home/tim/.julia/dev/CuArrays/test/runtests.jl:38
   [24] include at ./boot.jl:328 [inlined]
   [25] include_relative(::Module, ::String) at ./loading.jl:1105
   [26] include(::Module, ::String) at ./Base.jl:31
   [27] include(::String) at ./client.jl:424
   [28] top-level scope at REPL[5]:1
   [29] eval(::Module, ::Any) at ./boot.jl:330
   [30] eval_user_input(::Any, ::REPL.REPLBackend) at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:86
   [31] run_backend(::REPL.REPLBackend) at /home/tim/.julia/packages/Revise/ZOWOa/src/Revise.jl:1057
   [32] (::Revise.var"#85#87"{REPL.REPLBackend})() at ./task.jl:333
  
aliasing: Error During Test at /home/tim/.julia/dev/CuArrays/test/base.jl:420
  Got exception outside of a @test
  ArgumentError: an array of type `CuArray` shares memory with another argument and must
  make a preventative copy of itself in order to maintain consistent semantics,
  but `copy(A)` returns a new array of type `CuArray{Int64,1,Nothing}`. To fix, implement:
      `Base.unaliascopy(A::CuArray)::typeof(A)`
  Stacktrace:
   [1] _unaliascopy(::CuArray{Int64,1,CuArray{Int64,1,Nothing}}, ::CuArray{Int64,1,Nothing}) at ./abstractarray.jl:1174
   [2] unaliascopy(::CuArray{Int64,1,CuArray{Int64,1,Nothing}}) at ./abstractarray.jl:1172
   [3] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:428
   [4] top-level scope at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [5] top-level scope at /home/tim/.julia/dev/CuArrays/test/base.jl:421
   [6] include at ./boot.jl:328 [inlined]
   [7] include_relative(::Module, ::String) at ./loading.jl:1105
   [8] include(::Module, ::String) at ./Base.jl:31
   [9] include(::String) at ./client.jl:424
   [10] top-level scope at /home/tim/.julia/dev/CuArrays/test/runtests.jl:46
   [11] top-level scope at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/Test/src/Test.jl:1107
   [12] top-level scope at /home/tim/.julia/dev/CuArrays/test/runtests.jl:38
   [13] include at ./boot.jl:328 [inlined]
   [14] include_relative(::Module, ::String) at ./loading.jl:1105
   [15] include(::Module, ::String) at ./Base.jl:31
   [16] include(::String) at ./client.jl:424
   [17] top-level scope at REPL[5]:1
   [18] eval(::Module, ::Any) at ./boot.jl:330
   [19] eval_user_input(::Any, ::REPL.REPLBackend) at /usr/local/julia/julia-1/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:86
   [20] run_backend(::REPL.REPLBackend) at /home/tim/.julia/packages/Revise/ZOWOa/src/Revise.jl:1057
   [21] (::Revise.var"#85#87"{REPL.REPLBackend})() at ./task.jl:333
  
julia: symbol lookup error: /home/tim/.julia/artifacts/7f734dc78dfc6bfb8bdced4db84076c75131b09c/lib/libcusolver.so: undefined symbol: omp_get_max_threads

Build log

tim@cannon:~/.julia/dev/CuArrays/deps$ cat build.log 
Could not find library 'cusolver'.
Could not find library 'cudnn'.

CuArrays.jl has been built successfully, but there were warnings.
Some functionality may be unavailable.

Environment details (please complete this section)
Details on Julia:

Julia Version 1.3.2-pre.0
Commit 2e6715c (2019-12-31 00:49 UTC)
Platform Info:
  OS: Linux (x86_64-linux-gnu)
  CPU: Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, sandybridge)
Environment:
  JULIA_CUDA_VERBOSE = 1

Julia packages:

  • CuArrays.jl:
  • CUDAnative.jl:
  • ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions