[SYCL][CUDA] Multiple backends

[jpola]

Dear all,
I tried to check the test case where the one program can work on multiple backends but it doesn't seem to work for me.

• Currently I'm using: 2f64227 commit.
• Tot test the feature I used: lvm/sycl/test/plugins/sycl-targets-order.cpp source.
• I compiled the test using: clang++ -fsycl -fsycl-targets=spir64-unknown-unknown-sycldevice,nvptx64-nvidia-cuda-sycldevice ./sycl-targets-order.cpp -o tgt-spir64-nvptx64.out -v

when I executed the program i have following output:

./tgt-spir64-nvptx64.out 
Running on SYCL device Quadro RTX 5000, driver version CUDA 11.0
terminate called after throwing an instance of 'cl::sycl::runtime_error'
  what():  OpenCL API failed. OpenCL API returns: -42 (CL_INVALID_BINARY) -42 (CL_INVALID_BINARY)
Aborted (core dumped)

SYCL_BE=PI_OPENCL ./tgt-spir64-nvptx64.out 
Running on SYCL device Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, driver version 2020.10.7.0.15
The results are correct!

The clinfo detects following platforms on my machine.

Platform #0: Intel(R) OpenCL
`-- Device #0: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
Platform #1: NVIDIA CUDA
`-- Device #0: Quadro RTX 5000

Could you plase advice on that? How can I enable both backends?

Thanks.

Here is the compilation log:

clang version 12.0.0 (https://github.com/intel/llvm 2f64227a0405e6d4eacdf0974fb8e64462d01a8d)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/jpola/sycl_workspace/llvm/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda-10.2, version 10.2
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple spir64-unknown-unknown-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -mllvm -sycl-opt -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -emit-llvm-bc -emit-llvm-uselists -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -fno-verbose-asm -mconstructor-aliases -aux-target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -faddrsig -o /tmp/sycl-targets-order-db613b.bc -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/home/jpola/sycl_workspace/llvm/build/bin/llvm-link" /tmp/sycl-targets-order-db613b.bc -o /tmp/sycl-targets-order-285b65.bc --suppress-warnings
 "/home/jpola/sycl_workspace/llvm/build/bin/sycl-post-link" -symbols -spec-const=rt -o /tmp/sycl-targets-order-f01eea.table /tmp/sycl-targets-order-285b65.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/file-table-tform" -extract=Code -drop_titles -o /tmp/sycl-targets-order-6447e2.txt /tmp/sycl-targets-order-f01eea.table
 "/home/jpola/sycl_workspace/llvm/build/bin/llvm-foreach" --in-file-list=/tmp/sycl-targets-order-6447e2.txt --in-replace=/tmp/sycl-targets-order-6447e2.txt --out-ext=spv --out-file-list=/tmp/sycl-targets-order-111dc8.txt --out-replace=/tmp/sycl-targets-order-111dc8.txt -- /home/jpola/sycl_workspace/llvm/build/bin/llvm-spirv -o /tmp/sycl-targets-order-111dc8.txt -spirv-max-version=1.1 -spirv-ext=+all,-SPV_INTEL_usm_storage_classes /tmp/sycl-targets-order-6447e2.txt
 "/home/jpola/sycl_workspace/llvm/build/bin/file-table-tform" -replace=Code,Code -o /tmp/sycl-targets-order-f935a6.table /tmp/sycl-targets-order-f01eea.table /tmp/sycl-targets-order-111dc8.txt
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-offload-wrapper" -o=/tmp/wrapper-a65b53.bc -host=x86_64-unknown-linux-gnu -target=spir64 -kind=sycl -batch /tmp/sycl-targets-order-f935a6.table
 "/home/jpola/sycl_workspace/llvm/build/bin/llc" -filetype=obj -o /tmp/tgt-spir64-nvptx64-0ce575.o /tmp/wrapper-a65b53.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple nvptx64-nvidia-cuda-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -emit-llvm-bc -emit-llvm-uselists -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fno-rounding-math -fno-verbose-asm -no-integrated-as -aux-target-cpu x86-64 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -mlink-builtin-bitcode /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/../../clc/libspirv-nvptx64--nvidiacl.bc -mlink-builtin-bitcode /usr/local/cuda-10.2/nvvm/libdevice/libdevice.10.bc -target-feature +ptx65 -target-sdk-version=10.2 -target-cpu sm_50 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fno-dwarf-directory-asm -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -o /tmp/sycl-targets-order-5b7a26.bc -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/bin/../include/sycl"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/home/jpola/sycl_workspace/llvm/build/bin/llvm-link" /tmp/sycl-targets-order-5b7a26.bc -o /tmp/sycl-targets-order-c37d03.bc --suppress-warnings
 "/home/jpola/sycl_workspace/llvm/build/bin/sycl-post-link" -ir-output-only -spec-const=default -o /tmp/sycl-targets-order-5fd947.bc /tmp/sycl-targets-order-c37d03.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple nvptx64-nvidia-cuda-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -S -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fno-rounding-math -fno-verbose-asm -no-integrated-as -aux-target-cpu x86-64 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -mlink-builtin-bitcode /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/../../clc/libspirv-nvptx64--nvidiacl.bc -mlink-builtin-bitcode /usr/local/cuda-10.2/nvvm/libdevice/libdevice.10.bc -target-feature +ptx65 -target-sdk-version=10.2 -target-cpu sm_50 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -fno-dwarf-directory-asm -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcolor-diagnostics -o /tmp/sycl-targets-order-f44eab.s -x ir /tmp/sycl-targets-order-5fd947.bc
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
 "/usr/local/cuda-10.2/bin/ptxas" -m64 -O0 -v --gpu-name sm_50 --output-file /tmp/sycl-targets-order-342807.o /tmp/sycl-targets-order-f44eab.s
ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function '_ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer_with_offset' for 'sm_50'
ptxas info    : Function properties for _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer_with_offset
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 10 registers, 364 bytes cmem[0]
ptxas info    : Compiling entry function '_ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer' for 'sm_50'
ptxas info    : Function properties for _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 7 registers, 352 bytes cmem[0]
 "/usr/local/cuda-10.2/bin/fatbinary" -64 --create /tmp/sycl-targets-order-c523f4.fatbin --image=profile=compute_50,file=/tmp/sycl-targets-order-f44eab.s --image=profile=sm_50,file=/tmp/sycl-targets-order-342807.o
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-offload-wrapper" -o=/tmp/wrapper-4e717c.bc -host=x86_64-unknown-linux-gnu -target=nvptx64 -kind=sycl /tmp/sycl-targets-order-c523f4.fatbin
 "/home/jpola/sycl_workspace/llvm/build/bin/llc" -filetype=obj -o /tmp/tgt-spir64-nvptx64-9b7188.o /tmp/wrapper-4e717c.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple spir64-unknown-unknown-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -mllvm -sycl-opt -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -fsyntax-only -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -fno-verbose-asm -mconstructor-aliases -aux-target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -fsycl-int-header=/tmp/sycl-targets-order-960d4a.h -faddrsig -o /tmp/sycl-targets-order-960d4a.h -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple x86_64-unknown-linux-gnu -sycl-std=2017 -emit-obj -mrelax-all --mrelax-relocations -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -mconstructor-aliases -munwind-tables -target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -include /tmp/sycl-targets-order-960d4a.h -dependency-filter /tmp/sycl-targets-order-960d4a.h -fsycl -fsycl-is-host -faddrsig -o /tmp/sycl-targets-order-87fa0f.o -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/bin/../include/sycl"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o tgt-spir64-nvptx64.out /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/9/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/x86_64-linux-gnu/../../lib64 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../.. -L/home/jpola/sycl_workspace/llvm/build/bin/../lib -L/lib -L/usr/lib -L/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/lib -L/home/jpola/intel/oneapi/ippcp/latest/lib/intel64 -L/home/jpola/intel/oneapi/ccl/2021.1-beta08/lib/cpu_gpu_dpcpp -L/home/jpola/intel/oneapi/compiler/latest/linux/lib -L/home/jpola/intel/oneapi/mkl/2021.1-beta08/lib/intel64 -L/home/jpola/intel/oneapi/vpl/latest/lib -L/home/jpola/intel/oneapi/daal/latest/lib/intel64 -L/home/jpola/intel/oneapi/ipp/latest/lib/intel64 -L/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../lib/intel64/gcc4.8 -L/home/jpola/intel/oneapi/mpi/2021.1-beta08//libfabric/lib -L/home/jpola/intel/oneapi/mpi/2021.1-beta08//lib/release -L/home/jpola/intel/oneapi/mpi/2021.1-beta08//lib /tmp/sycl-targets-order-87fa0f.o /tmp/tgt-spir64-nvptx64-0ce575.o /tmp/tgt-spir64-nvptx64-9b7188.o -lstdc++ -lm -lgcc_s -lgcc -lsycl -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/9/crtend.o /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crtn.o

[dm-vodopyanov]

Hi @jpola, do you observe some regression? Could you please check if it works for you on the previous commit 047e2ec?
It will be really helpful if you enable some extra logs:

• in sycl/source/detail/program_manager/program_manager.cpp change line 43 static constexpr int DbgProgMgr = 0; to static constexpr int DbgProgMgr = 1;
• run the binary with SYCL_PI_TRACE=-1 env variable enabled, e.g., SYCL_PI_TRACE=-1 ./tgt-spir64-nvptx64.out

[jpola]

Hi @dm-vodopyanov
I'm obtaining the same results with the 2f64227 and 047e2ec comits.

Here are several tests I performed with the DbgProgMgr = 1.

I compiled the the test using following command: clang++ ./sycl-targets-order.cpp -o tgt-spir64-nvptx64.out -v -fsycl -fsycl-targets=nvptx64-nvidia-cuda-sycldevice,spir64-unknown-unknown-sycldevice. The order of triads does not matter.

Se attached log files for a trace outputs.
both_triads-be_any.log
both_triads-be_opencl.log
nv_triad-be_any.log
nv_triad-be_opencl.log

When both backends are available the CUDA backend somehow is not working. Picks upt wrong kernel image?
$ SYCL_PI_TRACE=-1 ./tgt-spir64-nvptx64.out &> both_triads-be_any.log

Running on SYCL device Quadro RTX 5000, driver version CUDA 11.0
>>> ProgramManager::getOrCreateKernel(-1, 0x17f0a20, 0x17fd320, _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer)
>>> ProgramManager::getDeviceImage(-1, "2", 0x17f0a20, 0x17fd320, 0)
available device images:
  ++++++ Kernel set: 2
  --- Image 0x40db30
    Version  : 2
    Kind     : 4
    Format   : 0
    Target   : spir64
    Bin size : 1236
    Compile options : 
    Link options    : 
    Entries  : _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer 
    Properties [0x40dae0-0x40db10]:
      Category SYCL/devicelib req mask [0x40dac0-0x40dae0]:
        [UINT32] DeviceLibReqMask=1
      Category SYCL/specialization constants [0-0]:
    OSModuleHandle=-1
  ++++++ Kernel set: 1
  --- Image 0x40d990
    Version  : 2
    Kind     : 4
    Format   : 0
    Target   : nvptx64
    Bin size : 5144
    Compile options : 
    Link options    : 
    Entries  : 
    Properties [0-0]:
    OSModuleHandle=-1
terminate called after throwing an instance of 'cl::sycl::runtime_error'
  what():  OpenCL API failed. OpenCL API returns: -42 (CL_INVALID_BINARY) -42 (CL_INVALID_BINARY)
Aborted (core dumped)

The OpenCL backend seems to work correctly.
SYCL_BE=PI_OPENCL SYCL_PI_TRACE=-1 ./tgt-spir64-nvptx64.out &> both_triads-be_opencl.log

Running on SYCL device Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, driver version 2020.10.7.0.15
>>> ProgramManager::getOrCreateKernel(-1, 0x21bcc40, _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer)
>>> ProgramManager::getDeviceImage(-1, "2", 0x21bcc40)
available device images:
  ++++++ Kernel set: 2
  --- Image 0x40db30
    Version  : 2
    Kind     : 4
    Format   : 0
    Target   : spir64
    Bin size : 1236
    Compile options : 
    Link options    : 
    Entries  : _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer 
    Properties [0x40dae0-0x40db10]:
      Category SYCL/devicelib req mask [0x40dac0-0x40dae0]:
        [UINT32] DeviceLibReqMask=1
      Category SYCL/specialization constants [0-0]:
    OSModuleHandle=-1
  ++++++ Kernel set: 1
  --- Image 0x40d990
    Version  : 2
    Kind     : 4
    Format   : 0
    Target   : nvptx64
    Bin size : 5144
    Compile options : 
    Link options    : 
    Entries  : 
    Properties [0-0]:
    OSModuleHandle=-1
selected device image: 0x40db30
  --- Image 0x40db30
    Version  : 2
    Kind     : 4
    Format   : 0
    Target   : spir64
    Bin size : 1236
    Compile options : 
    Link options    : 
    Entries  : _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer 
    Properties [0x40dae0-0x40db10]:
      Category SYCL/devicelib req mask [0x40dac0-0x40dae0]:
        [UINT32] DeviceLibReqMask=1
      Category SYCL/specialization constants [0-0]:
    OSModuleHandle=-1
>>> ProgramManager::createPIProgram(0x2194110)
>>> ProgramManager::build(0x36c8b28, , , ... 1)
The results are correct!

When I compile the test using the nvidia triad:
clang++ ./sycl-targets-order.cpp -o tgt-spir64-nvptx64.out -v -fsycl -fsycl-targets=nvptx64-nvidia-cuda-sycldevice

The CUDA backend works correctly.
$ SYCL_PI_TRACE=-1 ./tgt-spir64-nvptx64.out &> nv_triad-be_any.log

./tgt-spir64-nvptx64.out
Running on SYCL device Quadro RTX 5000, driver version CUDA 11.0
>>> ProgramManager::getOrCreateKernel(-1, 0x1ac37b0, _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer)
>>> ProgramManager::getDeviceImage(-1, "1", 0x1ac37b0)
available device images:
  ++++++ Kernel set: 1
  --- Image 0x40d990
    Version  : 2
    Kind     : 4
    Format   : 0
    Target   : nvptx64
    Bin size : 5144
    Compile options : 
    Link options    : 
    Entries  : 
    Properties [0-0]:
    OSModuleHandle=-1
selected device image: 0x40d990
  --- Image 0x40d990
    Version  : 2
    Kind     : 4
    Format   : 0
    Target   : nvptx64
    Bin size : 5144
    Compile options : 
    Link options    : 
    Entries  : 
    Properties [0-0]:
    OSModuleHandle=-1
>>> ProgramManager::createPIProgram(0x1a9a0d0)
>>> ProgramManager::build(0x2038ea0, , , ... 1)
The results are correct!

The OpenCL backend fails as expected.
$ SYCL_BE=PI_OPENCL SYCL_PI_TRACE=-1 ./tgt-spir64-nvptx64.out &> nv_triad-be_opencl.log

Running on SYCL device Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, driver version 2020.10.7.0.15
>>> ProgramManager::getOrCreateKernel(-1, 0xff99d0, _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer)
>>> ProgramManager::getDeviceImage(-1, "1", 0xff99d0)
available device images:
  ++++++ Kernel set: 1
  --- Image 0x40d990
    Version  : 2
    Kind     : 4
    Format   : 0
    Target   : nvptx64
    Bin size : 5144
    Compile options : 
    Link options    : 
    Entries  : 
    Properties [0-0]:
    OSModuleHandle=-1
terminate called after throwing an instance of 'cl::sycl::runtime_error'
  what():  OpenCL API failed. OpenCL API returns: -42 (CL_INVALID_BINARY) -42 (CL_INVALID_BINARY)
PLEASE submit a bug report to https://software.intel.com/en-us/support/priority-support and include the crash backtrace.
Aborted (core dumped)

[hiaselhans]

Hi @jpola

I did some investigations on this some time ago.
The reason seems to be that when compiling for the cuda backend the binaries don't include the table with all the symbol names.
So, the programmanager only selects from the opencl binaries and thus fails to find one...

here is what i found in the past but didn't have the time to continue:
#1588
#1670

[AidanBeltonS]

I believe this issue has been resolved, though I am unsure which patches fixed it.
From trying to reproduce the problem the error is not being created any more.

$ clang++ -fsycl -fsycl-targets=spir64-unknown-unknown-sycldevice,nvptx64-nvidia-cuda-sycldevice ./sycl-targets-order.cpp -o tgt-spir64-nvptx64.out -w
$ ./tgt-spir64-nvptx64.out 
Running on SYCL device TITAN RTX, driver version CUDA 11.0
The results are correct!

$ SYCL_BE=PI_OPENCL ./tgt-spir64-nvptx64.out 
WARNING: The legacy environment variables SYCL_BE and SYCL_DEVICE_TYPE are deprecated. Please use SYCL_DEVICE_FILTER instead. For details, please refer to https://github.com/intel/llvm/blob/sycl/sycl/doc/EnvironmentVariables.md
Running on SYCL device Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz, driver version 2020.10.6.0.04
The results are correct!

$ SYCL_BE=PI_CUDA ./tgt-spir64-nvptx64.out 
WARNING: The legacy environment variables SYCL_BE and SYCL_DEVICE_TYPE are deprecated. Please use SYCL_DEVICE_FILTER instead. For details, please refer to https://github.com/intel/llvm/blob/sycl/sycl/doc/EnvironmentVariables.md

Running on SYCL device TITAN RTX, driver version CUDA 11.0
The results are correct!

I believe the issue can be closed.
@jpola can you confirm that the issue has been resolved?

[jpola]

Unfortunately, I cannot confirm at this time. Please skip me in the comments, but thanks for your help.

[AerialMantis]

No problem, thank you for your reply, as we believe this issue to be resolved now we will close the ticket, if you are able to attempt this again and still experience the issue please feel free to comment and we can re-open the ticket.