Skip to content

[SYCL][CUDA] Fixes for multiple backends in the same program #1252

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Mar 25, 2020

Conversation

Ruyk
Copy link
Contributor

@Ruyk Ruyk commented Mar 5, 2020

This pull request contains two commits:

  • bcb51a1: Implementation of piEventSetCallback with tests
  • a979f69: A bugfix on the SYCL RT that used the wrong plugin to create GlueEvents

This two patches should fix #1231, but should not be merged until we run some additional testing on our side.

@Ruyk
Copy link
Contributor Author

Ruyk commented Mar 5, 2020

@fwyzard could you check if this branch fixes your problems in #1231 ?

@bader bader added the cuda CUDA back-end label Mar 5, 2020
@fwyzard
Copy link
Contributor

fwyzard commented Mar 5, 2020

Yes, it works:

$ ./simple-sycl-app cuda cpu cuda cpu
Available SYCL platforms:
  - Intel(R) OpenCL, driver version OpenCL 2.1 LINUX
    - Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz, driver version 2020.9.1.0.18

  - Intel(R) OpenCL HD Graphics, driver version OpenCL 2.1 
    - Intel(R) Gen9 HD Graphics NEO, driver version 20.08.15750

  - NVIDIA CUDA, driver version CUDA 10.2
    - Tesla K40c, driver version CUDA 10.2

  - SYCL host platform, driver version 1.2
    - SYCL host device, driver version 1.2

Running on SYCL device Tesla K40c, driver version CUDA 10.2
The results are correct!

Running on SYCL device Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz, driver version 2020.9.1.0.18
The results are correct!

Running on SYCL device Tesla K40c, driver version CUDA 10.2
The results are correct!

Running on SYCL device Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz, driver version 2020.9.1.0.18
The results are correct!

Thank you for the fixes !

fwyzard added a commit to cms-patatrack/patatrack-website that referenced this pull request Mar 5, 2020
@Ruyk
Copy link
Contributor Author

Ruyk commented Mar 10, 2020

@steffenlarsen

@steffenlarsen steffenlarsen force-pushed the ruyman/event-callback branch 3 times, most recently from 3595834 to 8a642f2 Compare March 10, 2020 16:35
@steffenlarsen
Copy link
Contributor

steffenlarsen commented Mar 10, 2020

An additional commit has been added:

This ensures that CUDA operations are allowed in device-event callbacks, which could be needed if e.g. an event ends its life in the callback, by introducing a worker-thread that sleeps until callbacks are added to its work-queue.

Edit: This commit has been dropped.

@steffenlarsen steffenlarsen force-pushed the ruyman/event-callback branch from 8a642f2 to f6904a8 Compare March 10, 2020 17:20
@steffenlarsen steffenlarsen force-pushed the ruyman/event-callback branch 2 times, most recently from 968e7e9 to 5baf374 Compare March 11, 2020 11:12
@romanovvlad
Copy link
Contributor

romanovvlad commented Mar 11, 2020

An additional commit has been added:

* [8a642f2](https://github.com/intel/llvm/commit/8a642f2847b6e19f0ced765bbbdd998ead428041)

This ensures that CUDA operations are allowed in device-event callbacks, which could be needed if e.g. an event ends its life in the callback, by introducing a worker-thread that sleeps until callbacks are added to its work-queue.

Could you please share more details?

e.g. an event ends its life in the callback

You mean you can call cuEventDestroy in the callback?

Are there other use cases than "an event ends its life in the callback"?
If it is the only use case and you don't foresee others, probably this solution is too have weight.
For example, we could clean up those events during some synchronous pi call(for example: cuda_piEventSetCallback and cuda_piContextRelease).

@steffenlarsen
Copy link
Contributor

You mean you can call cuEventDestroy in the callback?

Are there other use cases than "an event ends its life in the callback"?
If it is the only use case and you don't foresee others, probably this solution is too have weight.
For example, we could clean up those events during some synchronous pi call(for example: cuda_piEventSetCallback and cuda_piContextRelease).

Since CUDA events are explicitly destroyed, we must retain the event when we enqueue callbacks, so yes we need this for cuEventDestroy of the given event.

Likewise I believe for one of the current uses of the callbacks the registered callback also destroys the shared_ptr holding the receiving event. It's likely that the "glue" event in that case is kept alive elsewhere as well, but it goes to show how easily a registered callback could be subtly using PI operations, which is very likely to be prohibited in the CUDA BE but not necessarily in the OpenCL BE.

@Ruyk Ruyk changed the title [WIP][DONOTMERGE][SYCL] Fixes for multiple backends in the same program [SYCL][CUDA] Fixes for multiple backends in the same program Mar 12, 2020
fwyzard added a commit to fwyzard/llvm that referenced this pull request Mar 14, 2020
@bader
Copy link
Contributor

bader commented Mar 16, 2020

@Ruyk, @steffenlarsen, could you resolve merge conflicts, please?

Copy link
Contributor

@romanovvlad romanovvlad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Two last comments.

@steffenlarsen steffenlarsen force-pushed the ruyman/event-callback branch from 458754f to 146b163 Compare March 18, 2020 21:25
romanovvlad
romanovvlad previously approved these changes Mar 19, 2020
@bader
Copy link
Contributor

bader commented Mar 19, 2020

@steffenlarsen, @Ruyk, could you take a look at failed checks, please?

@steffenlarsen
Copy link
Contributor

The unittests are currently selecting the first plugin, which seems to be the OpenCL plugin. Is this failing on a machine with no available OpenCL devices?

This should definitely be fixed, but we need a way of selecting the plugins correctly.

@fwyzard
Copy link
Contributor

fwyzard commented Mar 20, 2020

Isn't that what the SYCL_BE variable is supposed to do, as long as the tests use the default device selector ?

@steffenlarsen
Copy link
Contributor

Isn't that what the SYCL_BE variable is supposed to do, as long as the tests use the default device selector ?

The unittests tests the PI API and therefore doesn't use the SYCL device selector. The initial way adjusting the tests to the plugin system was to select the first plugin in the collection returned by initialize, but that just only ever test the first plugin. We should have some common way of selecting a plugin based on the backend we want, but currently there is no good way of doing so.

@steffenlarsen
Copy link
Contributor

The EventTest unittest has been disabled.

I notice that most unittests are currently disabled, so I opened a new issue (#1362) to investigate this.

bader
bader previously approved these changes Mar 20, 2020
@steffenlarsen
Copy link
Contributor

steffenlarsen commented Mar 23, 2020

It seems to be failing with MSVC. From what I gather there is a problem with printing the parameters of piEventSetCallback through the PI tracer implementation, but I do not have a windows environment to test it on so I don't know what I can do about it.

@bader
Copy link
Contributor

bader commented Mar 23, 2020

Could you try to ifdef the check at line 150 i.e. sycl\unittests\pi\EventTest.cpp(150)?

@romanovvlad
Copy link
Contributor

romanovvlad commented Mar 23, 2020

It seems to be failing with MSVC. From what I gather there is a problem with printing the parameters of piEventSetCallback through the PI tracer implementation, but I do not have a windows environment to test it on so I don't know what I can do about it.

Or it has problems with printing lambda object.
Is it legal to pass lambda object to function which expects function pointer?

@steffenlarsen
Copy link
Contributor

Or it has problems with printing lambda object.
Is it legal to pass lambda object to function which expects function pointer?

That's an interesting point. I've tried to move the callback to a separate function. I can't test it, so let's see what the bot says.

nyalloc and others added 2 commits March 23, 2020 17:22
Signed-off-by: Stuart Adams <stuart.adams@codeplay.com>
Signed-off-by: Steffen Larsen <steffen.larsen@codeplay.com>
Signed-off-by: Ruyman Reyes <ruyman.reyes@codeplay.com>
The SYCL RT code for GlueEvent calls now
the right plugin to create the event that triggers the
dependency chain.
Renamed variables to clarify the source code and avoid
confusions between Context and Plugin

Signed-off-by: Ruyman Reyes <ruyman@codeplay.com>
@steffenlarsen steffenlarsen force-pushed the ruyman/event-callback branch from 585902a to cd9b90e Compare March 23, 2020 17:22
@bader bader merged commit 1c4137c into intel:sycl Mar 25, 2020
alexbatashev added a commit to alexbatashev/llvm that referenced this pull request Mar 27, 2020
…hinx

* upstream/sycl: (357 commits)
  [Support] Implement a simple tabular data management library (intel#1358)
  [Support] Implement a property set I/O library (intel#1357)
  [SYCL] Fix buffer constructor using iterators (intel#1386)
  [SYCL][FPGA] Enable a set of loop attributes (intel#1312)
  [Driver][SYCL][FPGA] Proper dependency output location when given /Fo<dir> (intel#1346)
  [SPIR-V] Enabling SPIR-V builtin lookup in device SYCL mode (intel#1384)
  [SYCL][NFC] Unify setting kernel arguments (intel#1379)
  [SYCL][Doc] First revision of standard layout relaxation extension (intel#1344)
  [SYCL] Fixed sub-buffer alloca search (intel#1385)
  [SYCL][FPGA] Emit multiple IR variants for the IVDep attribute (intel#1383)
  [SYCL] Add experimental flag to enable front-end optimizations (intel#1376)
  [SYCL] Remove unexpected double in complex SPIR-V for float support (intel#1381)
  [SYCL] Default work-group sizes based on max (intel#952)
  [SYCL][CUDA] Fix usage of multiple backends in the same program (intel#1252)
  [SPIR-V] Add SPIR-V builtin definitions to the builtin lookup.
  [SPIR-V] Add macro definition when -fdeclare-spirv-builtins is activated
  [SYCL] Fix sycl_generic printing
  [SYCL] Support intel::reqd_work_group_size (intel#1328)
  [SYCL][NFC] Make the RT::PiPlugin object private (intel#1375)
  [SPIRV] Add convergent attribute to SPIR-V built-ins (intel#1373)
  ...
@jpola
Copy link

jpola commented Aug 26, 2020

Dear all,
I tried to check the test case where the one program can work on multiple backends but it doesn't seem to work for me.

  • Currently I'm using: 2f64227 commit.
  • Tot test the feature I used: lvm/sycl/test/plugins/sycl-targets-order.cpp source.
  • I compiled the test using: clang++ -fsycl -fsycl-targets=spir64-unknown-unknown-sycldevice,nvptx64-nvidia-cuda-sycldevice ./sycl-targets-order.cpp -o tgt-spir64-nvptx64.out -v

when I executed the program i have following output:

./tgt-spir64-nvptx64.out 
Running on SYCL device Quadro RTX 5000, driver version CUDA 11.0
terminate called after throwing an instance of 'cl::sycl::runtime_error'
  what():  OpenCL API failed. OpenCL API returns: -42 (CL_INVALID_BINARY) -42 (CL_INVALID_BINARY)
Aborted (core dumped)
SYCL_BE=PI_OPENCL ./tgt-spir64-nvptx64.out 
Running on SYCL device Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, driver version 2020.10.7.0.15
The results are correct!

The clinfo detects following platforms on my machine.

Platform #0: Intel(R) OpenCL
`-- Device #0: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
Platform #1: NVIDIA CUDA
`-- Device #0: Quadro RTX 5000

Could you plase advice on that? How can I enable both backends?

Thanks.

Here is the compilation log:

clang version 12.0.0 (https://github.com/intel/llvm 2f64227a0405e6d4eacdf0974fb8e64462d01a8d)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/jpola/sycl_workspace/llvm/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda-10.2, version 10.2
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple spir64-unknown-unknown-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -mllvm -sycl-opt -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -emit-llvm-bc -emit-llvm-uselists -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -fno-verbose-asm -mconstructor-aliases -aux-target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -faddrsig -o /tmp/sycl-targets-order-db613b.bc -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/home/jpola/sycl_workspace/llvm/build/bin/llvm-link" /tmp/sycl-targets-order-db613b.bc -o /tmp/sycl-targets-order-285b65.bc --suppress-warnings
 "/home/jpola/sycl_workspace/llvm/build/bin/sycl-post-link" -symbols -spec-const=rt -o /tmp/sycl-targets-order-f01eea.table /tmp/sycl-targets-order-285b65.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/file-table-tform" -extract=Code -drop_titles -o /tmp/sycl-targets-order-6447e2.txt /tmp/sycl-targets-order-f01eea.table
 "/home/jpola/sycl_workspace/llvm/build/bin/llvm-foreach" --in-file-list=/tmp/sycl-targets-order-6447e2.txt --in-replace=/tmp/sycl-targets-order-6447e2.txt --out-ext=spv --out-file-list=/tmp/sycl-targets-order-111dc8.txt --out-replace=/tmp/sycl-targets-order-111dc8.txt -- /home/jpola/sycl_workspace/llvm/build/bin/llvm-spirv -o /tmp/sycl-targets-order-111dc8.txt -spirv-max-version=1.1 -spirv-ext=+all,-SPV_INTEL_usm_storage_classes /tmp/sycl-targets-order-6447e2.txt
 "/home/jpola/sycl_workspace/llvm/build/bin/file-table-tform" -replace=Code,Code -o /tmp/sycl-targets-order-f935a6.table /tmp/sycl-targets-order-f01eea.table /tmp/sycl-targets-order-111dc8.txt
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-offload-wrapper" -o=/tmp/wrapper-a65b53.bc -host=x86_64-unknown-linux-gnu -target=spir64 -kind=sycl -batch /tmp/sycl-targets-order-f935a6.table
 "/home/jpola/sycl_workspace/llvm/build/bin/llc" -filetype=obj -o /tmp/tgt-spir64-nvptx64-0ce575.o /tmp/wrapper-a65b53.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple nvptx64-nvidia-cuda-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -emit-llvm-bc -emit-llvm-uselists -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fno-rounding-math -fno-verbose-asm -no-integrated-as -aux-target-cpu x86-64 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -mlink-builtin-bitcode /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/../../clc/libspirv-nvptx64--nvidiacl.bc -mlink-builtin-bitcode /usr/local/cuda-10.2/nvvm/libdevice/libdevice.10.bc -target-feature +ptx65 -target-sdk-version=10.2 -target-cpu sm_50 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fno-dwarf-directory-asm -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -o /tmp/sycl-targets-order-5b7a26.bc -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/bin/../include/sycl"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/home/jpola/sycl_workspace/llvm/build/bin/llvm-link" /tmp/sycl-targets-order-5b7a26.bc -o /tmp/sycl-targets-order-c37d03.bc --suppress-warnings
 "/home/jpola/sycl_workspace/llvm/build/bin/sycl-post-link" -ir-output-only -spec-const=default -o /tmp/sycl-targets-order-5fd947.bc /tmp/sycl-targets-order-c37d03.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple nvptx64-nvidia-cuda-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -S -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fno-rounding-math -fno-verbose-asm -no-integrated-as -aux-target-cpu x86-64 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -mlink-builtin-bitcode /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/../../clc/libspirv-nvptx64--nvidiacl.bc -mlink-builtin-bitcode /usr/local/cuda-10.2/nvvm/libdevice/libdevice.10.bc -target-feature +ptx65 -target-sdk-version=10.2 -target-cpu sm_50 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -fno-dwarf-directory-asm -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcolor-diagnostics -o /tmp/sycl-targets-order-f44eab.s -x ir /tmp/sycl-targets-order-5fd947.bc
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
 "/usr/local/cuda-10.2/bin/ptxas" -m64 -O0 -v --gpu-name sm_50 --output-file /tmp/sycl-targets-order-342807.o /tmp/sycl-targets-order-f44eab.s
ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function '_ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer_with_offset' for 'sm_50'
ptxas info    : Function properties for _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer_with_offset
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 10 registers, 364 bytes cmem[0]
ptxas info    : Compiling entry function '_ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer' for 'sm_50'
ptxas info    : Function properties for _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 7 registers, 352 bytes cmem[0]
 "/usr/local/cuda-10.2/bin/fatbinary" -64 --create /tmp/sycl-targets-order-c523f4.fatbin --image=profile=compute_50,file=/tmp/sycl-targets-order-f44eab.s --image=profile=sm_50,file=/tmp/sycl-targets-order-342807.o
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-offload-wrapper" -o=/tmp/wrapper-4e717c.bc -host=x86_64-unknown-linux-gnu -target=nvptx64 -kind=sycl /tmp/sycl-targets-order-c523f4.fatbin
 "/home/jpola/sycl_workspace/llvm/build/bin/llc" -filetype=obj -o /tmp/tgt-spir64-nvptx64-9b7188.o /tmp/wrapper-4e717c.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple spir64-unknown-unknown-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -mllvm -sycl-opt -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -fsyntax-only -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -fno-verbose-asm -mconstructor-aliases -aux-target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -fsycl-int-header=/tmp/sycl-targets-order-960d4a.h -faddrsig -o /tmp/sycl-targets-order-960d4a.h -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple x86_64-unknown-linux-gnu -sycl-std=2017 -emit-obj -mrelax-all --mrelax-relocations -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -mconstructor-aliases -munwind-tables -target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -include /tmp/sycl-targets-order-960d4a.h -dependency-filter /tmp/sycl-targets-order-960d4a.h -fsycl -fsycl-is-host -faddrsig -o /tmp/sycl-targets-order-87fa0f.o -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/bin/../include/sycl"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o tgt-spir64-nvptx64.out /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/9/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/x86_64-linux-gnu/../../lib64 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../.. -L/home/jpola/sycl_workspace/llvm/build/bin/../lib -L/lib -L/usr/lib -L/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/lib -L/home/jpola/intel/oneapi/ippcp/latest/lib/intel64 -L/home/jpola/intel/oneapi/ccl/2021.1-beta08/lib/cpu_gpu_dpcpp -L/home/jpola/intel/oneapi/compiler/latest/linux/lib -L/home/jpola/intel/oneapi/mkl/2021.1-beta08/lib/intel64 -L/home/jpola/intel/oneapi/vpl/latest/lib -L/home/jpola/intel/oneapi/daal/latest/lib/intel64 -L/home/jpola/intel/oneapi/ipp/latest/lib/intel64 -L/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../lib/intel64/gcc4.8 -L/home/jpola/intel/oneapi/mpi/2021.1-beta08//libfabric/lib -L/home/jpola/intel/oneapi/mpi/2021.1-beta08//lib/release -L/home/jpola/intel/oneapi/mpi/2021.1-beta08//lib /tmp/sycl-targets-order-87fa0f.o /tmp/tgt-spir64-nvptx64-0ce575.o /tmp/tgt-spir64-nvptx64-9b7188.o -lstdc++ -lm -lgcc_s -lgcc -lsycl -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/9/crtend.o /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crtn.o

@romanovvlad
Copy link
Contributor

Dear all,
I tried to check the test case where the one program can work on multiple backends but it doesn't seem to work for me.

* Currently I'm using: [2f64227](https://github.com/intel/llvm/commit/2f64227a0405e6d4eacdf0974fb8e64462d01a8d) commit.

* Tot test the feature I used: `lvm/sycl/test/plugins/sycl-targets-order.cpp` source.

* I compiled the test using: `clang++ -fsycl -fsycl-targets=spir64-unknown-unknown-sycldevice,nvptx64-nvidia-cuda-sycldevice ./sycl-targets-order.cpp -o tgt-spir64-nvptx64.out -v`

when I executed the program i have following output:

./tgt-spir64-nvptx64.out 
Running on SYCL device Quadro RTX 5000, driver version CUDA 11.0
terminate called after throwing an instance of 'cl::sycl::runtime_error'
  what():  OpenCL API failed. OpenCL API returns: -42 (CL_INVALID_BINARY) -42 (CL_INVALID_BINARY)
Aborted (core dumped)
SYCL_BE=PI_OPENCL ./tgt-spir64-nvptx64.out 
Running on SYCL device Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz, driver version 2020.10.7.0.15
The results are correct!

The clinfo detects following platforms on my machine.

Platform #0: Intel(R) OpenCL
`-- Device #0: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
Platform #1: NVIDIA CUDA
`-- Device #0: Quadro RTX 5000

Could you plase advice on that? How can I enable both backends?

Thanks.

Here is the compilation log:

clang version 12.0.0 (https://github.com/intel/llvm 2f64227a0405e6d4eacdf0974fb8e64462d01a8d)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/jpola/sycl_workspace/llvm/build/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/7.5.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/8
Found candidate GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Selected GCC installation: /usr/lib/gcc/x86_64-linux-gnu/9
Candidate multilib: .;@m64
Selected multilib: .;@m64
Found CUDA installation: /usr/local/cuda-10.2, version 10.2
clang-12: warning: Unknown CUDA version 10.2. Assuming the latest supported version 10.1 [-Wunknown-cuda-version]
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple spir64-unknown-unknown-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -mllvm -sycl-opt -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -emit-llvm-bc -emit-llvm-uselists -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -fno-verbose-asm -mconstructor-aliases -aux-target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -faddrsig -o /tmp/sycl-targets-order-db613b.bc -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/home/jpola/sycl_workspace/llvm/build/bin/llvm-link" /tmp/sycl-targets-order-db613b.bc -o /tmp/sycl-targets-order-285b65.bc --suppress-warnings
 "/home/jpola/sycl_workspace/llvm/build/bin/sycl-post-link" -symbols -spec-const=rt -o /tmp/sycl-targets-order-f01eea.table /tmp/sycl-targets-order-285b65.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/file-table-tform" -extract=Code -drop_titles -o /tmp/sycl-targets-order-6447e2.txt /tmp/sycl-targets-order-f01eea.table
 "/home/jpola/sycl_workspace/llvm/build/bin/llvm-foreach" --in-file-list=/tmp/sycl-targets-order-6447e2.txt --in-replace=/tmp/sycl-targets-order-6447e2.txt --out-ext=spv --out-file-list=/tmp/sycl-targets-order-111dc8.txt --out-replace=/tmp/sycl-targets-order-111dc8.txt -- /home/jpola/sycl_workspace/llvm/build/bin/llvm-spirv -o /tmp/sycl-targets-order-111dc8.txt -spirv-max-version=1.1 -spirv-ext=+all,-SPV_INTEL_usm_storage_classes /tmp/sycl-targets-order-6447e2.txt
 "/home/jpola/sycl_workspace/llvm/build/bin/file-table-tform" -replace=Code,Code -o /tmp/sycl-targets-order-f935a6.table /tmp/sycl-targets-order-f01eea.table /tmp/sycl-targets-order-111dc8.txt
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-offload-wrapper" -o=/tmp/wrapper-a65b53.bc -host=x86_64-unknown-linux-gnu -target=spir64 -kind=sycl -batch /tmp/sycl-targets-order-f935a6.table
 "/home/jpola/sycl_workspace/llvm/build/bin/llc" -filetype=obj -o /tmp/tgt-spir64-nvptx64-0ce575.o /tmp/wrapper-a65b53.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple nvptx64-nvidia-cuda-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -emit-llvm-bc -emit-llvm-uselists -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fno-rounding-math -fno-verbose-asm -no-integrated-as -aux-target-cpu x86-64 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -mlink-builtin-bitcode /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/../../clc/libspirv-nvptx64--nvidiacl.bc -mlink-builtin-bitcode /usr/local/cuda-10.2/nvvm/libdevice/libdevice.10.bc -target-feature +ptx65 -target-sdk-version=10.2 -target-cpu sm_50 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fno-dwarf-directory-asm -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -o /tmp/sycl-targets-order-5b7a26.bc -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/bin/../include/sycl"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/home/jpola/sycl_workspace/llvm/build/bin/llvm-link" /tmp/sycl-targets-order-5b7a26.bc -o /tmp/sycl-targets-order-c37d03.bc --suppress-warnings
 "/home/jpola/sycl_workspace/llvm/build/bin/sycl-post-link" -ir-output-only -spec-const=default -o /tmp/sycl-targets-order-5fd947.bc /tmp/sycl-targets-order-c37d03.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple nvptx64-nvidia-cuda-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -S -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fno-rounding-math -fno-verbose-asm -no-integrated-as -aux-target-cpu x86-64 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -mlink-builtin-bitcode /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/../../clc/libspirv-nvptx64--nvidiacl.bc -mlink-builtin-bitcode /usr/local/cuda-10.2/nvvm/libdevice/libdevice.10.bc -target-feature +ptx65 -target-sdk-version=10.2 -target-cpu sm_50 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -fno-dwarf-directory-asm -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcolor-diagnostics -o /tmp/sycl-targets-order-f44eab.s -x ir /tmp/sycl-targets-order-5fd947.bc
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
 "/usr/local/cuda-10.2/bin/ptxas" -m64 -O0 -v --gpu-name sm_50 --output-file /tmp/sycl-targets-order-342807.o /tmp/sycl-targets-order-f44eab.s
ptxas info    : 0 bytes gmem
ptxas info    : Compiling entry function '_ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer_with_offset' for 'sm_50'
ptxas info    : Function properties for _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer_with_offset
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 10 registers, 364 bytes cmem[0]
ptxas info    : Compiling entry function '_ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer' for 'sm_50'
ptxas info    : Function properties for _ZTSZZ4mainENK3$_0clERN2cl4sycl7handlerEE10FillBuffer
    0 bytes stack frame, 0 bytes spill stores, 0 bytes spill loads
ptxas info    : Used 7 registers, 352 bytes cmem[0]
 "/usr/local/cuda-10.2/bin/fatbinary" -64 --create /tmp/sycl-targets-order-c523f4.fatbin --image=profile=compute_50,file=/tmp/sycl-targets-order-f44eab.s --image=profile=sm_50,file=/tmp/sycl-targets-order-342807.o
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-offload-wrapper" -o=/tmp/wrapper-4e717c.bc -host=x86_64-unknown-linux-gnu -target=nvptx64 -kind=sycl /tmp/sycl-targets-order-c523f4.fatbin
 "/home/jpola/sycl_workspace/llvm/build/bin/llc" -filetype=obj -o /tmp/tgt-spir64-nvptx64-9b7188.o /tmp/wrapper-4e717c.bc
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple spir64-unknown-unknown-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -mllvm -sycl-opt -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2017 -fsyntax-only -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -fno-verbose-asm -mconstructor-aliases -aux-target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -fsycl-int-header=/tmp/sycl-targets-order-960d4a.h -faddrsig -o /tmp/sycl-targets-order-960d4a.h -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/home/jpola/sycl_workspace/llvm/build/bin/clang-12" -cc1 -triple x86_64-unknown-linux-gnu -sycl-std=2017 -emit-obj -mrelax-all --mrelax-relocations -disable-free -main-file-name sycl-targets-order.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -mconstructor-aliases -munwind-tables -target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0 -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -I/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include -I/home/jpola/intel/oneapi/ippcp/latest/include -I/home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp -I/home/jpola/intel/oneapi/compiler/latest/linux/include -I/home/jpola/intel/oneapi/mkl/2021.1-beta08/include -I/home/jpola/intel/oneapi/vpl/latest/include -I/home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include -I/home/jpola/intel/oneapi/daal/latest/include -I/home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include -I/home/jpola/intel/oneapi/ipp/latest/include -I/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include -I/home/jpola/intel/oneapi/mpi/2021.1-beta08//include -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9 -internal-isystem /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl -internal-isystem /usr/local/include -internal-isystem /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include -internal-externc-isystem /usr/include/x86_64-linux-gnu -internal-externc-isystem /include -internal-externc-isystem /usr/include -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/jpola/sycl_workspace/llvm/sycl/test/plugins -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -include /tmp/sycl-targets-order-960d4a.h -dependency-filter /tmp/sycl-targets-order-960d4a.h -fsycl -fsycl-is-host -faddrsig -o /tmp/sycl-targets-order-87fa0f.o -x c++ ./sycl-targets-order.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9"
ignoring duplicate directory "/usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/bin/../include/sycl"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include/x86_64-linux-gnu"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/include
 /home/jpola/intel/oneapi/ippcp/latest/include
 /home/jpola/intel/oneapi/ccl/2021.1-beta08/include/cpu_gpu_dpcpp
 /home/jpola/intel/oneapi/compiler/latest/linux/include
 /home/jpola/intel/oneapi/mkl/2021.1-beta08/include
 /home/jpola/intel/oneapi/vpl/latest/include
 /home/jpola/intel/oneapi/dpcpp-ct/2021.1-beta08/include
 /home/jpola/intel/oneapi/daal/latest/include
 /home/jpola/intel/oneapi/dev-utilities/2021.1-beta08/include
 /home/jpola/intel/oneapi/ipp/latest/include
 /home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../include
 /home/jpola/intel/oneapi/mpi/2021.1-beta08//include
 /home/jpola/sycl_workspace/llvm/build/bin/../include/sycl
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/x86_64-linux-gnu/c++/9
 /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/backward
 /usr/local/include
 /home/jpola/sycl_workspace/llvm/build/lib/clang/12.0.0/include
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
 "/usr/bin/ld" -z relro --hash-style=gnu --eh-frame-hdr -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 -o tgt-spir64-nvptx64.out /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crt1.o /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/9/crtbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib64 -L/lib/x86_64-linux-gnu -L/lib/../lib64 -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib64 -L/usr/lib/x86_64-linux-gnu/../../lib64 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../.. -L/home/jpola/sycl_workspace/llvm/build/bin/../lib -L/lib -L/usr/lib -L/home/jpola/intel/oneapi/dnnl/2021.1-beta08/cpu_dpcpp_gpu_dpcpp/lib -L/home/jpola/intel/oneapi/ippcp/latest/lib/intel64 -L/home/jpola/intel/oneapi/ccl/2021.1-beta08/lib/cpu_gpu_dpcpp -L/home/jpola/intel/oneapi/compiler/latest/linux/lib -L/home/jpola/intel/oneapi/mkl/2021.1-beta08/lib/intel64 -L/home/jpola/intel/oneapi/vpl/latest/lib -L/home/jpola/intel/oneapi/daal/latest/lib/intel64 -L/home/jpola/intel/oneapi/ipp/latest/lib/intel64 -L/home/jpola/intel/oneapi/tbb/2021.1-beta08/env/../lib/intel64/gcc4.8 -L/home/jpola/intel/oneapi/mpi/2021.1-beta08//libfabric/lib -L/home/jpola/intel/oneapi/mpi/2021.1-beta08//lib/release -L/home/jpola/intel/oneapi/mpi/2021.1-beta08//lib /tmp/sycl-targets-order-87fa0f.o /tmp/tgt-spir64-nvptx64-0ce575.o /tmp/tgt-spir64-nvptx64-9b7188.o -lstdc++ -lm -lgcc_s -lgcc -lsycl -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/9/crtend.o /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crtn.o

Could you please open an issue? https://github.com/intel/llvm/issues

@jpola
Copy link

jpola commented Aug 26, 2020

Thanks, I created: #2372

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cuda CUDA back-end
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Execution fails when using both plugins
7 participants