[SYCL] Improve diagnostics around passing compiler options to GPU backend

[j-stephan]

Reproducer: Use this oneAPI sample.

I am trying to build the above file for a CPU and a GPU at the same time. However, I am unable to specify the target GPU because clang does not know the option:

clang++ -O3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=spir64_x86_64-unknown-unknown-sycldevice,spir64_gen-unknown-unknown-sycldevice -Xsycl-target-backend=spir64_x86_64-unknown-unknown-sycldevice "-march=avx512" -Xsycl-target-backend=spir64_gen-unknown-unknown-sycl-device "-device icllp" -o vector-add-buffers vector-add-buffers.cpp
clang-12: warning: argument unused during compilation: '-Xsycl-target-backend=spir64_gen-unknown-unknown-sycl-device -device icllp' [-Wunused-command-line-argument]
Platform name: Intel(R) OpenCL
Device name: Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz
Driver version: 2020.11.11.0.04_160000
Setting target CPU architecture to Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512)
OpenCL program was successfully created from SPIR-V file /tmp/vector-add-buffers-f3cbce.spv
Using build options:  -I "/tmp"
Compilation started
Compilation done
Linking started
Linking done
Device build started
Options used by backend compiler: 
Device build done
Kernel <_ZTSZZ9VectorAddRN2cl4sycl5queueERKSt5arrayIiLm10000EES6_RS4_ENKUlRNS0_7handlerEE67_12clES9_EUlT_E82_31> was successfully vectorized (16)
Done.
OpenCL program binary file was successfully created: /tmp/vector-add-buffers-e467ff.out
Error: Device name missing.
Command was: /usr/bin/ocloc -output /tmp/vector-add-buffers-1fed00.out -file /tmp/vector-add-buffers-1eea39.spv -output_no_suffix -spirv_input
clang-12: error: gen compiler command failed with exit code 226 (use -v to see invocation)
clang version 12.0.0 (https://github.com/intel/llvm e8f0b98e7f421d986d779316f7b826d8ab9f6ea1)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/stepha27/sycl-workspace/llvm/build/install/bin
clang-12: note: diagnostic msg: Error generating preprocessed source(s).

Verbose output:

clang++ -O3 -fsycl -fsycl-unnamed-lambda -fsycl-targets=spir64_x86_64-unknown-unknown-sycldevice,spir64_gen-unknown-unknown-sycldevice -Xsycl-target-backend=spir64_x86_64-unknown-unknown-sycldevice "-march=avx512" -Xsycl-target-backend=spir64_gen-unknown-unknown-sycl-device "-device icllp" -o vector-add-buffers vector-add-buffers.cpp -v
clang version 12.0.0 (https://github.com/intel/llvm e8f0b98e7f421d986d779316f7b826d8ab9f6ea1)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/stepha27/sycl-workspace/llvm/build/install/bin
Found candidate GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/10.2.0
Found candidate GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0
Selected GCC installation: /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
clang-12: warning: argument unused during compilation: '-Xsycl-target-backend=spir64_gen-unknown-unknown-sycl-device -device icllp' [-Wunused-command-line-argument]
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-12" -cc1 -triple spir64_x86_64-unknown-unknown-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -mllvm -sycl-opt -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2020 -emit-llvm-bc -emit-llvm-uselists -disable-free -main-file-name vector-add-buffers.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -fno-verbose-asm -mconstructor-aliases -aux-target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0 -internal-isystem /home/stepha27/sycl-workspace/llvm/build/install/bin/../include/sycl -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0 -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/backward -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0 -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/backward -internal-isystem /usr/local/include -internal-isystem /home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O3 -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/stepha27/sycl-workspace/issues/llvm/oneAPI-samples/DirectProgramming/DPC++/DenseLinearAlgebra/vector-add/src -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -fsycl-unnamed-lambda -faddrsig -o /tmp/vector-add-buffers-4a1a3b.bc -x c++ vector-add-buffers.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0"
ignoring duplicate directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu"
ignoring duplicate directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/stepha27/sycl-workspace/llvm/build/install/bin/../include/sycl
 /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0
 /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu
 /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/backward
 /usr/local/include
 /home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0/include
 /usr/include
End of search list.
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-crt.o -outputs=/tmp/libsycl-crt-f51d9e.o,/tmp/libsycl-crt-0bca90.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-complex.o -outputs=/tmp/libsycl-complex-d36937.o,/tmp/libsycl-complex-73d6a4.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-cmath.o -outputs=/tmp/libsycl-cmath-5ee791.o,/tmp/libsycl-cmath-28635f.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-fallback-cassert.o -outputs=/tmp/libsycl-fallback-cassert-15aef3.o,/tmp/libsycl-fallback-cassert-ed746a.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-fallback-complex.o -outputs=/tmp/libsycl-fallback-complex-ced947.o,/tmp/libsycl-fallback-complex-d96b0f.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-fallback-cmath.o -outputs=/tmp/libsycl-fallback-cmath-d2735d.o,/tmp/libsycl-fallback-cmath-372d52.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/llvm-link" /tmp/vector-add-buffers-4a1a3b.bc /tmp/libsycl-crt-f51d9e.o /tmp/libsycl-complex-d36937.o /tmp/libsycl-cmath-5ee791.o /tmp/libsycl-fallback-cassert-15aef3.o /tmp/libsycl-fallback-complex-ced947.o /tmp/libsycl-fallback-cmath-d2735d.o -o /tmp/vector-add-buffers-13f875.bc --suppress-warnings
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/sycl-post-link" -ir-output-only -spec-const=default -o /tmp/vector-add-buffers-2330c7.bc /tmp/vector-add-buffers-13f875.bc
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/llvm-spirv" -o /tmp/vector-add-buffers-791d51.spv -spirv-max-version=1.1 -spirv-debug-info-version=legacy -spirv-allow-extra-diexpressions -spirv-ext=+all,-SPV_INTEL_usm_storage_classes /tmp/vector-add-buffers-2330c7.bc
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/opencl-aot" -o=/tmp/vector-add-buffers-c24459.out --device=cpu /tmp/vector-add-buffers-791d51.spv -march=avx512
Platform name: Intel(R) OpenCL
Device name: Intel(R) Core(TM) i7-1065G7 CPU @ 1.30GHz
Driver version: 2020.11.11.0.04_160000
Setting target CPU architecture to Intel(R) Advanced Vector Extensions 512 (Intel(R) AVX-512)
OpenCL program was successfully created from SPIR-V file /tmp/vector-add-buffers-791d51.spv
Using build options:  -I "/tmp"
Compilation started
Compilation done
Linking started
Linking done
Device build started
Options used by backend compiler: 
Device build done
Kernel <_ZTSZZ9VectorAddRN2cl4sycl5queueERKSt5arrayIiLm10000EES6_RS4_ENKUlRNS0_7handlerEE67_12clES9_EUlT_E82_31> was successfully vectorized (16)
Done.
OpenCL program binary file was successfully created: /tmp/vector-add-buffers-c24459.out
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-wrapper" -o=/tmp/wrapper-5784f0.bc -host=x86_64-unknown-linux-gnu -target=spir64_x86_64 -kind=sycl /tmp/vector-add-buffers-c24459.out
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/llc" -filetype=obj -o /tmp/vector-add-buffers-180365.o /tmp/wrapper-5784f0.bc
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-12" -cc1 -triple spir64_gen-unknown-unknown-sycldevice -fsycl -fsycl-is-device -fdeclare-spirv-builtins -mllvm -sycl-opt -aux-triple x86_64-unknown-linux-gnu -Wno-sycl-strict -sycl-std=2020 -emit-llvm-bc -emit-llvm-uselists -disable-free -main-file-name vector-add-buffers.cpp -mrelocation-model static -mframe-pointer=all -fmath-errno -fno-rounding-math -fno-verbose-asm -mconstructor-aliases -aux-target-cpu x86-64 -fno-split-dwarf-inlining -debugger-tuning=gdb -v -resource-dir /home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0 -internal-isystem /home/stepha27/sycl-workspace/llvm/build/install/bin/../include/sycl -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0 -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/backward -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0 -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu -internal-isystem /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/backward -internal-isystem /usr/local/include -internal-isystem /home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -internal-isystem /usr/local/include -internal-isystem /home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -O3 -std=c++17 -fdeprecated-macro -fdebug-compilation-dir /home/stepha27/sycl-workspace/issues/llvm/oneAPI-samples/DirectProgramming/DPC++/DenseLinearAlgebra/vector-add/src -ferror-limit 19 -fgnuc-version=4.2.1 -fcxx-exceptions -fexceptions -fcolor-diagnostics -fsycl-unnamed-lambda -faddrsig -o /tmp/vector-add-buffers-a13d10.bc -x c++ vector-add-buffers.cpp
clang -cc1 version 12.0.0 based upon LLVM 12.0.0git default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
ignoring nonexistent directory "/include"
ignoring duplicate directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0"
ignoring duplicate directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu"
ignoring duplicate directory "/usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0/include"
ignoring duplicate directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /home/stepha27/sycl-workspace/llvm/build/install/bin/../include/sycl
 /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0
 /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/x86_64-pc-linux-gnu
 /usr/lib64/gcc/x86_64-pc-linux-gnu/10.2.0/../../../../include/c++/10.2.0/backward
 /usr/local/include
 /home/stepha27/sycl-workspace/llvm/build/install/lib/clang/12.0.0/include
 /usr/include
End of search list.
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-crt.o -outputs=/tmp/libsycl-crt-59966e.o,/tmp/libsycl-crt-730b76.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-complex.o -outputs=/tmp/libsycl-complex-58e5b4.o,/tmp/libsycl-complex-b1b35a.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-cmath.o -outputs=/tmp/libsycl-cmath-3fc7dd.o,/tmp/libsycl-cmath-72608d.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-fallback-cassert.o -outputs=/tmp/libsycl-fallback-cassert-ff0fb7.o,/tmp/libsycl-fallback-cassert-50f45a.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-fallback-complex.o -outputs=/tmp/libsycl-fallback-complex-80b441.o,/tmp/libsycl-fallback-complex-e70aed.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/clang-offload-bundler" -type=o -targets=sycl-spir64_x86_64-unknown-unknown-sycldevice,sycl-spir64_gen-unknown-unknown-sycldevice -inputs=/home/stepha27/sycl-workspace/llvm/build/install/bin/../lib/libsycl-fallback-cmath.o -outputs=/tmp/libsycl-fallback-cmath-85fe67.o,/tmp/libsycl-fallback-cmath-b57b42.o -unbundle
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/llvm-link" /tmp/vector-add-buffers-a13d10.bc /tmp/libsycl-crt-730b76.o /tmp/libsycl-complex-b1b35a.o /tmp/libsycl-cmath-72608d.o /tmp/libsycl-fallback-cassert-50f45a.o /tmp/libsycl-fallback-complex-e70aed.o /tmp/libsycl-fallback-cmath-b57b42.o -o /tmp/vector-add-buffers-2a21e7.bc --suppress-warnings
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/sycl-post-link" -ir-output-only -spec-const=default -o /tmp/vector-add-buffers-c687ac.bc /tmp/vector-add-buffers-2a21e7.bc
 "/home/stepha27/sycl-workspace/llvm/build/install/bin/llvm-spirv" -o /tmp/vector-add-buffers-893932.spv -spirv-max-version=1.1 -spirv-debug-info-version=legacy -spirv-allow-extra-diexpressions -spirv-ext=+all,-SPV_INTEL_usm_storage_classes /tmp/vector-add-buffers-c687ac.bc
 "/usr/bin/ocloc" -output /tmp/vector-add-buffers-6b76ad.out -file /tmp/vector-add-buffers-893932.spv -output_no_suffix -spirv_input
Error: Device name missing.
Command was: /usr/bin/ocloc -output /tmp/vector-add-buffers-6b76ad.out -file /tmp/vector-add-buffers-893932.spv -output_no_suffix -spirv_input
clang-12: error: gen compiler command failed with exit code 226 (use -v to see invocation)
clang version 12.0.0 (https://github.com/intel/llvm e8f0b98e7f421d986d779316f7b826d8ab9f6ea1)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/stepha27/sycl-workspace/llvm/build/install/bin
clang-12: note: diagnostic msg: Error generating preprocessed source(s).

As far as I can see this only affects the GPU options. Both CPU and FPGA don't seem to be affected.

[AlexeySachkov]

Hi @j-stephan,

This is a guess, but probably it didn't work because of the typo in the triple in -Xsycl-target-backend= option: spir64_gen-unknown-unknown-sycl-device instead of spir64_gen-unknown-unknown-sycldevice. Note dash in the middle of sycldevice.

Tagging @mdtoguchi and @AGindinson here - probably we can improve our driver to report such kind of errors in a more verbose way or get some kind of "did you mean [correct-version-of-option]" suggestion displayed to user? I can totally understand that making a typo in this long triple is quite an easy thing to do, which apparently might results in quite obscure error

[j-stephan]

Oh dear, it was indeed a typo that bothered me way too long. Thanks for the hint!

[bader]

Tagging @mdtoguchi and @AGindinson here - probably we can improve our driver to report such kind of errors in a more verbose way or get some kind of "did you mean [correct-version-of-option]" suggestion displayed to user? I can totally understand that making a typo in this long triple is quite an easy thing to do, which apparently might results in quite obscure error

👍. I would be expect -fsycl to be enough to use sycldevice as default value for environment component of the triple, so compiler users could set just architecture component e.g. -Xsycl-target-backend=spir64_gen.

[j-stephan]

That sounds like a very useful thing to have! Especially when compiling ahead of time for all device types the compiler command becomes very long (and prone to typos / copy&paste errors).

[AlexeySachkov]

I think that we can close this one, because we have #3984, which allows to specify much shorter names for targets, like simply spir64_gen.

Additionally, it seems like we have a diagnostic for invalid targets:

$ clang++  -fsycl -fsycl-targets=unknown test.cpp
clang-16: error: SYCL target is invalid: 'unknown'