Please allow using AdaptiveCpp (acpp) as a replacement for icpx

[zboszor]

hipBLAS and other optional libraries in chipStar use the git submodule H4I-MKLShim which relies on the Intel icpx SYCL compiler.

However, Yocto with its cross-compiler framework, cannot easily use icpx, even with meta-intel present in the build.

Please, allow AdaptiveCpp to be used for compiling the SYCL sources, which may be more accessible.

[pvelesko]

@zboszor what about actual MKL? If that's not present in Yocto then there is no point in enabling adaptivecpp

[zboszor]

MKL is present in meta-intel, and also the Intel Compute Runtime a.k.a. OpenCL. AdaptiveCpp can also use OpenCL, just like chipStar.

[zboszor]

FWIW, I have already created a Yocto recipe for chipStar with only a few minimal patches needed. The samples do run with both Intel OpenCL and Level0.

[zboszor]

I somewhat lied. Most of the samples do run, but a few segfaults, like fp16_half2_math and fp16_math. Is it worth reporting them, or are these known issues? I can run them through gdb if needed.

[pvelesko]

Please dump the traces here along with platform information.
I'll work on adaptivecpp support today

[zboszor]

lscpu

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          39 bits physical, 48 bits virtual
  Byte Order:             Little Endian
CPU(s):                   12
  On-line CPU(s) list:    0-11
Vendor ID:                GenuineIntel
  BIOS Vendor ID:         Intel(R) Corporation
  Model name:             Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
    BIOS Model name:      Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz To Be Filled By O.E.M. CPU @ 3.1GHz
    BIOS CPU family:      198
    CPU family:           6
    Model:                158
    Thread(s) per core:   2
    Core(s) per socket:   6
    Socket(s):            1
    Stepping:             10
    CPU(s) scaling MHz:   18%
    CPU max MHz:          4600.0000
    CPU min MHz:          800.0000
    BogoMIPS:             6399.96
    Flags:                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_per
                          fmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic
                           movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow flexpriority ept vpid ept_ad fsgsbase
                           tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_
                          epp vnmi md_clear flush_l1d arch_capabilities
Virtualization features:  
  Virtualization:         VT-x
Caches (sum of all):      
  L1d:                    192 KiB (6 instances)
  L1i:                    192 KiB (6 instances)
  L2:                     1.5 MiB (6 instances)
  L3:                     12 MiB (1 instance)
NUMA:                     
  NUMA node(s):           1
  NUMA node0 CPU(s):      0-11
Vulnerabilities:          
  Gather data sampling:   Mitigation; Microcode
  Itlb multihit:          KVM: Mitigation: VMX disabled
  L1tf:                   Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
  Mds:                    Mitigation; Clear CPU buffers; SMT vulnerable
  Meltdown:               Mitigation; PTI
  Mmio stale data:        Mitigation; Clear CPU buffers; SMT vulnerable
  Reg file data sampling: Not affected
  Retbleed:               Mitigation; IBRS
  Spec rstack overflow:   Not affected
  Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
  Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:             Mitigation; IBRS; IBPB conditional; STIBP conditional; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
  Srbds:                  Mitigation; Microcode
  Tsx async abort:        Mitigation; TSX disabled

lspci

00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics 630]
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller (rev 10)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #6 (rev f0)
00:1c.6 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #7 (rev f0)
00:1f.0 ISA bridge: Intel Corporation H310 Chipset LPC/eSPI Controller (rev 10)
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (7) I219-LM (rev 10)
01:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
02:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8822BE 802.11a/b/g/n/ac WiFi adapter

Output of fp16_half2_math

[root@localhost chip_spv_samples]# ./fp16_half2_math
CHIP info [TID 2554] [1732873171.179529349] : CHIP_PLATFORM=0
CHIP info [TID 2554] [1732873171.179582884] : CHIP_DEVICE_TYPE=gpu
CHIP info [TID 2554] [1732873171.179589354] : CHIP_DEVICE=0
CHIP info [TID 2554] [1732873171.179613632] : CHIP_BE=opencl
CHIP info [TID 2554] [1732873171.179620129] : CHIP_DUMP_SPIRV=off
CHIP info [TID 2554] [1732873171.179626729] : CHIP_JIT_FLAGS_OVERRIDE=
CHIP info [TID 2554] [1732873171.179633126] : CHIP_L0_COLLECT_EVENTS_TIMEOUT=0
CHIP info [TID 2554] [1732873171.179640503] : CHIP_L0_EVENT_TIMEOUT=0
CHIP info [TID 2554] [1732873171.179647408] : CHIP_SKIP_UNINIT=off
CHIP info [TID 2554] [1732873171.179653663] : CHIP_LAZY_JIT=on
CHIP info [TID 2554] [1732873171.179672833] : CHIP_OCL_DISABLE_QUEUE_PROFILING=off
CHIP info [TID 2554] [1732873171.179679604] : CHIP_OCL_USE_ALLOC_STRATEGY=off
CHIP info [TID 2554] [1732873171.179701060] : CHIP_MODULE_CACHE_DIR=/root/.cache/chipStar
CHIP info [TID 2554] [1732873171.211346954] : OpenCL Devices of type gpu with SPIR-V_1 support:
Intel(R) UHD Graphics 630  is supported.

CHIP info [TID 2554] [1732873171.215136096] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2554] [1732873171.215170432] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2554] [1732873171.215198277] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2554] [1732873171.215227768] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2554] [1732873171.215237756] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2554] [1732873171.215244688] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2554] [1732873171.215251927] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2554] [1732873171.215259405] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2554] [1732873171.215268269] : JIT Link flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2554] [1732873171.217844867] : Program LOG for device #0:Intel(R) UHD Graphics 630:
IGC: Internal Compiler Error: Segmentation violation

CHIP error [TID 2554] [1732873171.217870563] : hipErrorNotInitialized (Device library link step failed.) in /usr/src/debug/chipstar/1.2.1/src/backend/OpenCL/CHIPBackendOpenCL.cc:1184:compile

CHIP error [TID 2554] [1732873171.218710906] : Caught Error: hipErrorNotInitialized
HIP Runtime Error: hipErrorNotInitialized
Test sub failed at : 0 || x[i]: 98 sub y[i]: -28 || GPU: 0 CPU: 126
Test sub failed at : 0 || x[i]: 98 sub y[i]: -28 || GPU: 0 CPU: 126
Test sub failed at : 1 || x[i]: -41 sub y[i]: -55 || GPU: 0 CPU: 14
Test sub failed at : 1 || x[i]: -41 sub y[i]: -55 || GPU: 0 CPU: 14
Test sub failed at : 2 || x[i]: -1 sub y[i]: 68 || GPU: 0 CPU: -69
Test sub failed at : 2 || x[i]: -1 sub y[i]: 68 || GPU: 0 CPU: -69
Test sub failed at : 3 || x[i]: -12 sub y[i]: 78 || GPU: 0 CPU: -90
Test sub failed at : 3 || x[i]: -12 sub y[i]: 78 || GPU: 0 CPU: -90
Test sub errors: 127916
CHIP info [TID 2554] [1732873171.220731604] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2554] [1732873171.220740470] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2554] [1732873171.220747789] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2554] [1732873171.220767190] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2554] [1732873171.220773872] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2554] [1732873171.220779898] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2554] [1732873171.220803858] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2554] [1732873171.220827636] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2554] [1732873171.220850036] : JIT Link flags:  -cl-kernel-arg-info -cl-std=CL3.0
Segmentation fault (core dumped)

backtrace:

[root@localhost chip_spv_samples]# gdb ./fp16_half2_math
GNU gdb (GDB) 15.1
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-oe-linux".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./fp16_half2_math...
(No debugging symbols found in ./fp16_half2_math)
(gdb) run
Starting program: /usr/bin/chip_spv_samples/fp16_half2_math 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
CHIP info [TID 2595] [1732873369.050304188] : CHIP_PLATFORM=0
CHIP info [TID 2595] [1732873369.050350797] : CHIP_DEVICE_TYPE=gpu
CHIP info [TID 2595] [1732873369.050356493] : CHIP_DEVICE=0
CHIP info [TID 2595] [1732873369.050361974] : CHIP_BE=opencl
CHIP info [TID 2595] [1732873369.050367204] : CHIP_DUMP_SPIRV=off
CHIP info [TID 2595] [1732873369.050372440] : CHIP_JIT_FLAGS_OVERRIDE=
CHIP info [TID 2595] [1732873369.050377649] : CHIP_L0_COLLECT_EVENTS_TIMEOUT=0
CHIP info [TID 2595] [1732873369.050386399] : CHIP_L0_EVENT_TIMEOUT=0
CHIP info [TID 2595] [1732873369.050391675] : CHIP_SKIP_UNINIT=off
CHIP info [TID 2595] [1732873369.050398458] : CHIP_LAZY_JIT=on
CHIP info [TID 2595] [1732873369.050404499] : CHIP_OCL_DISABLE_QUEUE_PROFILING=off
CHIP info [TID 2595] [1732873369.050410748] : CHIP_OCL_USE_ALLOC_STRATEGY=off
CHIP info [TID 2595] [1732873369.050418339] : CHIP_MODULE_CACHE_DIR=/root/.cache/chipStar
[New Thread 0x7fffe967a6c0 (LWP 2598)]
CHIP info [TID 2595] [1732873369.303598903] : OpenCL Devices of type gpu with SPIR-V_1 support:
Intel(R) UHD Graphics 630  is supported.

CHIP info [TID 2595] [1732873369.306880636] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2595] [1732873369.306899636] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2595] [1732873369.306907792] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2595] [1732873369.306913914] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2595] [1732873369.306921572] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2595] [1732873369.306929420] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2595] [1732873369.306937322] : JIT flags:  -cl-kernel-arg-info -cl-std=CL3.0
CHIP info [TID 2595] [1732873369.306944548] : Program LOG for device #0:Intel(R) UHD Graphics 630:


CHIP info [TID 2595] [1732873369.306954010] : JIT Link flags:  -cl-kernel-arg-info -cl-std=CL3.0

Thread 1 "fp16_half2_math" received signal SIGSEGV, Segmentation fault.
0x00007fffdd1324aa in ?? () from /usr/lib/libLLVM-15.so
(gdb) bt
#0  0x00007fffdd1324aa in ?? () from /usr/lib/libLLVM-15.so
#1  0x00007fffdd1496ba in llvm::BitcodeWriter::writeModule(llvm::Module const&, bool, llvm::ModuleSummaryIndex const*, bool, std::array<unsigned int, 5ul>*) () from /usr/lib/libLLVM-15.so
#2  0x00007fffdd14b2a1 in llvm::WriteBitcodeToFile(llvm::Module const&, llvm::raw_ostream&, bool, llvm::ModuleSummaryIndex const*, bool, std::array<unsigned int, 5ul>*) () from /usr/lib/libLLVM-15.so
#3  0x00007fffd50332c6 in ?? () from /usr/lib/libigc.so.1
#4  0x00007fffd50ad241 in IGC::IgcOclTranslationCtx<0ul>::Impl::Translate(unsigned long, CIF::Builtins::Buffer<1ul>*, CIF::Builtins::Buffer<1ul>*, CIF::Builtins::Buffer<1ul>*, CIF::Builtins::Buffer<1ul>*, CIF::Builtins::Buffer<1ul>*, CIF::Builtins::Buffer<1ul>*, unsigned int, void*) const () from /usr/lib/libigc.so.1
#5  0x00007fffd50af033 in IGC::IgcOclTranslationCtx<2ul>::TranslateImpl(unsigned long, CIF::Builtins::Buffer<1ul>*, CIF::Builtins::Buffer<1ul>*, CIF::Builtins::Buffer<1ul>*, CIF::Builtins::Buffer<1ul>*, unsigned int, void*) () from /usr/lib/libigc.so.1
#6  0x00007ffff701a184 in ?? () from /usr/lib/intel-opencl/libigdrcl.so
#7  0x00007ffff6f97eda in ?? () from /usr/lib/intel-opencl/libigdrcl.so
#8  0x00007ffff7862dc7 in clLinkProgram () from /usr/lib/libOpenCL.so.1
#9  0x00007ffff7f07faf in cl::linkProgram (inputPrograms=..., options=0x5555557687e0 " -cl-kernel-arg-info -cl-std=CL3.0", notifyFptr=0x0, data=0x0, err=<optimized out>) at include/CL/opencl.hpp:6976
#10 CHIPModuleOpenCL::compile (this=0x5555556dcc60, ChipDev=0x555555733510) at /usr/src/debug/chipstar/1.2.1/src/backend/OpenCL/CHIPBackendOpenCL.cc:1175
#11 0x00007ffff7f2259f in CHIPDeviceOpenCL::compile (this=0x555555733510, SrcMod=...) at src/backend/OpenCL/CHIPBackendOpenCL.hh:334
#12 0x00007ffff7e62cac in chipstar::Device::getOrCreateModule (this=0x555555733510, SrcMod=...) at /usr/src/debug/chipstar/1.2.1/src/CHIPBackend.cc:1051
#13 0x00007ffff7e6210a in chipstar::Device::getOrCreateModule (this=0x555555733510, Ptr=...) at /usr/src/debug/chipstar/1.2.1/src/CHIPBackend.cc:986
#14 0x00007ffff7ee02ee in chipstar::Device::prepareDeviceVariables (this=0x555555733510, Ptr=...) at src/CHIPBackend.cc:946
#15 hipLaunchKernelInternal (HostFunction=0x55555555fc38, GridDim=..., BlockDim=..., Args=0x7fffffffe230, SharedMem=0, Stream=0x0) at /usr/src/debug/chipstar/1.2.1/src/CHIPBindings.cc:5599
#16 0x00007ffff7ee001d in hipLaunchKernel (HostFunction=0x55555555fc38, GridDim=..., BlockDim=..., Args=0x7fffffffe230, SharedMem=<optimized out>, Stream=<optimized out>)
    at /usr/src/debug/chipstar/1.2.1/src/CHIPBindings.cc:5618
#17 0x00005555555576ee in ?? ()
#18 0x00007ffff796afdb in __libc_start_call_main (main=main@entry=0x555555557400, argc=argc@entry=1, argv=argv@entry=0x7fffffffe398)
    at /usr/src/debug/glibc/2.40+git/sysdeps/nptl/libc_start_call_main.h:58
#19 0x00007ffff796b099 in __libc_start_main_impl (main=0x555555557400, argc=1, argv=0x7fffffffe398, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe388)
    at /usr/src/debug/glibc/2.40+git/csu/libc-start.c:360
#20 0x0000555555556255 in ?? ()
(gdb) quit
A debugging session is active.

	Inferior 1 [process 2595] will be killed.

Quit anyway? (y or n) y

The crash is inside the Intel OpenCL driver and eventually in libLLVM-15.so.

I will try to rebuild it with LLVM 14, as LLVM 15 support in the Intel Graphics Compiler is considered beta quality.

[zboszor]

Fixed the IGC / Compute Runtime build on my side with Yocto. Now it uses LLVM 14 and an updated version of the SPIRV-LLVM-Translator from the LLVM 14 compatible branch.

zboszor/meta-clang-revival#3

The previously crashing fp16 related tests work and report "passed". I tried running every HIP and CUDA samples and they all work.

[pvelesko]

@zboszor ran into issues compiling AdaptiveCpp, will try again today

[pvelesko]

@zboszor switching to acpp for compiling the MKLShim layer is not as trivial as I expected as all of the cmake export targets from MKL (which we do not control) expect icpx
This will take some more time but I'll try to look into it more next week.

# We use the SYCL version of MKL, so ensure that our compiler
# knows how to compile SYCL.
# TODO this *only* supports the Intel C++ compiler.
find_package(IntelSYCL REQUIRED)
if(IntelSYCL_FOUND)
    # Ensure that when we look for MKL, we import the
    # support for using its SYCL version.
    # Despite the name of the variable, we are only
    # after the SYCL version, not DPC++ support.
    set(DPCPP_COMPILER ON)
endif()

[zboszor]

Take your time. Thank you very much for looking into it.

[zboszor]

Has there been any progress on this front?

FWIW, AdaptiveCpp has grown something called Portable CUDA since I opened this issue.

[pvelesko]

Haven't looked into this yet since we are focused on supporting more scientific applications and libraries at this time.

FWIW, AdaptiveCpp has grown something called Portable CUDA since I opened this issue.

Not as relevant for us as we support CUDA already. Unless I am missing something?

I think we could enable AdaptiveCpp if we drop support for HipBLAS and other libraries.

[zboszor]

Haven't looked into this yet since we are focused on supporting more scientific applications and libraries at this time.

FWIW, AdaptiveCpp has grown something called Portable CUDA since I opened this issue.

Not as relevant for us as we support CUDA already. Unless I am missing something?

I was thinking whether AdaptiveCpp can be used already for compiling e.g. PyTorch, or chipStar would be a better choice.

I think we could enable AdaptiveCpp if we drop support for HipBLAS and other libraries.

That would be a pity. I gather HipBLAS is used as a replacement for cuBLAS, right? As such, it would be a loss because PyTorch wants to use that in its CUDA build.

I will somehow try to support icpx (in x86-64 target builds) so hopefully the nvcc replacement of chipStar can use the complete set of replacements in a cross-compiler environment, too.

My goal is to be able build some Python modules using either chipStar or AdaptiveCpp to fake CUDA enough in Yocto's cross-compiler environment, so that those modules may be used via a mature OpenCL driver, which should be Rusticl in the hopefully not too far future.

Currently the main contender for me is cupy, and eventually PyTorch. Anything that builds on PyTorch usually wants the CUDA build and does not account for any other accelerated methods, like AMD ROCm, or the Intel XPU, of which the latter also needs icpx.

But AFAIK icpx does not support anything besides x86(-64) so that would be somewhat limited.

So my question is: if I get to the point of building chipStar using icpx as the native compiler, can chipStar's nvcc replacement then build anything else than x86-64? The big problem here is that the supplementary libraries (like HipBLAS) would still not be available for the cross-compiled target (e.g. arm32, aarch64, or something else) if I assessed the situation correctly.

[pvelesko]

So my question is: if I get to the point of building chipStar using icpx as the native compiler, can chipStar's nvcc replacement then build anything else than x86-64? The big problem here is that the supplementary libraries (like HipBLAS) would still not be available for the cross-compiled target (e.g. arm32, aarch64, or something else) if I assessed the situation correctly.

I think that's correct. Currently we support hipBLAS and similar libraries by converting calls to MKL.