[libclc] Support LLVM_ENABLE_RUNTIMES when building

[frasercrmck]

This commit deprecates the use of LLVM_ENABLE_PROJECTS in favour of LLVM_ENABLE_RUNTIMES when building libclc.

Alternatively, using -DLLVM_RUNTIME_TARGETS= combined with -DRUNTIMES__LLVM_ENABLE_RUNTIMES=libclc also gets pretty far but fails due to zlib problems building the LLVM utility 'prepare_builtins'. I'm not sure what's going on there but I don't think it's required at this stage. More work would be required to support that option.

This does nothing to change how the host tools are found in order to be used to actually build the libclc libraries.

Note that under such a configuration the final libclc builtin libraries are placed in <build>/runtimes/runtimes-bins/libclc/, which differs from a non-runtimes build. The installation location remains the same.

Fixes #124013.

[jhuber6]

The changes make sense for just adding it, but does it actually work? The CMake I've seen in libclc does multiple compilations and some with --target=amdgcn for example, which is a little different from how the runtimes builds handle it.

Do you have an example of how to actually build this stuff?

[frasercrmck]

The changes make sense for just adding it, but does it actually work?

Truth be told I haven't grasped exactly how the runtimes system works. This "works" in that -DLLVM_ENABLE_RUNTIMES=libclc will build all libclc targets in, e.g., build/runtimes/runtimes-bins/libclc/amdgcn--amdhsa.bc and CMake options appear to be passed through to libclc correctly: -DLIBCLC_TARGETS_TO_BUILD=amdgcn--amdhsa will build only that libclc target.

That said, I don't believe it "works" in the way it's supposed to. It still grabs the host tools using get_host_tool_path in CMake, and custom commands to build with that. I take it we're supposed to use CMAKE_C_COMPILER as if we were a regular CMake project? Are we able to use opt and llvm-link and other LLVM tools?

The CMake I've seen in libclc does multiple compilations and some with --target=amdgcn for example, which is a little different from how the runtimes builds handle it.

Yes the libclc build system is a little quirky. It uses custom commands to compile OpenCL C to LLVM IR. This was to allow in-tree builds without requiring the compiler to be built before configure time. Now it seems that perhaps, ironically, that this is exactly how the runtimes build works and so we might be best to undo all of that. It was done partly also as OpenCL C and LLVM IR (the two source languages) aren't recognized by CMake as languages and so needed some unsafe & undocumented CMake code to get working. I hope we'd be able to avoid that.

Do you have an example of how to actually build this stuff?

I've been using this command to test:

cmake -S llvm -B build -G Ninja -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="libclc" -DLIBCLC_TARGETS_TO_BUILD=amdgcn--amdhsa -DCMAKE_BUILD_TYPE=Release

That's about it as far as libclc is concerned. It's not integrated into clang and there's no tests for it. That's something I'm hoping to change.

One question you might be able to answer: how can I treat the runtimes build more like a "regular" build? For instance ninja -v doesn't forward -v onto the build so I can't inspect what's happening while building libclc. I also can't build specific libclc targets such as ninja prepare-amdgcn--amdhsa.bc.

[jhuber6]

That said, I don't believe it "works" in the way it's supposed to. It still grabs the host tools using get_host_tool_path in CMake, and custom commands to build with that. I take it we're supposed to use CMAKE_C_COMPILER as if we were a regular CMake project? Are we able to use opt and llvm-link and other LLVM tools?

Yes, the whole point is that we can use add_library instead of custom commands. You can still find those tools and use them if necessary however. For that you'd want to emit an object library and then add a custom command that uses all the generated objects to link / opt them and finally output them. OpenMP's GPU runtime used to do that but I hacked around it with LTO stuff.

I've been using this command to test:

cmake -S llvm -B build -G Ninja -DLLVM_ENABLE_ASSERTIONS=ON -DLLVM_ENABLE_PROJECTS="clang" -DLLVM_ENABLE_RUNTIMES="libclc" -DLIBCLC_TARGETS_TO_BUILD=amdgcn--amdhsa -DCMAKE_BUILD_TYPE=Release

Yeah, something like LIBCLC_TARGETS_TO_BUILD is likely what should be implied by the LLVM_DEFAULT_TARGET_TRIPLE value that the runtimes build passes in. (Also, does amdgcn-amd-amdhsa not work? That's the canonical one).

One question you might be able to answer: how can I treat the runtimes build more like a "regular" build? For instance ninja -v doesn't forward -v onto the build so I can't inspect what's happening while building libclc. I also can't build specific libclc targets such as ninja prepare-amdgcn--amdhsa.bc.

It's an external project, it uses the same handling. You can do ninja -C ./runtimes/runtimes-<triple>-bins and it will behave the same.

[frasercrmck]

That said, I don't believe it "works" in the way it's supposed to. It still grabs the host tools using get_host_tool_path in CMake, and custom commands to build with that. I take it we're supposed to use CMAKE_C_COMPILER as if we were a regular CMake project? Are we able to use opt and llvm-link and other LLVM tools?

Yes, the whole point is that we can use add_library instead of custom commands. You can still find those tools and use them if necessary however. For that you'd want to emit an object library and then add a custom command that uses all the generated objects to link / opt them and finally output them. OpenMP's GPU runtime used to do that but I hacked around it with LTO stuff.

I'll need to look into that - maybe we can talk offline. Since libclc is used by downstream toolchains I feel it'll be hard to significantly change how it's built or presented to the host. We could support two methods of building but that would get sticky pretty quickly.

Yeah, something like LIBCLC_TARGETS_TO_BUILD is likely what should be implied by the LLVM_DEFAULT_TARGET_TRIPLE value that the runtimes build passes in. (Also, does amdgcn-amd-amdhsa not work? That's the canonical one).

I'm not sure about how best to deal with LIBCLC_TARGETS_TO_BUILD. It could also pick up from LLVM_TARGETS_TO_BUILD but we have targets like clspv and clspv64 which don't conveniently match any LLVM triple and are used by a downstream.

I'm not sure about how the AMD triples work, perhaps @arsenm can answer that. I know that downstream we're changing the final bytecode library to amdgcn-amd-amdhsa - perhaps for a similar reason.

It's an external project, it uses the same handling. You can do ninja -C ./runtimes/runtimes-<triple>-bins and it will behave the same.

Ah yes, thank you 👍.

[arsenm]

There are 3 triples. amdgcn-mesa-mesa3d is the only one that ever used libclc for OpenCL. amdgcn-- ideally wouldn't exist, and would migrate to using mesa-mesa3d, but I think only OpenGL ever used it. There's also amdgcn-amd-amdpal used on Vulkan, but also does not use libclc. amdgcn-amd-amdhsa is slightly different from amdgcn-mesa-mesa3d (the mesa3d one was attempting to emulate amdhsa but couldn't 100% match)

[jhuber6]

I'll need to look into that - maybe we can talk offline. Since libclc is used by downstream toolchains I feel it'll be hard to significantly change how it's built or presented to the host. We could support two methods of building but that would get sticky pretty quickly.

Sure, feel free to email me or ping me on the LLVM Discord.

[frasercrmck]

I'm assuming the libc++ tests aren't failing due to this PR. They seem unstable, looking at jobs run for other PRs.

[frasercrmck]

I'd quite like to keep this check/error if possible. The error you get if you forget -LLVM_ENABLE_PROJECTS=clang is horrible. Do you think it would it acceptable to pass through LLVM_ENABLE_PROJECTS to the runtimes build, @jhuber6?

[jhuber6]

The runtimes builds pass this USE_TOOLCHAIN argument which tells the external project utility to grab the clang from the build directory. If we don't have clang it might be more reasonable to just emit a warning there. @petrhosek Is there a reason we don't already emit a warning / error if the user does a runtimes build without clang to bootstrap?

[frasercrmck]

Thanks. Yes I suppose in the absence of a higher-level warning like you describe, we could probably use USE_TOOLCHAIN to accomplish the same thing in a runtimes build.

With any luck this is all temporary, and a runtimes build will allow us to just take the cmake c compiler.