Closed
Description
Describe the bug
When we upgraded to CUDA.jl 5, non-init jobs started triggering precompilation (which then triggere a race condition in deleting a scratch directory, which threw an error due to a bug fixed by JuliaLang/julia#50842, which is not yet in a release version of Julia).
This was caused by adding the CUDA driver library as an "include_dependency" to CUDA_Runtime_jll (JuliaGPU/CUDA.jl#1877), which meant that if we loaded CUDA on a node without the driver available, it would trigger recompilation.
I proposed a fix in JuliaPackaging/Yggdrasil#7523, but it needs more work.