Manage NVIDIA GPU slots in local executor

[bentsherman]

Experimenting with GPU slots for the local executor.

When a pipeline runs multiple GPU-enabled tasks on the same node, each task will see all GPUs and will not try to coordinate which task should use which GPU.

NVIDIA provides the CUDA_VISIBLE_DEVICES / NVIDIA_VISIBLE_DEVICES environment variables to control which tasks can see which GPUs, and users generally have to manage this variable themselves. Some HPC schedulers can assign this variable automatically, or use cgroups to control GPU visibility at a lower level.

Nextflow should be able to manage this variable for the local executor, so that the user doesn't have to add complex pipeline logic to do the same. Running a GPU workload locally on a multi-GPU node is a common use case, so it might be worth doing.

To use this feature, set executor.gpus in your config to the total number of available GPUs. Then Nextflow will automatically set the CUDA_VISIBLE_DEVICES for each task based on the accelerator directive.

To use with containers, you might have to add CUDA_VISIBLE_DEVICES to docker.envWhitelist.

• I don't remember if CUDA_VISIBLE_DEVICES works with containers or if you have to set NVIDIA_VISIBLE_DEVICES.
• I don't remember if you have to set --gpus for the docker command in order to use the GPUs at all, that can be set in docker.runOptions

See also: #5570

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	4f7b7ab
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/67d554239590420008266872
😎 Deploy Preview	https://deploy-preview-5850--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.