Consider cudatoolkit==11.8 for Docker/Singularity image

[mjw99]

Using the openfe_1.7.0-apptainer.sif singularity image with a L40S GPU, for a simple benzene AHFE calculation, I see the following crash:

Traceback (most recent call last):
  File "/tmp/AHFE_benzene.py", line 90, in <module>
    dag_results = execute_DAG(dag, scratch_basedir=path, shared_basedir=path, n_retries=3)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/gufe/protocols/protocoldag.py", line 450, in execute_DAG
    result = unit.execute(context=context, raise_error=raise_error, **inputs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/gufe/protocols/protocolunit.py", line 371, in execute
    outputs = self._execute(context, **inputs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/openfe/protocols/openmm_afe/base.py", line 1252, in _execute
    outputs = self.run(scratch_basepath=ctx.scratch, shared_basepath=ctx.shared)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/openfe/protocols/openmm_afe/base.py", line 1124, in run
    positions, box_vectors = self._pre_equilibrate(
                             ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/openfe/protocols/openmm_afe/base.py", line 215, in _pre_equilibrate
    simulation = openmm.app.Simulation(
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/openmm/app/simulation.py", line 104, in __init__
    self.context = mm.Context(self.system, self.integrator, platform)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.12/site-packages/openmm/openmm.py", line 12171, in __init__
    _openmm.Context_swiginit(self, _openmm.new_Context(*args))
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
openmm.OpenMMException: Error initializing FFT: 5

Also note:

$ singularity shell openfe_1.7.0-apptainer.sif
Apptainer> /opt/conda/bin/python
Python 3.12.12 | packaged by conda-forge | (main, Oct 13 2025, 14:34:15) [GCC 14.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import openmm

>>> print(openmm.__version__)
8.1.1

This is a function of cudatoolkit==11.7 and bumping this to 11.8 in production/environment.yml , rebuilding the images, resolves my issue.

[mikemhenry]

Thanks for this report! Can you send me the output of nvidia-smi on the node where you ran this calculation? I know that you got it to work with bumping the cudatoolkit version but I'm curious what driver version you had running.

I think we are going to start building multiple images with different tags based on the cuda version and I'd like to get a sense of what versions people need.

Thanks!
Mike

[mjw99]

Sure; thanks for looking at this:

$ nvidia-smi 
Tue Oct 28 09:09:43 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 565.57.01              Driver Version: 565.57.01      CUDA Version: 12.7     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40S                    On  |   00000000:25:00.0 Off |                    0 |
| N/A   50C    P0            116W /  350W |    1755MiB /  46068MiB |     10%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA H100 NVL                On  |   00000000:E1:00.0 Off |                    0 |
| N/A   36C    P0             63W /  400W |       1MiB /  95830MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A    525366      C   /opt/conda/bin/python                        1730MiB |
+-----------------------------------------------------------------------------------------+

[mikemhenry]

Thanks! I am guessing this weirdness is more on the conda-forge side of things wrt how packaging cuda versions works since according to this complicated table:
https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/#id6
and this easier to grok table:
https://docs.nvidia.com/deploy/cuda-compatibility/minor-version-compatibility.html#id1
your driver should work fine with an application built with cuda 11.7 or cuda 11.8.
Things have started to settle on conda-forge for cuda packaging, but for awhile (cuda < 12) things were done very bespoke.

[mjw99]

Thank for this. I think specifically with the L40S GPU, it is cc8.9 (compute capabilities)[1] == Ada Lovelace.
Then from table [2], I think CUDA SDK version 11.8 is needed.

[1] https://developer.nvidia.com/cuda-gpus
[2] https://en.wikipedia.org/wiki/CUDA#GPUs_supported