Single node LUMI example - Failed to infer device type

[kaiserdan]

Unfortunately also the single node LUMI example script does not work as provided as the dependencies provided in /appl/local/csc/modulefiles/ fail with the error RuntimeError: Failed to infer device type during startup.

NOTE: This module uses Singularity. Some commands execute inside the container
(e.g. python3, pip3).

This module has been installed by CSC.

Documentation: https://docs.csc.fi/apps/pytorch/
Support: https://docs.csc.fi/support/contact/

Starting vLLM process 152953 - logs go to /scratch/project_xxx/xxx/vllm-logs/.log
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.10/dist-packages/vllm-0.6.6.dev27+ge461c262.rocm624-py3.10-linux-x86_64.egg/vllm/entrypoints/openai/api_server.py", line 709, in <module>
    uvloop.run(run_server(args))
  File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.10/dist-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/usr/local/lib/python3.10/dist-packages/vllm-0.6.6.dev27+ge461c262.rocm624-py3.10-linux-x86_64.egg/vllm/entrypoints/openai/api_server.py", line 675, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/vllm-0.6.6.dev27+ge461c262.rocm624-py3.10-linux-x86_64.egg/vllm/entrypoints/openai/api_server.py", line 118, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/usr/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/usr/local/lib/python3.10/dist-packages/vllm-0.6.6.dev27+ge461c262.rocm624-py3.10-linux-x86_64.egg/vllm/entrypoints/openai/api_server.py", line 140, in build_async_engine_client_from_engine_args
    engine_config = engine_args.create_engine_config(
  File "/usr/local/lib/python3.10/dist-packages/vllm-0.6.6.dev27+ge461c262.rocm624-py3.10-linux-x86_64.egg/vllm/engine/arg_utils.py", line 1041, in create_engine_config
    device_config = DeviceConfig(device=self.device)
  File "/usr/local/lib/python3.10/dist-packages/vllm-0.6.6.dev27+ge461c262.rocm624-py3.10-linux-x86_64.egg/vllm/config.py", line 1491, in __init__
    raise RuntimeError("Failed to infer device type")
RuntimeError: Failed to infer device type

[kaiserdan]

In the most recent version of vllm a new debug output became available to pinpoint this issue: vllm-project/vllm#14195

The issue is just that the non-working version in /appl/local/csc/modulefiles/ is 0.6.6 while the latest is 0.8.2.

[mvsjober]

Hello, this is strange. I just tried it myself and it worked fine:

NOTE: This module uses Singularity. Some commands execute inside the container
(e.g. python3, pip3).

This module has been installed by CSC.

Documentation: https://docs.csc.fi/apps/pytorch/
Support: https://docs.csc.fi/support/contact/

Starting vLLM process 98617 - logs go to /scratch/project_462000007/mvsjober/vllm-logs/10053151.log
Loading safetensors checkpoint shards:   0% Completed | 0/17 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   6% Completed | 1/17 [00:00<00:07,  2.28it/s]
Loading safetensors checkpoint shards:  12% Completed | 2/17 [00:02<00:16,  1.11s/it]
Loading safetensors checkpoint shards:  18% Completed | 3/17 [00:04<00:22,  1.58s/it]
Loading safetensors checkpoint shards:  24% Completed | 4/17 [00:14<01:05,  5.03s/it]
Loading safetensors checkpoint shards:  29% Completed | 5/17 [00:24<01:19,  6.66s/it]
Loading safetensors checkpoint shards:  35% Completed | 6/17 [00:24<00:51,  4.72s/it]
Loading safetensors checkpoint shards:  41% Completed | 7/17 [00:34<01:03,  6.32s/it]
Loading safetensors checkpoint shards:  47% Completed | 8/17 [00:35<00:41,  4.63s/it]
Loading safetensors checkpoint shards:  53% Completed | 9/17 [00:36<00:26,  3.36s/it]
Loading safetensors checkpoint shards:  59% Completed | 10/17 [00:47<00:40,  5.75s/it]
Loading safetensors checkpoint shards:  65% Completed | 11/17 [00:47<00:24,  4.03s/it]
Loading safetensors checkpoint shards:  71% Completed | 12/17 [00:47<00:14,  2.89s/it]
Loading safetensors checkpoint shards:  76% Completed | 13/17 [00:57<00:19,  4.93s/it]
Loading safetensors checkpoint shards:  82% Completed | 14/17 [01:07<00:19,  6.49s/it]
Loading safetensors checkpoint shards:  88% Completed | 15/17 [01:16<00:14,  7.18s/it]
Loading safetensors checkpoint shards:  94% Completed | 16/17 [01:16<00:05,  5.20s/it]
Loading safetensors checkpoint shards: 100% Completed | 17/17 [01:26<00:00,  6.42s/it]
Loading safetensors checkpoint shards: 100% Completed | 17/17 [01:26<00:00,  5.06s/it]

INFO:     Started server process [98640]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

Somehow your environment must be different. What modules do you have loaded? Or perhaps you were unlucky and got a broken node?

You can upgrade vLLM yourself using a venv, something along these lines:

module use /appl/local/csc/modulefiles/
module load pytorch/2.5
python3 -m venv --system-site-packages venv
source venv/bin/activate
git clone https://github.com/vllm-project/vllm
cd vllm
pip install setuptools --upgrade
python setup.py install

[mvsjober]

Did you submit the script with sbatch? I guess this wasn't explicitly stated in the README, I now added a note about that.

[kaiserdan]

Thank you very, very much for the quick help! I'm trying it out right now (also for the multi-node ray script), and i think this was the problem.

[kaiserdan]

Yes, I can confirm, this was the issue. It's working great now! Thank you!