Skip to content

[Misc] Do not override NCCL_CUMEM_ENABLE if set explicitly #19105

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jun 5, 2025
Merged

[Misc] Do not override NCCL_CUMEM_ENABLE if set explicitly #19105

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7174b2e
Do not override NCCL_CUMEM_ENABLE if set explicitly
22quinn Jun 3, 2025
086ec9d
Remove noisy warnings
22quinn Jun 4, 2025
3895180
bot is awesome
22quinn Jun 5, 2025
bce20cc
show NCCL_CUMEM_ENABLE value too
22quinn Jun 5, 2025
8fd1f7c
show NCCL_CUMEM_ENABLE value too
22quinn Jun 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 12 additions & 6 deletions vllm/env_override.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,17 +3,23 @@

import torch

from vllm.logger import init_logger

logger = init_logger(__name__)

# set some common config/environment variables that should be set
# for all processes created by vllm and all processes
# that interact with vllm workers.
# they are executed whenever `import vllm` is called.

if not os.path.exists('/dev/nvidia-caps-imex-channels'):
# normally, we disable NCCL_CUMEM_ENABLE because it
# will cost 1~2 GiB GPU memory with cudagraph+allreduce,
# see https://github.com/NVIDIA/nccl/issues/1234
# for more details.
# However, NCCL requires NCCL_CUMEM_ENABLE to work with
if 'NCCL_CUMEM_ENABLE' in os.environ:
logger.warning(
"NCCL_CUMEM_ENABLE is set to %s, skipping override. "
"This may increase memory overhead with cudagraph+allreduce: "
"https://github.com/NVIDIA/nccl/issues/1234",
os.environ['NCCL_CUMEM_ENABLE'])
elif not os.path.exists('/dev/nvidia-caps-imex-channels'):
# NCCL requires NCCL_CUMEM_ENABLE to work with
# multi-node NVLink, typically on GB200-NVL72 systems.
# The ultimate way to detect multi-node NVLink is to use
# NVML APIs, which are too expensive to call here.
Expand Down