Error: `resource already mapped` in `custom_all_reduce.cuh`

[yippp]

This is a issue relared to #2619 (comment)

I have tried ray=2.9.1 with dev code in commit #2636

vllm.entrypoints.openai.api_server --model ./Mistral-7B-Instruct-v0.2-AWQ --quantization awq --dtype auto --host 0.0.0.0 --port 8081 --tensor-parallel-size 2
but I meet another error
Failed: Cuda error /home/my/vllm/csrc/custom_all_reduce.cuh:417 'resource already mapped' Segmentation fault (core dumped)

I am running with python=3.11, CUDA 12.1, driver 530 with 2x RTX 3090 NVLink.
When I rollback to commit #2622, the program works well. So it seems it is caused by #2192

[hanzhi713]

@yippp Can you try if upgrading to driver 535 helps? I highly suspect this is a driver bug, as cudaIpcOpenMemHandle shouldn't return such an error and should return an existing handle of the IPC address is already mapped.

If not, I can do a software fix as well.

[yippp]

@yippp Can you try if upgrading to driver 535 helps? I highly suspect this is a driver bug, as cudaIpcOpenMemHandle shouldn't return such an error and should return an existing handle of the IPC address is already mapped.

If not, I can do a software fix as well.

@hanzhi713 Thanks for your sugguestion, I have tried to update the driver to newest 535.154.05, unfortunalty the same error appears.

Failed: Cuda error /home/my/vllm/csrc/custom_all_reduce.cuh:417 'resource already mapped'
[W CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]

[hanzhi713]

@yippp
I've pushed a software fix: #2642. Please check it out.

Also, could you please check if you're using the latest version of cuda 12.1 (i.e. 12.1 update 1)? Because other maintainers and I don't have this issue on our platforms.

[yippp]

@yippp I've pushed a software fix: #2642. Please check it out.

Also, could you please check if you're using the latest version of cuda 12.1 (i.e. 12.1 update 1)? Because other maintainers and I don't have this issue on our platforms.

thanks for your work, I have tried to run the code with fix #2642. Unfortunalty, the same error appears. I also check the cuda version, it is 12.1 update 1. Maybe I need to further check what is the problem with my enviroment

[WoosukKwon]

@hanzhi713 Can we directly use cudaOpenIpcHandle (w/ dummy inputs) instead of torch.cuda.can_device_access_peer to check whether the user env is suitable for the custom all reduce kernel?

[hanzhi713]

@yippp Can you double check the line number when you got the error? It should be line 432 instead of 417. Just making sure you've compiled the code.
@WoosukKwon His case is weird, because the cudaIpcOpenMemHandle that errors out is the second invocation. The first invocation is in the constructor of CustomAllreduce.

[yippp]

@hanzhi713 I have doudle check it, I use the correct source code and definity the error is in line 417 of file csrc/custom_all_reduce.cuh, i.e. the line CUDACHECK(cudaIpcOpenMemHandle(
I tried to removal the entire enviroment and reinsall it from source, but still same error.
I notice that before the error show, the graphic memory usage increased to ~300MB, and then show the error, hope this limited info could help you.

[hanzhi713]

@yippp No I mean once you use #2642, the error should be at line 432, not 417. Can you double check that?

[yippp]

@hanzhi713 Thanks for your patient instruction! I have correctly build it, and the error disappear. I have checked it, currently the program could run normally.

[WoosukKwon]

@yippp Thanks for sharing! @hanzhi713 Thanks for the fix!