GRPO training timeout after trl vllm-serve 'reset prefix cache'

[mgcyung]

A error "Operation timed out!" is reported in GRPO training as follows

Exception ignored in: <finalize object at 0x7f7d7b9bc920; dead>
Traceback (most recent call last):
  File "/root/miniconda3/envs/open-r1/lib/python3.12/weakref.py", line 585, in __call__
    def __call__(self, _=None):

  File "/root/miniconda3/envs/open-r1/lib/python3.12/site-packages/math_verify/utils.py", line 43, in handler
    raise TimeoutError("Operation timed out!")
TimeoutError: Operation timed out!
[rank2]:[E516 06:09:16.295666043 ProcessGroupNCCL.cpp:629] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=49296, OpType=_ALLGATHER_BASE, NumelIn=192, NumelOut=576, Timeout(ms)=1800000) ran for 1800076 milliseconds before timing out.
[rank1]:[E516 06:09:16.296189442 ProcessGroupNCCL.cpp:629] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=49296, OpType=_ALLGATHER_BASE, NumelIn=192, NumelOut=576, Timeout(ms)=1800000) ran for 1800030 milliseconds before timing out.
[rank1]:[E516 06:09:16.297295298 ProcessGroupNCCL.cpp:2168] [PG ID 0 PG GUID 0(default_pg) Rank 1]  failure detected by watchdog at work sequence id: 49296 PG status: last enqueued work: 49296, last completed work: 49295
[rank1]:[E516 06:09:16.297336852 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank2]:[E516 06:09:16.297331820 ProcessGroupNCCL.cpp:2168] [PG ID 0 PG GUID 0(default_pg) Rank 2]  failure detected by watchdog at work sequence id: 49296 PG status: last enqueued work: 49296, last completed work: 49295
[rank2]:[E516 06:09:16.297375817 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank2]:[E516 06:09:16.678756367 ProcessGroupNCCL.cpp:681] [Rank 2] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank2]:[E516 06:09:16.678781139 ProcessGroupNCCL.cpp:695] [Rank 2] To avoid data inconsistency, we are taking the entire process down.
[rank2]:[E516 06:09:16.682309480 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=49296, OpType=_ALLGATHER_BASE, NumelIn=192, NumelOut=576, Timeout(ms)=1800000) ran for 1800076 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f602a4581b6 in /root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7f602b7a1c74 in /root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7f602b7a37d0 in /root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f602b7a46ed in /root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x145c0 (0x7f60748275c0 in /root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/lib/libtorch.so)
frame #5: <unknown function> + 0x8609 (0x7f609167f609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f609144a133 in /lib/x86_64-linux-gnu/libc.so.6)

[rank1]:[E516 06:09:17.984764325 ProcessGroupNCCL.cpp:681] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E516 06:09:17.984795568 ProcessGroupNCCL.cpp:695] [Rank 1] To avoid data inconsistency, we are taking the entire process down.
[rank1]:[E516 06:09:17.987988525 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=49296, OpType=_ALLGATHER_BASE, NumelIn=192, NumelOut=576, Timeout(ms)=1800000) ran for 1800030 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f75077cb1b6 in /root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7f7508b14c74 in /root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7f7508b167d0 in /root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f7508b176ed in /root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x145c0 (0x7f7551b9a5c0 in /root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/lib/libtorch.so)
frame #5: <unknown function> + 0x8609 (0x7f756e9f5609 in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #6: clone + 0x43 (0x7f756e7c0133 in /lib/x86_64-linux-gnu/libc.so.6)

W0516 06:09:17.399000 3593512 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3593730 closing signal SIGTERM
W0516 06:09:17.402000 3593512 site-packages/torch/distributed/elastic/multiprocessing/api.py:897] Sending process 3593731 closing signal SIGTERM
E0516 06:09:17.718000 3593512 site-packages/torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -6) local_rank: 2 (pid: 3593732) of binary: /root/miniconda3/envs/open-r1/bin/python3.12
Traceback (most recent call last):
  File "/root/miniconda3/envs/open-r1/bin/accelerate", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/miniconda3/envs/open-r1/lib/python3.12/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
    args.func(args)
  File "/root/miniconda3/envs/open-r1/lib/python3.12/site-packages/accelerate/commands/launch.py", line 1182, in launch_command
    deepspeed_launcher(args)
  File "/root/miniconda3/envs/open-r1/lib/python3.12/site-packages/accelerate/commands/launch.py", line 861, in deepspeed_launcher
    distrib_run.run(args)
  File "/root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/distributed/run.py", line 909, in run
    elastic_launch(
  File "/root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 138, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/open-r1/lib/python3.12/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
========================================================
src/open_r1/grpo.py FAILED
--------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
--------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2025-05-16_06:09:17
  host      : 355be013b6cd
  rank      : 2 (local_rank: 2)
  exitcode  : -6 (pid: 3593732)
  error_file: <N/A>
  traceback : Signal 6 (SIGABRT) received by PID 3593732
========================================================

in about thirty minutes after trl vllm-serve 'reset prefix cache'

INFO:     127.0.0.1:55546 - "POST /update_named_param/ HTTP/1.1" 200 OK
INFO 05-16 05:38:52 [block_pool.py:264] Successfully reset prefix cache
INFO:     127.0.0.1:55546 - "POST /reset_prefix_cache/ HTTP/1.1" 200 OK
Processed prompts: 100%|█████████████████████████████████████████████████████████████████████████████████████| 192/192 [00:20<00:00,  9.52it/s, est. speed input: 1573.88 toks/s, output: 10739.85 toks/s]
INFO:     127.0.0.1:55546 - "POST /generate/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:34192 - "GET /health/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:34200 - "GET /get_world_size/ HTTP/1.1" 200 OK
INFO:     127.0.0.1:34214 - "POST /init_communicator/ HTTP/1.1" 200 OK

[Er-gou123]

I've encountered the same problem. Have you found any good solutions?

[Er-gou123]

When I tried to restart the training code, it wouldn't work and timeout.Then I stopped the VLLM server, restarted it, and the training code was able to work again. Is it the same situation with you?

[mgcyung]

I think so. And the same error showed in re-runs.

[lewtun]

Hello @mgcyung @Er-gou123 this looks like an issue with math-very timing out when trying to compute accuracy. Do you by any chance have extremely long completions or math statements to verify?

[Er-gou123]

@lewtun hello，my task has nothing to do with mathematics and its mean length of completions is about 780. I have carefully checked my issue, and it seems to be different from this. Below is the error message from the program. Could you help me with it? Is it possible that it's related to my CUDA version?Thank you

GRPO training as follow:
[rank1]:[E527 11:46:20.965210535 ProcessGroupNCCL.cpp:629] [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=30425, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=1800000) ran for 1800015 milliseconds before timing out.
[rank2]:[E527 11:46:20.023106829 ProcessGroupNCCL.cpp:629] [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=30425, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=1800000) ran for 1800073 milliseconds before timing out.
[rank3]:[E527 11:46:20.040722757 ProcessGroupNCCL.cpp:629] [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=30425, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=1800000) ran for 1800091 milliseconds before timing out.
[rank1]:[E527 11:55:15.824176957 ProcessGroupNCCL.cpp:1753] [PG ID 0 PG GUID 0(default_pg) Rank 1] ProcessGroupNCCL's watchdog got stuck for 480 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API (e.g., CudaEventDestroy) hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api (for example, CudaEventDestroy), or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang.
[rank2]:[E527 11:55:16.979961624 ProcessGroupNCCL.cpp:1753] [PG ID 0 PG GUID 0(default_pg) Rank 2] ProcessGroupNCCL's watchdog got stuck for 480 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API (e.g., CudaEventDestroy) hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api (for example, CudaEventDestroy), or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang.
[rank1]:[E527 11:57:37.153862059 ProcessGroupNCCL.cpp:2168] [PG ID 0 PG GUID 0(default_pg) Rank 1] failure detected by watchdog at work sequence id: 30425 PG status: last enqueued work: 30425, last completed work: 30424
[rank1]:[E527 11:58:37.146697955 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank3]:[E527 11:55:16.861443227 ProcessGroupNCCL.cpp:1753] [PG ID 0 PG GUID 0(default_pg) Rank 3] ProcessGroupNCCL's watchdog got stuck for 480 seconds without making progress in monitoring enqueued collectives. This typically indicates a NCCL/CUDA API (e.g., CudaEventDestroy) hang blocking the watchdog, and could be triggered by another thread holding the GIL inside a CUDA api (for example, CudaEventDestroy), or other deadlock-prone behaviors.If you suspect the watchdog is not actually stuck and a longer timeout would help, you can either increase the timeout (TORCH_NCCL_HEARTBEAT_TIMEOUT_SEC) to a larger value or disable the heartbeat monitor (TORCH_NCCL_ENABLE_MONITORING=0).If either of aforementioned helps, feel free to file an issue to PyTorch about the short timeout or false positive abort; otherwise, please attempt to debug the hang.
[rank2]:[E527 11:57:37.154038797 ProcessGroupNCCL.cpp:2168] [PG ID 0 PG GUID 0(default_pg) Rank 2] failure detected by watchdog at work sequence id: 30425 PG status: last enqueued work: 30425, last completed work: 30424
[INFO|trainer.py:3978] 2025-05-27 11:57:42,181 >> Saving model checkpoint to /data/cyh/open-r1-1/traffic/Qwen2.5-3B-2016-VPN/checkpoint-400
[rank2]:[E527 11:58:37.146841853 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[rank3]:[E527 11:57:37.154109235 ProcessGroupNCCL.cpp:2168] [PG ID 0 PG GUID 0(default_pg) Rank 3] failure detected by watchdog at work sequence id: 30425 PG status: last enqueued work: 30425, last completed work: 30424
[rank3]:[E527 11:58:37.146977764 ProcessGroupNCCL.cpp:667] Stack trace of the failed collective not found, potentially because FlightRecorder is disabled. You can enable it by setting TORCH_NCCL_TRACE_BUFFER_SIZE to a non-zero value.
[INFO|configuration_utils.py:420] 2025-05-27 11:58:37,632 >> Configuration saved in /data/cyh/open-r1-1/traffic/Qwen2.5-3B-2016-VPN/checkpoint-400/config.json
[INFO|configuration_utils.py:917] 2025-05-27 11:58:37,634 >> Configuration saved in /data/cyh/open-r1-1/traffic/Qwen2.5-3B-2016-VPN/checkpoint-400/generation_config.json
[rank2]:[E527 11:58:38.709970393 ProcessGroupNCCL.cpp:681] [Rank 2] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank2]:[E527 11:58:38.710002977 ProcessGroupNCCL.cpp:695] [Rank 2] To avoid data inconsistency, we are taking the entire process down.
[rank2]:[E527 11:58:38.714027310 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=30425, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=1800000) ran for 1800073 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f963236c1b6 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7f95dfffec74 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7f95e00007d0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f95e00016ed in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7f96326fc5c0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #5: + 0x94ac3 (0x7f9644c94ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x126660 (0x7f9644d26660 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 2] Process group watchdog thread terminated with exception: [Rank 2] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=30425, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=1800000) ran for 1800073 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f963236c1b6 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7f95dfffec74 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7f95e00007d0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f95e00016ed in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7f96326fc5c0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #5: + 0x94ac3 (0x7f9644c94ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x126660 (0x7f9644d26660 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1901 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f963236c1b6 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: + 0xe5c6fc (0x7f95dfc5c6fc in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7f96326fc5c0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #3: + 0x94ac3 (0x7f9644c94ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: + 0x126660 (0x7f9644d26660 in /lib/x86_64-linux-gnu/libc.so.6)

[rank3]:[E527 11:58:38.816576345 ProcessGroupNCCL.cpp:681] [Rank 3] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank3]:[E527 11:58:38.816601408 ProcessGroupNCCL.cpp:695] [Rank 3] To avoid data inconsistency, we are taking the entire process down.
[rank3]:[E527 11:58:38.820092474 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=30425, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=1800000) ran for 1800091 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd1041f31b6 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7fd0ad7fec74 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7fd0ad8007d0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fd0ad8016ed in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fd1043525c0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #5: + 0x94ac3 (0x7fd112894ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x126660 (0x7fd112926660 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 3] Process group watchdog thread terminated with exception: [Rank 3] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=30425, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=1800000) ran for 1800091 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd1041f31b6 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7fd0ad7fec74 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7fd0ad8007d0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fd0ad8016ed in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fd1043525c0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #5: + 0x94ac3 (0x7fd112894ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x126660 (0x7fd112926660 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1901 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd1041f31b6 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: + 0xe5c6fc (0x7fd0ad45c6fc in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7fd1043525c0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #3: + 0x94ac3 (0x7fd112894ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: + 0x126660 (0x7fd112926660 in /lib/x86_64-linux-gnu/libc.so.6)

[rank1]:[E527 11:58:38.902180331 ProcessGroupNCCL.cpp:681] [Rank 1] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
[rank1]:[E527 11:58:38.902211602 ProcessGroupNCCL.cpp:695] [Rank 1] To avoid data inconsistency, we are taking the entire process down.
[rank1]:[E527 11:58:38.906281869 ProcessGroupNCCL.cpp:1895] [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=30425, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=1800000) ran for 1800015 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd2c536c1b6 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7fd2733fec74 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7fd2734007d0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fd2734016ed in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fd2c5a735c0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #5: + 0x94ac3 (0x7fd2d7e94ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x126660 (0x7fd2d7f26660 in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG ID 0 PG GUID 0(default_pg) Rank 1] Process group watchdog thread terminated with exception: [Rank 1] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=30425, OpType=ALLGATHER, NumelIn=1, NumelOut=4, Timeout(ms)=1800000) ran for 1800015 milliseconds before timing out.
Exception raised from checkTimeout at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:632 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd2c536c1b6 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x2b4 (0x7fd2733fec74 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x890 (0x7fd2734007d0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7fd2734016ed in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #4: + 0x145c0 (0x7fd2c5a735c0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #5: + 0x94ac3 (0x7fd2d7e94ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #6: + 0x126660 (0x7fd2d7f26660 in /lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at /pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1901 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fd2c536c1b6 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: + 0xe5c6fc (0x7fd27305c6fc in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0x145c0 (0x7fd2c5a735c0 in /data/cyh/openr1/lib/python3.11/site-packages/torch/lib/libtorch.so)
frame #3: + 0x94ac3 (0x7fd2d7e94ac3 in /lib/x86_64-linux-gnu/libc.so.6)
frame #4: + 0x126660 (0x7fd2d7f26660 in /lib/x86_64-linux-gnu/libc.so.6)

W0527 11:58:39.008000 966097 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 966363 closing signal SIGTERM
W0527 11:58:39.011000 966097 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 966364 closing signal SIGTERM
W0527 11:58:39.012000 966097 torch/distributed/elastic/multiprocessing/api.py:897] Sending process 966366 closing signal SIGTERM
E0527 11:58:39.829000 966097 torch/distributed/elastic/multiprocessing/api.py:869] failed (exitcode: -6) local_rank: 2 (pid: 966365) of binary: /data/cyh/openr1/bin/python3
Traceback (most recent call last):
File "/data/cyh/openr1/bin/accelerate", line 10, in
sys.exit(main())
^^^^^^
File "/data/cyh/openr1/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 48, in main
args.func(args)
File "/data/cyh/openr1/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1182, in launch_command
deepspeed_launcher(args)
File "/data/cyh/openr1/lib/python3.11/site-packages/accelerate/commands/launch.py", line 861, in deepspeed_launcher
distrib_run.run(args)
File "/data/cyh/openr1/lib/python3.11/site-packages/torch/distributed/run.py", line 909, in run
elastic_launch(
File "/data/cyh/openr1/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 138, in call
return launch_agent(self._config, self._entrypoint, list(args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/cyh/openr1/lib/python3.11/site-packages/torch/distributed/launcher/api.py", line 269, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

src/open_r1/grpo.py FAILED

Failures:
<NO_OTHER_FAILURES>

Root Cause (first observed failure):
[0]:
time : 2025-05-27_11:58:39
host : uubn
rank : 2 (local_rank: 2)
exitcode : -6 (pid: 966365)
error_file: <N/A>
traceback : Signal 6 (SIGABRT) received by PID 966365

The VLLM server is stuck at the following lines：
INFO 05-27 11:14:15 [block_pool.py:264] Successfully reset prefix cache
INFO: 127.0.0.1:35280 - "POST /reset_prefix_cache/ HTTP/1.1" 200 OK
Processed prompts: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████| 256/256 [00:32<00:00, 7.89it/s, est. speed input: 9889.41 toks/s, output: 5697.17 toks/s]
INFO: 127.0.0.1:35280 - "POST /generate/ HTTP/1.1" 200 OK

[mgcyung]

Hello @mgcyung @Er-gou123 this looks like an issue with math-very timing out when trying to compute accuracy. Do you by any chance have extremely long completions or math statements to verify?

The training was the GRPO example in README

[lewtun]

Thanks @mgcyung let me try to reproduce it on my side