[Bug]: vLLM Worker Process Crash (died unexpectedly) with Qwen3-Next Model when Enabling MTP on NVIDIA A800 #25368
Description
Your current environment
vLLM Version: 0.10.2
Model: Qwen3-Next
Hardware: NVIDIA A800 GPUs
Startup Command: The engine was configured with speculative decoding:
speculative_config=SpeculativeConfig(method='qwen3_next_mtp', model='/model', num_spec_tokens=2) # <-- CRASH CONDITION
Other Relevant Config:
tensor_parallel_size=4
max_seq_len=131072
🐛 Describe the bug
I'm experiencing a consistent and fatal crash when running the Qwen3-Next model on NVIDIA A800 GPUs using vLLM. The crash occurs specifically and only when MTP speculative decoding is enabled. The worker process dies unexpectedly, leading to a complete shutdown of the executor.
The model operates perfectly normally and is stable under high load when MTP is disabled.
When enable MTP, with the configuration of 100 input tokens, 1000 output tokens, and 128 concurrent requests, we can stably reproduce this issue.
Error info：
Engine 000: Avg prompt throughput: 108.0 tokens/s, Avg generation throughput: 138.7 tokens/s, Running: 128 reqs, Waiting: 0 reqs, GPU KV cache usage: 19.8%, Prefix cache hit rate: 0.0% SpecDecoding metrics: Mean acceptance length: 2.25, Accepted throughput: 76.61 tokens/s, Drafted throughput: 122.21 tokens/s, Accepted: 766 tokens, Drafted: 1222 tokens, Per-position acceptance rate: 0.756, 0.498, Avg Draft acceptance rate: 62.7% multiproc_executor.py:149 MultiprocWorkerMonitor - �[31;21m[ERROR]�[0m Worker proc VllmWorker-0 died unexpectedly, shutting down executor. dump_input.py:69 MainThread - �[31;21m[ERROR]�[0m Dumping input data for V1 LLM engine (v0.10.2) with config: model='/model', speculative_config=SpeculativeConfig(method='qwen3_next_mtp', model='/model', num_spec_tokens=2), tokenizer='/model', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=4, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/model, enable_prefix_caching=False, chunked_prefill_enabled=True, use_async_output_proc=False, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null}, dump_input.py:76 MainThread - �[31;21m[ERROR]�[0m Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[], scheduled_cached_reqs=CachedRequestData(req_ids=['7fadcecad1b64d7ea4da7564ae7c3cdc', '9eabcf4509a44ee38b974c241e706a5c', 'f31d3f6293a647dcb4aa45aea8f9afb1', 'c3b9c0c2d5674c60bd08a6d378c6e21a', '5d7187b6d60a47ba8cbf528289e9c6df', '80164578daa441588f327ce6d64c7e54', '3458f4c138e24ad0a2af46dfa179c5fa', 'b6c9b4f3348447579dda853819dda212', '37e80dac593f4052a19cc772fd097897', 'c31e792cf41b40c19fd6452afb94f384', '5370361ad3a242fcb8c5b9d6d29c19b1', 'cc1596084ad24ba69931a1df3ee98cf7', '87061036b97d4206a1dcd11fc96c2227', 'ee31cd6cf8e540a9b9720e027762c954', '6ccceebaca024b9992779e5865ce2ed9', 'e213c126746a410fac2ad7ab2e8437f8', '435c9618639a4ddbaf6c2a1ee1ad563d', '981825b3eb224989aeafa9404d66c446', '02dcac9142954f99a242e670bdb2d83e', '5e2d2e81a9b547f29275d71eb79458b5', '8a1b06ae647e4494a184e4e4d408469f', '9444981618924434b0e1e55c8f6493b8', 'bafc6b3acfe44e7cb4f336cd1c68f3ed', '5338ec36b81a497dad50936a5bd0c65b', '5111343544fa4c958b830438192221b9', '51848725ef8542119b6ea8ea512900a6', 'f4c5d98705c74475ac2f5c96ab2d1a41', '4a1965a25bfe4f6fb6e95a44a2cdb907', 'dc0e290a5eee45ac9691bfafd4f48104', '99f05ce11a2943ec8df9d2be6dcfe59d', '2e0f13f13580443ca7d92d7a4d1f558e', '785c6537ac9542d785c4259aef7145f4', '577c69425f06416d9d7788de39c2099d', '119cc7aedf5f41c99ee87e404e57cde6', 'e89f37f66b0a4c069eaf75ca3730ce92', '7b2a6cc328594cb2a5578b4cc41fb740', '0eac17850d6d45ce8cc2549278109510', '13e2d03cceaa49db99584f4c7c4f7b61', 'f90379c3ac5a48f497a9b1f606356f0e', '510a75147f264db3999f480edd6cacc6', 'e6f8d1653b634ba5b442fafbdf7dae8e', 'cb76dd4c0e594840a446cdb79e1dff75', '12f98b569d6d4086aa8cd7bded4aee26', 'd400c3d26f314f44b1a8e6ee7bf1647e', '889026084b9b469ba7e168670b9e9fba', 'd8fafa010b4b4dd3a0d274f6b954e5d6', '8b35b8d7c927461d8598ea0ccbd4622f', '628b3dd206c4467d929d9bc952b87424', '0e8d32ae4e1e4e0583a9e07f624d4a4c', '47e46aa7dea94e42a02baf9b29531392', '086b73a798f441cd83275dd0192ed08a', '9088d539271b4e3e856b9722cba329f0', 'd8ca2015f8f3448caa95404a5ce36b57', 'cdb20865506b47ffa1d6e21057701413', 'f6b4b02745ff4b789ef9f38bc3860708', 'b91fe135aa82409d81d51ceca07aab62', '9a884782de8b4ef582311229295271c3', '29f07d16cfd74cefbd2b79867148d56a', '0cf9ccd2449546038a7a3e99ae026c10', 'ddc3c806808945c3ad5153898ec50087', '195ddee189ed4691a8183d2b4b44d5c4', 'f591456bbcab4f918c7b0b0ffe8c8541', '67c7849bac85449aac72e57f77e71dd8', 'f64978d20a8a46bdb8fc026d1ae40981', 'c0866f9b63e343c8bac45885b0e0f15b', '96d086926e5b4aa699267bf64e81d3e6', 'e2b433a0027845859b8ce16694340414', '8962c0213a464cb784fa112f605c83d4', '0235e20d48c34bf8b65a6908c432f104', '92a71e9878584cba98e896c4bff034b1', 'deaf88d84b2f4a268680a5114e91364a', '47f6d77401d44e379d1818202c9b2df8', 'a0378bf0b3e54e7186973ff79915895b', '43db11292ae844ba92628f2b2ba55ec2', 'ef184abe74494d8882cf5569c247eedb', '6b31646cd8ae45629eb62c9428a0ed5b', '18cf43fad27f48c68ccd2cd4d36cfeab', '47956f824940400c8b3b754dafeb0b80', '390734752bb043dea61e9a99656a9024', 'e6b2d9b123134b2d846919a5ec2c8090', 'c9bf2a7817bc477a93f40dc3074c34f5', '95cc9e6a0db841289cf0ca3ddeea1154', 'd76d935f85c446f1906a60bd313d166c', 'e9767d375c96466695b240f6c2f0863c', '3f3a81045109472baab796ff8ad484be', '3b5c630888f645a6a178ac5f429ebdaa', '23f6ee214c4449e596865e21b5f02040', 'e9b4ec37a2264a39965e487035532ffc', 'e40869d83c5045e4b87c06177ec8a85a', '0f0c96f34bea4c03b30d733a28eda9e6', '03f5cc2996a94ac9921f1c0b53dbe4ef', '153c4bbac7494691a4a814590eae0f9c', '5ac00ffc519b43ce9bbf1f9354b269bf', '6efe0850aae64c0e972cf0aa9bee6fb9', 'e3b12402ac0b4894a6aa8d574c4f9980', 'a912d3d606434a1d8f53944da9b90a90', 'ccb99efffbf24bc4863525842a738608', 'bd5512a873484daeba9cf2944e6f8bc8', '804b1e69be124c1ca043ad0cb3ff4559', '07dd93a7985f4a95a47e0b86c486ef70', 'ee5399644c8949ab9283e61bd5c81efd', 'f9f74ab6e8294273a3c9b6dedbb3f2a9', '46969f3d5eaf42f4a606c9e5ac72f318', '7033058401784b04910c90842d9e7096', 'ef1be69bf338454196d83c6354f4bb2d', 'e4ce90cd8efc47bfac23f28f1ff9fdbb', '57db340b03d34f0aac2dadab9d053c83', 'c4367688b04a455689c5a3914d341cdb', 'a34b617183a44615beda877b717ba451', '60ac55948a9c4d30a48cb79d726448a0', '488e4af954c840d3afc0251855ab4dee', '3d8fbbc4fa5842b89aa7a73ca8690890', '25c944e3cdea4d43adba1e0f3313dc8f', '16361aad625047d6b52c8b1996b5c335', '4bc559bda7eb4b49a7de203df70a5820', '4ccb4f841bce4c2eafc85c305de590cc', '532973765d144f319136ff10bf634cab', '85425908fa8e44a897220c301e4bc4f5', '54ba5ae6fd7d4770bdb8ea9ab7b2250e', '0997f6e07a494ecbab48f377462aa334', '5a4642fdde7440b98a7bb464fef58574', '82c1b91299cb46219e7d99637d98d734', 'af8055901e2c4f8289a893a2e1121b86', 'c245701fa47348f8965daea3042dcf7f', '8f05384c8b2746c8a77127bc5bfd0e08', 'fdc5e954e8a645f1ae479b74715a3014', 'fa6d01bf643a4d43aabb67998e82e910', '2885e3eddbb744a3a8c3fbc2debb7b51'], resumed_from_preemption=[false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false, false], new_token_ids=[], new_block_ids=[null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null, null], num_computed_tokens=[551, 405, 371, 426, 391, 429, 391, 373, 355, 378, 359, 329, 357, 423, 359, 335, 332, 342, 377, 340, 375, 338, 345, 309, 331, 291, 356, 336, 295, 316, 312, 339, 279, 310, 276, 280, 255, 274, 255, 274, 251, 244, 247, 246, 250, 258, 222, 241, 234, 220, 214, 217, 222, 238, 205, 222, 211, 193, 210, 182, 194, 193, 186, 177, 175, 171, 180, 165, 157, 171, 163, 166, 164, 166, 173, 170, 168, 155, 160, 160, 154, 160, 147, 149, 145, 152, 145, 148, 134, 142, 146, 145, 144, 145, 148, 139, 142, 148, 142, 136, 140, 133, 136, 141, 132, 134, 121, 129, 127, 127, 126, 120, 125, 124, 120, 118, 119, 115, 116, 120, 116, 116, 113, 112, 111, 111, 111, 108]), num_scheduled_tokens={510a75147f264db3999f480edd6cacc6: 3, 5370361ad3a242fcb8c5b9d6d29c19b1: 3, 8b35b8d7c927461d8598ea0ccbd4622f: 3, 8962c0213a464cb784fa112f605c83d4: 3, 5338ec36b81a497dad50936a5bd0c65b: 3, f64978d20a8a46bdb8fc026d1ae40981: 3, 2885e3eddbb744a3a8c3fbc2debb7b51: 3, d8fafa010b4b4dd3a0d274f6b954e5d6: 3, 95cc9e6a0db841289cf0ca3ddeea1154: 3, 8a1b06ae647e4494a184e4e4d408469f: 3, 23f6ee214c4449e596865e21b5f02040: 3, 8f05384c8b2746c8a77127bc5bfd0e08: 3, 7033058401784b04910c90842d9e7096: 3, 87061036b97d4206a1dcd11fc96c2227: 3, e3b12402ac0b4894a6aa8d574c4f9980: 3, a34b617183a44615beda877b717ba451: 3, f31d3f6293a647dcb4aa45aea8f9afb1: 3, 0235e20d48c34bf8b65a6908c432f104: 3, fdc5e954e8a645f1ae479b74715a3014: 3, 3458f4c138e24ad0a2af46dfa179c5fa: 3, d8ca2015f8f3448caa95404a5ce36b57: 3, 67c7849bac85449aac72e57f77e71dd8: 3, c31e792cf41b40c19fd6452afb94f384: 3, 3b5c630888f645a6a178ac5f429ebdaa: 3, 51848725ef8542119b6ea8ea512900a6: 3, 9444981618924434b0e1e55c8f6493b8: 3, 60ac55948a9c4d30a48cb79d726448a0: 3, 5111343544fa4c958b830438192221b9: 3, 16361aad625047d6b52c8b1996b5c335: 3, 6b31646cd8ae45629eb62c9428a0ed5b: 3, 488e4af954c840d3afc0251855ab4dee: 3, 119cc7aedf5f41c99ee87e404e57cde6: 3, 0cf9ccd2449546038a7a3e99ae026c10: 3, 07dd93a7985f4a95a47e0b86c486ef70: 3, c245701fa47348f8965daea3042dcf7f: 3, 3d8fbbc4fa5842b89aa7a73ca8690890: 3, 532973765d144f319136ff10bf634cab: 3, 981825b3eb224989aeafa9404d66c446: 3, 390734752bb043dea61e9a99656a9024: 3, c4367688b04a455689c5a3914d341cdb: 3, 85425908fa8e44a897220c301e4bc4f5: 3, 02dcac9142954f99a242e670bdb2d83e: 3, dc0e290a5eee45ac9691bfafd4f48104: 3, e89f37f66b0a4c069eaf75ca3730ce92: 3, 47e46aa7dea94e42a02baf9b29531392: 3, af8055901e2c4f8289a893a2e1121b86: 3, 47956f824940400c8b3b754dafeb0b80: 3, d400c3d26f314f44b1a8e6ee7bf1647e: 3, 37e80dac593f4052a19cc772fd097897: 3, e6f8d1653b634ba5b442fafbdf7dae8e: 3, 54ba5ae6fd7d4770bdb8ea9ab7b2250e: 3, bafc6b3acfe44e7cb4f336cd1c68f3ed: 3, c0866f9b63e343c8bac45885b0e0f15b: 3, e9b4ec37a2264a39965e487035532ffc: 3, 6efe0850aae64c0e972cf0aa9bee6fb9: 3, 6ccceebaca024b9992779e5865ce2ed9: 3, e4ce90cd8efc47bfac23f28f1ff9fdbb: 3, 92a71e9878584cba98e896c4bff034b1: 3, cc1596084ad24ba69931a1df3ee98cf7: 3, f90379c3ac5a48f497a9b1f606356f0e: 3, c3b9c0c2d5674c60bd08a6d378c6e21a: 3, 9eabcf4509a44ee38b974c241e706a5c: 3, f591456bbcab4f918c7b0b0ffe8c8541: 3, d76d935f85c446f1906a60bd313d166c: 3, deaf88d84b2f4a268680a5114e91364a: 3, 785c6537ac9542d785c4259aef7145f4: 3, 5ac00ffc519b43ce9bbf1f9354b269bf: 3, 4ccb4f841bce4c2eafc85c305de590cc: 3, 82c1b91299cb46219e7d99637d98d734: 3, 96d086926e5b4aa699267bf64e81d3e6: 3, c9bf2a7817bc477a93f40dc3074c34f5: 3, ef184abe74494d8882cf5569c247eedb: 3, 99f05ce11a2943ec8df9d2be6dcfe59d: 3, 43db11292ae844ba92628f2b2ba55ec2: 3, 9088d539271b4e3e856b9722cba329f0: 3, f6b4b02745ff4b789ef9f38bc3860708: 3, 12f98b569d6d4086aa8cd7bded4aee26: 3, cb76dd4c0e594840a446cdb79e1dff75: 3, 086b73a798f441cd83275dd0192ed08a: 3, 25c944e3cdea4d43adba1e0f3313dc8f: 3, 889026084b9b469ba7e168670b9e9fba: 3, 5d7187b6d60a47ba8cbf528289e9c6df: 3, a0378bf0b3e54e7186973ff79915895b: 3, 2e0f13f13580443ca7d92d7a4d1f558e: 3, f4c5d98705c74475ac2f5c96ab2d1a41: 3, f9f74ab6e8294273a3c9b6dedbb3f2a9: 3, 804b1e69be124c1ca043ad0cb3ff4559: 3, 7b2a6cc328594cb2a5578b4cc41fb740: 3, 628b3dd206c4467d929d9bc952b87424: 3, e9767d375c96466695b240f6c2f0863c: 3, 46969f3d5eaf42f4a606c9e5ac72f318: 3, 5e2d2e81a9b547f29275d71eb79458b5: 3, 03f5cc2996a94ac9921f1c0b53dbe4ef: 3, 0f0c96f34bea4c03b30d733a28eda9e6: 3, ee5399644c8949ab9283e61bd5c81efd: 3, a912d3d606434a1d8f53944da9b90a90: 3, 577c69425f06416d9d7788de39c2099d: 3, e2b433a0027845859b8ce16694340414: 3, ef1be69bf338454196d83c6354f4bb2d: 3, fa6d01bf643a4d43aabb67998e82e910: 3, ccb99efffbf24bc4863525842a738608: 3, cdb20865506b47ffa1d6e21057701413: 3, 0e8d32ae4e1e4e0583a9e07f624d4a4c: 3, b91fe135aa82409d81d51ceca07aab62: 3, ee31cd6cf8e540a9b9720e027762c954: 3, 80164578daa441588f327ce6d64c7e54: 3, 4a1965a25bfe4f6fb6e95a44a2cdb907: 3, 195ddee189ed4691a8183d2b4b44d5c4: 3, 29f07d16cfd74cefbd2b79867148d56a: 3, e213c126746a410fac2ad7ab2e8437f8: 3, 3f3a81045109472baab796ff8ad484be: 3, 47f6d77401d44e379d1818202c9b2df8: 3, 9a884782de8b4ef582311229295271c3: 3, 435c9618639a4ddbaf6c2a1ee1ad563d: 3, 57db340b03d34f0aac2dadab9d053c83: 3, 0997f6e07a494ecbab48f377462aa334: 3, 5a4642fdde7440b98a7bb464fef58574: 3, 0eac17850d6d45ce8cc2549278109510: 3, 7fadcecad1b64d7ea4da7564ae7c3cdc: 3, bd5512a873484daeba9cf2944e6f8bc8: 3, ddc3c806808945c3ad5153898ec50087: 3, e40869d83c5045e4b87c06177ec8a85a: 3, 4bc559bda7eb4b49a7de203df70a5820: 3, 13e2d03cceaa49db99584f4c7c4f7b61: 3, 153c4bbac7494691a4a814590eae0f9c: 3, b6c9b4f3348447579dda853819dda212: 3, e6b2d9b123134b2d846919a5ec2c8090: 3, 18cf43fad27f48c68ccd2cd4d36cfeab: 3}, total_num_scheduled_tokens=384, scheduled_spec_decode_tokens={f591456bbcab4f918c7b0b0ffe8c8541: [3890, 304], c3b9c0c2d5674c60bd08a6d378c6e21a: [304, 425], 5ac00ffc519b43ce9bbf1f9354b269bf: [646, 17247], 47f6d77401d44e379d1818202c9b2df8: [369, 7263], d76d935f85c446f1906a60bd313d166c: [438, 3078], c9bf2a7817bc477a93f40dc3074c34f5: [1654, 3129], ccb99efffbf24bc4863525842a738608: [25, 220], 6ccceebaca024b9992779e5865ce2ed9: [476, 31528], 46969f3d5eaf42f4a606c9e5ac72f318: [553, 279], 99f05ce11a2943ec8df9d2be6dcfe59d: [678, 31428], 0235e20d48c34bf8b65a6908c432f104: [39118, 323], 5e2d2e81a9b547f29275d71eb79458b5: [1260, 15634], 7b2a6cc328594cb2a5578b4cc41fb740: [102989, 101896], e3b12402ac0b4894a6aa8d574c4f9980: [9086, 1447], 7fadcecad1b64d7ea4da7564ae7c3cdc: [1308, 17068], 0eac17850d6d45ce8cc2549278109510: [806, 41223], b6c9b4f3348447579dda853819dda212: [323, 5312], 804b1e69be124c1ca043ad0cb3ff4559: [5972, 23782], 2885e3eddbb744a3a8c3fbc2debb7b51: [0, 5692], 37e80dac593f4052a19cc772fd097897: [11, 323], 3f3a81045109472baab796ff8ad484be: [11, 323], 80164578daa441588f327ce6d64c7e54: [11, 293], 03f5cc2996a94ac9921f1c0b53dbe4ef: [5068, 3941], fdc5e954e8a645f1ae479b74715a3014: [264, 4583], fa6d01bf643a4d43aabb67998e82e910: [2303, 78516], 29f07d16cfd74cefbd2b79867148d56a: [12, 576], e2b433a0027845859b8ce16694340414: [311, 3395], 43db11292ae844ba92628f2b2ba55ec2: [25, 28338], 9444981618924434b0e1e55c8f6493b8: [5547, 1078], e4ce90cd8efc47bfac23f28f1ff9fdbb: [1393, 22205], a912d3d606434a1d8f53944da9b90a90: [49272, 45779], f9f74ab6e8294273a3c9b6dedbb3f2a9: [6765, 11], 82c1b91299cb46219e7d99637d98d734: [34119, 6040], 8962c0213a464cb784fa112f605c83d4: [334, 6923], a34b617183a44615beda877b717ba451: [40526, 21597], 02dcac9142954f99a242e670bdb2d83e: [32908, 311], c0866f9b63e343c8bac45885b0e0f15b: [18626, 434], 0cf9ccd2449546038a7a3e99ae026c10: [1019, 12], 85425908fa8e44a897220c301e4bc4f5: [1091, 68672], e6f8d1653b634ba5b442fafbdf7dae8e: [26117, 10072], ef184abe74494d8882cf5569c247eedb: [76337, 63332], cdb20865506b47ffa1d6e21057701413: [1075, 1565], b91fe135aa82409d81d51ceca07aab62: [274, 21785], 628b3dd206c4467d929d9bc952b87424: [40582, 133935], 13e2d03cceaa49db99584f4c7c4f7b61: [323, 2266], c245701fa47348f8965daea3042dcf7f: [311, 6481], e89f37f66b0a4c069eaf75ca3730ce92: [311, 387], ee5399644c8949ab9283e61bd5c81efd: [264, 6524], 8f05384c8b2746c8a77127bc5bfd0e08: [14436, 292], 3458f4c138e24ad0a2af46dfa179c5fa: [3083, 4759], ddc3c806808945c3ad5153898ec50087: [80419, 382], c4367688b04a455689c5a3914d341cdb: [71654, 41780], 07dd93a7985f4a95a47e0b86c486ef70: [26741, 2021], 87061036b97d4206a1dcd11fc96c2227: [279, 6174], cb76dd4c0e594840a446cdb79e1dff75: [11, 19241], 4ccb4f841bce4c2eafc85c305de590cc: [311, 387], 16361aad625047d6b52c8b1996b5c335: [23126, 323], f4c5d98705c74475ac2f5c96ab2d1a41: [13, 220], dc0e290a5eee45ac9691bfafd4f48104: [2293, 327], f6b4b02745ff4b789ef9f38bc3860708: [68, 1302], 153c4bbac7494691a4a814590eae0f9c: [59413, 287], 47956f824940400c8b3b754dafeb0b80: [646, 3070], 510a75147f264db3999f480edd6cacc6: [87, 936], 9eabcf4509a44ee38b974c241e706a5c: [11, 17461], 488e4af954c840d3afc0251855ab4dee: [10161, 892], e6b2d9b123134b2d846919a5ec2c8090: [323, 73191], 6efe0850aae64c0e972cf0aa9bee6fb9: [1387, 309], 981825b3eb224989aeafa9404d66c446: [432, 382], 47e46aa7dea94e42a02baf9b29531392: [2188, 382], 6b31646cd8ae45629eb62c9428a0ed5b: [10689, 38425], 9088d539271b4e3e856b9722cba329f0: [854, 56177], f90379c3ac5a48f497a9b1f606356f0e: [28372, 334], 4bc559bda7eb4b49a7de203df70a5820: [54962, 13], ef1be69bf338454196d83c6354f4bb2d: [44730, 448], 5338ec36b81a497dad50936a5bd0c65b: [382, 641], 5111343544fa4c958b830438192221b9: [53386, 2303], 195ddee189ed4691a8183d2b4b44d5c4: [13443, 2959], 119cc7aedf5f41c99ee87e404e57cde6: [330, 5113], e213c126746a410fac2ad7ab2e8437f8: [279, 4265], 8b35b8d7c927461d8598ea0ccbd4622f: [1405, 1101], d8ca2015f8f3448caa95404a5ce36b57: [5729, 48155], 23f6ee214c4449e596865e21b5f02040: [315, 279], f31d3f6293a647dcb4aa45aea8f9afb1: [3207, 1405], 67c7849bac85449aac72e57f77e71dd8: [7225, 8975], deaf88d84b2f4a268680a5114e91364a: [20239, 11], 3b5c630888f645a6a178ac5f429ebdaa: [334, 1865], 18cf43fad27f48c68ccd2cd4d36cfeab: [369, 320], 9a884782de8b4ef582311229295271c3: [4185, 11513], 086b73a798f441cd83275dd0192ed08a: [63270, 82], 57db340b03d34f0aac2dadab9d053c83: [26103, 1959], bafc6b3acfe44e7cb4f336cd1c68f3ed: [264, 19221], 390734752bb043dea61e9a99656a9024: [374, 2669], 25c944e3cdea4d43adba1e0f3313dc8f: [11050, 1616], 0f0c96f34bea4c03b30d733a28eda9e6: [879, 30], d400c3d26f314f44b1a8e6ee7bf1647e: [9402, 323], 7033058401784b04910c90842d9e7096: [12417, 4380], 96d086926e5b4aa699267bf64e81d3e6: [16645, 13], 60ac55948a9c4d30a48cb79d726448a0: [3170, 1447], 0997f6e07a494ecbab48f377462aa334: [67, 6654], 51848725ef8542119b6ea8ea512900a6: [279, 42984], c31e792cf41b40c19fd6452afb94f384: [31902, 8743], e9b4ec37a2264a39965e487035532ffc: [9237, 32295], 8a1b06ae647e4494a184e4e4d408469f: [32468, 15627], ee31cd6cf8e540a9b9720e027762c954: [589, 364], 532973765d144f319136ff10bf634cab: [94763, 11], 889026084b9b469ba7e168670b9e9fba: [220, 16], cc1596084ad24ba69931a1df3ee98cf7: [2293, 2591], af8055901e2c4f8289a893a2e1121b86: [11, 86416], bd5512a873484daeba9cf2944e6f8bc8: [1467, 11], 92a71e9878584cba98e896c4bff034b1: [60403, 12], 95cc9e6a0db841289cf0ca3ddeea1154: [16801, 62465], f64978d20a8a46bdb8fc026d1ae40981: [785, 4531], 12f98b569d6d4086aa8cd7bded4aee26: [7771, 5887], e40869d83c5045e4b87c06177ec8a85a: [4479, 323], 3d8fbbc4fa5842b89aa7a73ca8690890: [752, 6248], 2e0f13f13580443ca7d92d7a4d1f558e: [6771, 752], 4a1965a25bfe4f6fb6e95a44a2cdb907: [1246, 264], a0378bf0b3e54e7186973ff79915895b: [2213, 9700], 5d7187b6d60a47ba8cbf528289e9c6df: [4216, 817], 785c6537ac9542d785c4259aef7145f4: [8299, 4286], d8fafa010b4b4dd3a0d274f6b954e5d6: [1366, 311], 5a4642fdde7440b98a7bb464fef58574: [382, 785], e9767d375c96466695b240f6c2f0863c: [11, 8480], 577c69425f06416d9d7788de39c2099d: [7567, 95518], 5370361ad3a242fcb8c5b9d6d29c19b1: [264, 11477], 435c9618639a4ddbaf6c2a1ee1ad563d: [2494, 498], 0e8d32ae4e1e4e0583a9e07f624d4a4c: [11, 22111], 54ba5ae6fd7d4770bdb8ea9ab7b2250e: [11, 2477]}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[0, 0, 0, 0], finished_req_ids=[], free_encoder_mm_hashes=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null) Dumping scheduler stats: SchedulerStats(num_running_reqs=128, num_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.19836833358513373, prefix_cache_stats=PrefixCacheStats(reset=False, requests=0, queries=0, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0) EngineCore encountered a fatal error. Traceback (most recent call last): File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 711, in run_engine_core engine_core.run_busy_loop() File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 738, in run_busy_loop self._process_engine_step() File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 764, in _process_engine_step outputs, model_executed = self.step_fn() ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 292, in step model_output = self.execute_model_with_error_logging( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 278, in execute_model_with_error_logging raise err File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 269, in execute_model_with_error_logging return model_fn(scheduler_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 176, in execute_model (output, ) = self.collective_rpc( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 259, in collective_rpc result = get_response(w, dequeue_timeout, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 239, in get_response status, result = w.worker_response_mq.dequeue( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 507, in dequeue with self.acquire_read(timeout, cancel) as buf: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 464, in acquire_read raise RuntimeError("cancelled") RuntimeError: cancelled async_llm.py:485 MainThread - �[31;21m[ERROR]�[0m AsyncLLM output_handler failed. Traceback (most recent call last): File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 444, in output_handler outputs = await engine_core.get_output_async() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 845, in get_output_async raise self._format_exception(outputs) from None vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.