[Bug]: CUDA error: an illegal memory access was encountered

[jifa513]

Your current environment

Failed to pickle inputs of failed execution: CUDA error: an illegal memory access was encountered

🐛 Describe the bug

running Llama-3.3-70B-Instruct-FP8-Dynamic with vllm-0.6.6 , and the server crashed randomly, does anyone has encountered the same problem? And here is the error log:

�[1;36m(VllmWorkerProcess pid=210)�[0;0m WARNING 04-08 01:00:08 model_runner_base.py:143] Failed to pickle inputs of failed execution: CUDA error: an illegal memory access was encountered
�[1;36m(VllmWorkerProcess pid=210)�[0;0m WARNING 04-08 01:00:08 model_runner_base.py:143] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
�[1;36m(VllmWorkerProcess pid=210)�[0;0m WARNING 04-08 01:00:08 model_runner_base.py:143] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
�[1;36m(VllmWorkerProcess pid=210)�[0;0m WARNING 04-08 01:00:08 model_runner_base.py:143] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
�[1;36m(VllmWorkerProcess pid=210)�[0;0m WARNING 04-08 01:00:08 model_runner_base.py:143]
INFO 04-08 01:00:08 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20250408-010008.pkl...
WARNING 04-08 01:00:08 model_runner_base.py:143] Failed to pickle inputs of failed execution: CUDA error: an illegal memory access was encountered
WARNING 04-08 01:00:08 model_runner_base.py:143] CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
WARNING 04-08 01:00:08 model_runner_base.py:143] For debugging consider passing CUDA_LAUNCH_BLOCKING=1
WARNING 04-08 01:00:08 model_runner_base.py:143] Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
WARNING 04-08 01:00:08 model_runner_base.py:143]
ERROR 04-08 01:00:08 engine.py:135] RuntimeError('Error in model execution: Error Internal')
ERROR 04-08 01:00:08 engine.py:135] Traceback (most recent call last):
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
ERROR 04-08 01:00:08 engine.py:135] return func(*args, **kwargs)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 2028, in execute_model
ERROR 04-08 01:00:08 engine.py:135] hidden_or_intermediate_states = model_executable(
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 04-08 01:00:08 engine.py:135] return self._call_impl(*args, **kwargs)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 04-08 01:00:08 engine.py:135] return forward_call(*args, **kwargs)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 578, in forward
ERROR 04-08 01:00:08 engine.py:135] model_output = self.model(input_ids, positions, kv_caches,
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/compilation/decorators.py", line 168, in call
ERROR 04-08 01:00:08 engine.py:135] return self.forward(*args, **kwargs)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 366, in forward
ERROR 04-08 01:00:08 engine.py:135] hidden_states, residual = layer(positions, hidden_states,
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 04-08 01:00:08 engine.py:135] return self._call_impl(*args, **kwargs)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 04-08 01:00:08 engine.py:135] return forward_call(*args, **kwargs)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 279, in forward
ERROR 04-08 01:00:08 engine.py:135] hidden_states = self.self_attn(positions=positions,
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 04-08 01:00:08 engine.py:135] return self._call_impl(*args, **kwargs)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 04-08 01:00:08 engine.py:135] return forward_call(*args, **kwargs)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/llama.py", line 204, in forward
ERROR 04-08 01:00:08 engine.py:135] qkv, _ = self.qkv_proj(hidden_states)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 04-08 01:00:08 engine.py:135] return self._call_impl(*args, **kwargs)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in call_impl
ERROR 04-08 01:00:08 engine.py:135] return forward_call(*args, **kwargs)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 376, in forward
ERROR 04-08 01:00:08 engine.py:135] output_parallel = self.quant_method.apply(self, input, bias)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors.py", line 511, in apply
ERROR 04-08 01:00:08 engine.py:135] return scheme.apply_weights(layer, x, bias=bias)
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py", line 135, in apply_weights
ERROR 04-08 01:00:08 engine.py:135] return apply_fp8_linear(
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/w8a8_utils.py", line 120, in apply_fp8_linear
ERROR 04-08 01:00:08 engine.py:135] output = ops.cutlass_scaled_mm(qinput,
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/_custom_ops.py", line 493, in cutlass_scaled_mm
ERROR 04-08 01:00:08 engine.py:135] torch.ops._C.cutlass_scaled_mm(out, a, b, scale_a, scale_b, bias)
ERROR 04-08 01:00:08 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1116, in call
ERROR 04-08 01:00:08 engine.py:135] return self._op(*args, **(kwargs or {}))
ERROR 04-08 01:00:08 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-08 01:00:08 engine.py:135] RuntimeError: Error Internal

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[QuanhuiGuan]

i met the sample problem in vllm 0.6.3 . Does anyone have the solution that could fix this problem? thanks a lot

[rank0]:[E414 02:22:43.769134885 ProcessGroupNCCL.cpp:1515] [PG 3 Rank 0] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe5a88cbf86 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fe5a887ad10 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fe5a89a6f08 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fe55a95b3e6 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fe55a960600 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7fe55a9672ba in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fe55a9696fc in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0xd44a3 (0x7fe5a80ba4a3 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: + 0x891c4 (0x7fe5a96f11c4 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: __clone + 0x40 (0x7fe5a9770ac0 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG 3 Rank 0] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe5a88cbf86 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fe5a887ad10 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fe5a89a6f08 in /usr/local/lib/python3.9/site-packages/torch/l
ib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fe55a95b3e6 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_c
uda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fe55a960600 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7fe55a9672ba in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fe55a9696fc in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0xd44a3 (0x7fe5a80ba4a3 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: + 0x891c4 (0x7fe5a96f11c4 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: __clone + 0x40 (0x7fe5a9770ac0 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1521 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe5a88cbf86 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: + 0xe5aa84 (0x7fe55a5f2a84 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0xd44a3 (0x7fe5a80ba4a3 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: + 0x891c4 (0x7fe5a96f11c4 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #4: __clone + 0x40 (0x7fe5a9770ac0 in /usr/lib/x86_64-linux-gnu/libc.so.6)

[rank1]:[E414 02:22:43.776456891 ProcessGroupNCCL.cpp:1515] [PG 3 Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory acces
s was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe5a88cbf86 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fe5a887ad10 in /usr/local/lib/python3.9/site-packages/tor
ch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fe5a89a6f08 in /usr/local/lib/python3.9/site-packages/torch/l
ib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fe55a95b3e6 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_c
uda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fe55a960600 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7fe55a9672ba in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fe55a9696fc in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0xd44a3 (0x7fe5a80ba4a3 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: + 0x891c4 (0x7fe5a96f11c4 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: __clone + 0x40 (0x7fe5a9770ac0 in /usr/lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'c10::DistBackendError'
what(): [PG 3 Rank 1] Process group watchdog thread terminated with exception: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe5a88cbf86 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x7fe5a887ad10 in /usr/local/lib/python3.9/site-packages/tor
ch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x7fe5a89a6f08 in /usr/local/lib/python3.9/site-packages/torch/l
ib/libc10_cuda.so)
frame #3: c10d::ProcessGroupNCCL::WorkNCCL::finishedGPUExecutionInternal() const + 0x56 (0x7fe55a95b3e6 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_c
uda.so)
frame #4: c10d::ProcessGroupNCCL::WorkNCCL::isCompleted() + 0xa0 (0x7fe55a960600 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #5: c10d::ProcessGroupNCCL::watchdogHandler() + 0x1da (0x7fe55a9672ba in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #6: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x10c (0x7fe55a9696fc in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #7: + 0xd44a3 (0x7fe5a80ba4a3 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #8: + 0x891c4 (0x7fe5a96f11c4 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #9: __clone + 0x40 (0x7fe5a9770ac0 in /usr/lib/x86_64-linux-gnu/libc.so.6)

Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1521 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7fe5a88cbf86 in /usr/local/lib/python3.9/site-packages/torch/lib/libc10.so)
frame #1: + 0xe5aa84 (0x7fe55a5f2a84 in /usr/local/lib/python3.9/site-packages/torch/lib/libtorch_cuda.so)
frame #2: + 0xd44a3 (0x7fe5a80ba4a3 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #3: + 0x891c4 (0x7fe5a96f11c4 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #4: __clone + 0x40 (0x7fe5a9770ac0 in /usr/lib/x86_64-linux-gnu/libc.so.6)

ERROR 04-14 02:22:52 client.py:250] TimeoutError('No heartbeat received from MQLLMEngine')
ERROR 04-14 02:22:52 client.py:250] NoneType: None
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48514 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48516 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48518 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48520 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48522 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48524 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48526 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48528 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48530 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48532 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
CRITICAL 04-14 02:22:52 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 10.5.115.138:48534 - "POST /v1/chat/completions HTTP/1.0" 500 Internal Server Error
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [7]

[whNLP]

Same issue

[PLUTO-SCY]

same issue, have tried some approaches, failed

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!