[Bug]: Incremental detokenization error when running `llama-3.3-70b-fp8` model

[njhill]

Recently reported by @npalaska:

we have been seeing the following error with vLLM 0.9.2 and transformers 4.52.4 when serving the llama-3.3-70b-fp8 and recently with maverick fp8 as well.

INFO:     10.131.1.21:55318 - "POST /v1/completions HTTP/1.1" 200 OK
ERROR 07-24 22:01:26 [async_llm.py:419] AsyncLLM output_handler failed.
ERROR 07-24 22:01:26 [async_llm.py:419] Traceback (most recent call last):
ERROR 07-24 22:01:26 [async_llm.py:419]   File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 396, in output_handler
ERROR 07-24 22:01:26 [async_llm.py:419]     processed_outputs = output_processor.process_outputs(
ERROR 07-24 22:01:26 [async_llm.py:419]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-24 22:01:26 [async_llm.py:419]   File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/output_processor.py", line 398, in process_outputs
ERROR 07-24 22:01:26 [async_llm.py:419]     stop_string = req_state.detokenizer.update(
ERROR 07-24 22:01:26 [async_llm.py:419]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-24 22:01:26 [async_llm.py:419]   File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/detokenizer.py", line 117, in update
ERROR 07-24 22:01:26 [async_llm.py:419]     self.output_text += self.decode_next(new_token_id)
ERROR 07-24 22:01:26 [async_llm.py:419]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-24 22:01:26 [async_llm.py:419]   File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/detokenizer.py", line 216, in decode_next
ERROR 07-24 22:01:26 [async_llm.py:419]     token = self._protected_step(next_token_id)
ERROR 07-24 22:01:26 [async_llm.py:419]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-24 22:01:26 [async_llm.py:419]   File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
ERROR 07-24 22:01:26 [async_llm.py:419]     raise e
ERROR 07-24 22:01:26 [async_llm.py:419]   File "/opt/app-root/lib64/python3.12/site-packages/vllm/v1/engine/detokenizer.py", line 230, in _protected_step
ERROR 07-24 22:01:26 [async_llm.py:419]     token = self.stream.step(self.tokenizer, next_token_id)
ERROR 07-24 22:01:26 [async_llm.py:419]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 07-24 22:01:26 [async_llm.py:419] OverflowError: out of range integral type conversion attempted
WARNING 07-24 22:01:26 [protocol.py:58] The following fields were present in the request but ignored: {'max_completion_tokens'} 

model launch args are:

    - --model=RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic
    - --port=8080
    - --max-model-len=8192
    - --uvicorn-log-level=debug
    - --trust-remote-code
    - --tensor-parallel-size=4
    - --gpu-memory-utilization=0.92
    - --no-enable-prefix-caching
    - --disable-log-requests
    - --kv-cache-dtype=fp8 

Used the guidellm to run the performance sweep:

guidellm benchmark --target '<openai_endpoint>' \
         --model RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
         --processor RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
         --data='{"prompt_tokens":512, "prompt_tokens_stdev":128, "prompt_tokens_min":256, "prompt_tokens_max":1024, "output_tokens":2048, "output_tokens_stdev":512, "output_tokens_min":1024, "output_tokens_max":3072}'  \
         --rate-type concurrent --rate 5,25,100,500 \
         --max-seconds 300 \
         --output-path output.json

[david6666666]

I can try it.

[njhill]

Thanks @david6666666! I guess first step would be to reproduce ... seems most likely some kind of overflow resulting in a negative token id being produced somehow.

[david6666666]

Thanks @david6666666! I guess first step would be to reproduce ... seems most likely some kind of overflow resulting in a negative token id being produced somehow.

Yes, I will check the env info of the issue and the latest main branch code to see if it can be reproduce.

[molowny]

I have the same for RedHatAI/Llama-4-Scout-17B-16E-Instruct-FP8-dynamic

[molowny]

From my further observations, it can fail with this error even without using --kv-cache-dtype=fp8

[BruceW-07]

I can't seem to reproduce this issue neither with the latest main code nor with vllm=0.9.2 and transformers=4.52.4

[molowny]

Based on my observation, it fails randomly after a few minutes of moderate inference

[molowny]

ERROR 08-01 00:15:16 [core.py:634] EngineCore encountered a fatal error.
ERROR 08-01 00:15:16 [core.py:634] Traceback (most recent call last):
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 625, in run_engine_core
ERROR 08-01 00:15:16 [core.py:634]     engine_core.run_busy_loop()
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 652, in run_busy_loop
ERROR 08-01 00:15:16 [core.py:634]     self._process_engine_step()
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 677, in _process_engine_step
ERROR 08-01 00:15:16 [core.py:634]     outputs, model_executed = self.step_fn()
ERROR 08-01 00:15:16 [core.py:634]                               ^^^^^^^^^^^^^^
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 266, in step
ERROR 08-01 00:15:16 [core.py:634]     scheduler_output = self.scheduler.schedule()
ERROR 08-01 00:15:16 [core.py:634]                        ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 440, in schedule
ERROR 08-01 00:15:16 [core.py:634]     new_blocks = self.kv_cache_manager.allocate_slots(
ERROR 08-01 00:15:16 [core.py:634]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 302, in allocate_slots
ERROR 08-01 00:15:16 [core.py:634]     self.coordinator.cache_blocks(
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 113, in cache_blocks
ERROR 08-01 00:15:16 [core.py:634]     manager.cache_blocks(request, block_hashes, num_computed_tokens)
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/single_type_kv_cache_manager.py", line 146, in cache_blocks
ERROR 08-01 00:15:16 [core.py:634]     self.block_pool.cache_full_blocks(
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/block_pool.py", line 138, in cache_full_blocks
ERROR 08-01 00:15:16 [core.py:634]     assert prev_block.block_hash is not None
ERROR 08-01 00:15:16 [core.py:634]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-01 00:15:16 [core.py:634] AssertionError
ERROR 08-01 00:15:16 [async_llm.py:416] AsyncLLM output_handler failed.
ERROR 08-01 00:15:16 [async_llm.py:416] Traceback (most recent call last):
ERROR 08-01 00:15:16 [async_llm.py:416]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 375, in output_handler
ERROR 08-01 00:15:16 [async_llm.py:416]     outputs = await engine_core.get_output_async()
ERROR 08-01 00:15:16 [async_llm.py:416]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-01 00:15:16 [async_llm.py:416]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 751, in get_output_async
ERROR 08-01 00:15:16 [async_llm.py:416]     raise self._format_exception(outputs) from None
ERROR 08-01 00:15:16 [async_llm.py:416] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

[david6666666]

@molowny can you provide your collect_env and startup parameters?

[molowny]

vllm serve RedHatAI/Llama-3.3-70B-Instruct-FP8-dynamic \
    --port 8080 \
    --tensor-parallel-size 2 \
    --max-model-len 50000 \
    --max-num-batched-tokens 10000 \
    --max-num-seqs 24 \
    --gpu-memory-utilization 0.95 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --disable-log-requests \
    --enable-prompt-tokens-details \
    --enable-auto-tool-choice \
    --tool-call-parser llama3_json

Collecting environment information...
==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 4.0.3
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.7.0+cu126
Is debug build               : False
CUDA used to build PyTorch   : 12.6
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.11 (main, Jun  4 2025, 08:56:18) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-5.15.0-144-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : 
GPU 0: NVIDIA H100 PCIe
GPU 1: NVIDIA H100 PCIe

Nvidia driver version        : 570.169
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           52 bits physical, 57 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  48
On-line CPU(s) list:                     0-47
Vendor ID:                               AuthenticAMD
Model name:                              AMD EPYC 9334 32-Core Processor
CPU family:                              25
Model:                                   17
Thread(s) per core:                      1
Core(s) per socket:                      48
Socket(s):                               1
Stepping:                                1
BogoMIPS:                                5391.88
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save tsc_scale vmcb_clean pausefilter pfthreshold v_vmsave_vmload vgif avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid fsrm arch_capabilities
Virtualization:                          AMD-V
Hypervisor vendor:                       KVM
Virtualization type:                     full
L1d cache:                               3 MiB (48 instances)
L1i cache:                               3 MiB (48 instances)
L2 cache:                                24 MiB (48 instances)
L3 cache:                                768 MiB (48 instances)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-47
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Mitigation; safe RET, no microcode
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                     Not affected
Vulnerability Tsx async abort:           Not affected

==============================
Versions of relevant libraries
==============================
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.6.4.1
[pip3] nvidia-cuda-cupti-cu12==12.6.80
[pip3] nvidia-cuda-nvrtc-cu12==12.6.77
[pip3] nvidia-cuda-runtime-cu12==12.6.77
[pip3] nvidia-cudnn-cu12==9.5.1.17
[pip3] nvidia-cufft-cu12==11.3.0.4
[pip3] nvidia-cufile-cu12==1.11.1.6
[pip3] nvidia-curand-cu12==10.3.7.77
[pip3] nvidia-cusolver-cu12==11.7.1.2
[pip3] nvidia-cusparse-cu12==12.5.4.2
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-ml-py==12.575.51
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.6.85
[pip3] nvidia-nvshmem-cu12==3.3.9
[pip3] nvidia-nvtx-cu12==12.6.77
[pip3] pynvml==12.0.0
[pip3] pyzmq==27.0.0
[pip3] torch==2.7.0
[pip3] torchaudio==2.7.1+cu128
[pip3] torchvision==0.22.0
[pip3] transformers==4.53.3
[pip3] triton==3.3.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
Neuron SDK Version           : N/A
vLLM Version                 : 0.10.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
  	GPU0	GPU1	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	X 	PHB	0-47	0		N/A
GPU1	PHB	X 	0-47	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NCCL_VERSION=2.25.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.8.1
VLLM_DISABLE_COMPILE_CACHE=1
LD_LIBRARY_PATH=/usr/local/cuda/lib64
VLLM_USE_V1=1
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

[david6666666]

Still can't reproduce

[hongqing1986]

i have the same for Qwen3-32B:

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 327, in generate
    out = q.get_nowait() or await q.get()
                                  ^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 52, in get
    raise output
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 397, in output_handler
    processed_outputs = output_processor.process_outputs(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 351, in process_outputs
    stop_string = req_state.detokenizer.update(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/detokenizer.py", line 107, in update
    self.output_text += self.decode_next(new_token_id)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/detokenizer.py", line 202, in decode_next
    token = self.stream.step(self.tokenizer, next_token_id)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OverflowError: out of range integral type conversion attempted

[howdylovely]

ERROR 08-08 06:18:35 [serving_chat.py:948] Traceback (most recent call last):
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 326, in generate
ERROR 08-08 06:18:35 [serving_chat.py:948]     out = q.get_nowait() or await q.get()
ERROR 08-08 06:18:35 [serving_chat.py:948]                             ^^^^^^^^^^^^^
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 57, in get
ERROR 08-08 06:18:35 [serving_chat.py:948]     raise output
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 326, in generate
ERROR 08-08 06:18:35 [serving_chat.py:948]     out = q.get_nowait() or await q.get()
ERROR 08-08 06:18:35 [serving_chat.py:948]                             ^^^^^^^^^^^^^
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 57, in get
ERROR 08-08 06:18:35 [serving_chat.py:948]     raise output
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 326, in generate
ERROR 08-08 06:18:35 [serving_chat.py:948]     out = q.get_nowait() or await q.get()
ERROR 08-08 06:18:35 [serving_chat.py:948]                             ^^^^^^^^^^^^^
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 57, in get
ERROR 08-08 06:18:35 [serving_chat.py:948]     raise output
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 326, in generate
ERROR 08-08 06:18:35 [serving_chat.py:948]     out = q.get_nowait() or await q.get()
ERROR 08-08 06:18:35 [serving_chat.py:948]                             ^^^^^^^^^^^^^
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 57, in get
ERROR 08-08 06:18:35 [serving_chat.py:948]     raise output
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 326, in generate
ERROR 08-08 06:18:35 [serving_chat.py:948]     out = q.get_nowait() or await q.get()
ERROR 08-08 06:18:35 [serving_chat.py:948]                             ^^^^^^^^^^^^^
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 57, in get
ERROR 08-08 06:18:35 [serving_chat.py:948]     raise output
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 396, in output_handler
ERROR 08-08 06:18:35 [serving_chat.py:948]     processed_outputs = output_processor.process_outputs(
ERROR 08-08 06:18:35 [serving_chat.py:948]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 398, in process_outputs
ERROR 08-08 06:18:35 [serving_chat.py:948]     stop_string = req_state.detokenizer.update(
ERROR 08-08 06:18:35 [serving_chat.py:948]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/detokenizer.py", line 117, in update
ERROR 08-08 06:18:35 [serving_chat.py:948]     self.output_text += self.decode_next(new_token_id)
ERROR 08-08 06:18:35 [serving_chat.py:948]                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/detokenizer.py", line 216, in decode_next
ERROR 08-08 06:18:35 [serving_chat.py:948]     token = self._protected_step(next_token_id)
ERROR 08-08 06:18:35 [serving_chat.py:948]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/detokenizer.py", line 233, in _protected_step
ERROR 08-08 06:18:35 [serving_chat.py:948]     raise e
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/detokenizer.py", line 230, in _protected_step
ERROR 08-08 06:18:35 [serving_chat.py:948]     token = self.stream.step(self.tokenizer, next_token_id)
ERROR 08-08 06:18:35 [serving_chat.py:948]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-08 06:18:35 [serving_chat.py:948] OverflowError: out of range integral type conversion attempted
ERROR 08-08 06:18:35 [serving_chat.py:948] 
ERROR 08-08 06:18:35 [serving_chat.py:948] The above exception was the direct cause of the following exception:
ERROR 08-08 06:18:35 [serving_chat.py:948] 
ERROR 08-08 06:18:35 [serving_chat.py:948] Traceback (most recent call last):
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 518, in chat_completion_stream_generator
ERROR 08-08 06:18:35 [serving_chat.py:948]     async for res in result_generator:
ERROR 08-08 06:18:35 [serving_chat.py:948]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 359, in generate
ERROR 08-08 06:18:35 [serving_chat.py:948]     raise EngineGenerateError() from e
ERROR 08-08 06:18:35 [serving_chat.py:948] vllm.v1.engine.exceptions.EngineGenerateError

[njhill]

ERROR 08-01 00:15:16 [core.py:634] EngineCore encountered a fatal error.
ERROR 08-01 00:15:16 [core.py:634] Traceback (most recent call last):
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 625, in run_engine_core
ERROR 08-01 00:15:16 [core.py:634]     engine_core.run_busy_loop()
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 652, in run_busy_loop
ERROR 08-01 00:15:16 [core.py:634]     self._process_engine_step()
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 677, in _process_engine_step
ERROR 08-01 00:15:16 [core.py:634]     outputs, model_executed = self.step_fn()
ERROR 08-01 00:15:16 [core.py:634]                               ^^^^^^^^^^^^^^
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 266, in step
ERROR 08-01 00:15:16 [core.py:634]     scheduler_output = self.scheduler.schedule()
ERROR 08-01 00:15:16 [core.py:634]                        ^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 440, in schedule
ERROR 08-01 00:15:16 [core.py:634]     new_blocks = self.kv_cache_manager.allocate_slots(
ERROR 08-01 00:15:16 [core.py:634]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 302, in allocate_slots
ERROR 08-01 00:15:16 [core.py:634]     self.coordinator.cache_blocks(
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 113, in cache_blocks
ERROR 08-01 00:15:16 [core.py:634]     manager.cache_blocks(request, block_hashes, num_computed_tokens)
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/single_type_kv_cache_manager.py", line 146, in cache_blocks
ERROR 08-01 00:15:16 [core.py:634]     self.block_pool.cache_full_blocks(
ERROR 08-01 00:15:16 [core.py:634]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/block_pool.py", line 138, in cache_full_blocks
ERROR 08-01 00:15:16 [core.py:634]     assert prev_block.block_hash is not None
ERROR 08-01 00:15:16 [core.py:634]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-01 00:15:16 [core.py:634] AssertionError
ERROR 08-01 00:15:16 [async_llm.py:416] AsyncLLM output_handler failed.
ERROR 08-01 00:15:16 [async_llm.py:416] Traceback (most recent call last):
ERROR 08-01 00:15:16 [async_llm.py:416]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 375, in output_handler
ERROR 08-01 00:15:16 [async_llm.py:416]     outputs = await engine_core.get_output_async()
ERROR 08-01 00:15:16 [async_llm.py:416]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 08-01 00:15:16 [async_llm.py:416]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 751, in get_output_async
ERROR 08-01 00:15:16 [async_llm.py:416]     raise self._format_exception(outputs) from None
ERROR 08-01 00:15:16 [async_llm.py:416] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

@molowny this looks like a different error. Do you mean that this is occurring in the same run before or after the other error?