[Bug]: Health endpoint not responsive while preprocessing a chat request (and other requests types too probably)

[scwgoire]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version                : Could not collect
CMake version                : version 4.1.0
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.7.1+cu128
Is debug build               : False
CUDA used to build PyTorch   : 12.8
ROCM used to build PyTorch   : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.11 (main, Jun  4 2025, 08:56:18) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.8.0-63-generic-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : True
CUDA runtime version         : 12.8.93
CUDA_MODULE_LOADING set to   : LAZY
GPU models and configuration : GPU 0: NVIDIA H100 PCIe
Nvidia driver version        : 570.169
cuDNN version                : Could not collect
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        52 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               24
On-line CPU(s) list:                  0-23
Vendor ID:                            AuthenticAMD
Model name:                           AMD EPYC 9334 32-Core Processor
CPU family:                           25
Model:                                17
Thread(s) per core:                   1
Core(s) per socket:                   24
Socket(s):                            1
Stepping:                             1
BogoMIPS:                             5391.90
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save tsc_scale vmcb_clean pausefilter pfthreshold v_vmsave_vmload vgif avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq la57 rdpid fsrm arch_capabilities
Virtualization:                       AMD-V
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            1.5 MiB (24 instances)
L1i cache:                            1.5 MiB (24 instances)
L2 cache:                             12 MiB (24 instances)
L3 cache:                             384 MiB (24 instances)
NUMA node(s):                         1
NUMA node0 CPU(s):                    0-23
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Vulnerable: Safe RET, no microcode
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP disabled; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.2.11
[pip3] numpy==2.2.6
[pip3] nvidia-cublas-cu12==12.8.3.14
[pip3] nvidia-cuda-cupti-cu12==12.8.57
[pip3] nvidia-cuda-nvrtc-cu12==12.8.61
[pip3] nvidia-cuda-runtime-cu12==12.8.57
[pip3] nvidia-cudnn-cu12==9.7.1.26
[pip3] nvidia-cufft-cu12==11.3.3.41
[pip3] nvidia-cufile-cu12==1.13.0.11
[pip3] nvidia-curand-cu12==10.3.9.55
[pip3] nvidia-cusolver-cu12==11.7.2.55
[pip3] nvidia-cusparse-cu12==12.5.7.53
[pip3] nvidia-cusparselt-cu12==0.6.3
[pip3] nvidia-nccl-cu12==2.26.2
[pip3] nvidia-nvjitlink-cu12==12.8.61
[pip3] nvidia-nvtx-cu12==12.8.55
[pip3] pyzmq==27.0.1
[pip3] torch==2.7.1+cu128
[pip3] torchaudio==2.7.1+cu128
[pip3] torchvision==0.22.1+cu128
[pip3] transformers==4.55.2
[pip3] triton==3.3.1
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.10.1.1
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled
GPU Topology:
  	GPU0	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	0-23	0		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

==============================
     Environment Variables
==============================
NVIDIA_VISIBLE_DEVICES=all
NVIDIA_REQUIRE_CUDA=cuda>=12.8 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551 brand=unknown,driver>=560,driver<561 brand=grid,driver>=560,driver<561 brand=tesla,driver>=560,driver<561 brand=nvidia,driver>=560,driver<561 brand=quadro,driver>=560,driver<561 brand=quadrortx,driver>=560,driver<561 brand=nvidiartx,driver>=560,driver<561 brand=vapps,driver>=560,driver<561 brand=vpc,driver>=560,driver<561 brand=vcs,driver>=560,driver<561 brand=vws,driver>=560,driver<561 brand=cloudgaming,driver>=560,driver<561 brand=unknown,driver>=565,driver<566 brand=grid,driver>=565,driver<566 brand=tesla,driver>=565,driver<566 brand=nvidia,driver>=565,driver<566 brand=quadro,driver>=565,driver<566 brand=quadrortx,driver>=565,driver<566 brand=nvidiartx,driver>=565,driver<566 brand=vapps,driver>=565,driver<566 brand=vpc,driver>=565,driver<566 brand=vcs,driver>=565,driver<566 brand=vws,driver>=565,driver<566 brand=cloudgaming,driver>=565,driver<566
NCCL_VERSION=2.25.1-1
NVIDIA_DRIVER_CAPABILITIES=compute,utility
NVIDIA_PRODUCT_NAME=CUDA
VLLM_USAGE_SOURCE=production-docker-image
CUDA_VERSION=12.8.1
LD_LIBRARY_PATH=/usr/local/cuda/lib64
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

🐛 Describe the bug

Here is a software pattern problem rather than a bug per-se. I couldn't find an open issue about it.

It looks like HTTP Requests handling is mono-threaded. I stumbled upon a 439KB request which takes 53 seconds to preprocess (specifically, applying a mistral chat template with MistralTokenizer.mistral.encode_chat_completion).

Usually, the /health endpoint takes ~10ms max to reply so I expect most Kubernetes liveness probes will be configured with a 3x 1-10s timeout. 53 seconds of /health HTTP endpoint unresponsiveness is enough to have the Pod terminated under these conditions.

Because preprocessing performance is not 100% under vLLM control due to 3rd party libraries, it is unclear to me what is our best option here, here are a few ideas:

• make the HTTP server multithreaded: most probably very complex to implement downstream, I guess request preprocessing hasn't been coded thread-safe if there was no use for it
• open a second HTTP server for health and metrics (on its own thread, provided we can easily make health/metrics code thread-safe)
• work with 3rd party library providers so they improve their code performance (the CPU used is a very fast one, there definitely is room for improvement there)
• enforce a preprocessing timeout

What do you think ?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

cc @patrickvonplaten

[DarkLight1337]

I think we can try to run the Mistral tokenizer on a separate thread.

[DarkLight1337]

Not sure if HF tokenizer works well with multithreading though. cc @njhill

[bbrowning]

This looks like a case where we're running blocking operations on the async event loop, at least from my reading of the code. Normally you'd fix this by doing the blocking operations in another thread (via asyncio.to_thread or similar), but as was pointed out it's not clear if all HuggingFace Tokenizers are actually threadsafe. The thread at huggingface/tokenizers#1726 definitely gives me pause, given that we're passing around instances of tokenizers here so would likely end up sharing them across threads.

We could potentially wrap the tokenizer early in its lifecycle, such as:

vllm/vllm/transformers_utils/tokenizer.py

Line 277 in f708bd4

def init_tokenizer_from_configs(model_config: ModelConfig):

There we could wrap this in a new class that implements TokenizerBase and wraps each call with thread locking, which would allow us to pass the tokenizer around to different threads while ensuring we don't run into any thread safety issues. That feels a bit like a hack to me, and it would obviously be better if we had native async versions of all these tokenizer methods or a guarantee of thread safety in tokenizers.

[bbrowning]

Just another note that things like huggingface/tokenizers#1843 exist that were meant to make some of this easier to do with native async bindings, but we have a few layers of abstraction here between the incoming requests and the raw tokenizer encode methods that, at least at first glance, would make it challenging to directly use those async_encode methods.

[bbrowning]

@scwgoire Is this only an issue when using the Mistral tokenizers? Or have you been able to reproduce this with non-Mistral tokenizers? There's some logic for other tokenizers that should be doing at least some of this work in an async fashion. But, the Mistral tokenizers go down a different code path.

[scwgoire]

Thanks for the followup Ben !

I just made a few tries with fresh new requests with oversized requests to emphasize the problem and make sure the reply is the result of the preprocessing only. Obviously those requests fail with a 400 http code saying the request has too many input tokens.

I used a mistralai/Mistral-7B-Instruct-v0.3 to perform the tests.

Here are the results:

tokenizer_mode	request size	request duration	health probe freeze duration
mistral	5.4MB	4.9s	4.8s
mistral	27M	28.9s	28.8s
(none)	5.4MB	4.1s	< 100 ms
(none)	27M	22.4	< 100 ms

( < 100 ms indicates I couldn't quickly locate a "timestamp gap" in my terminal output)

Those results are intriguing. I can't explain why the request duration is long but doesn't freeze the health probe in the case of the non-mistral tokenizer.

[bbrowning]

@scwgoire Great, thank you for that testing, as it confirms the issue is specific to how we're using the mistral tokenizers.

The reason the request duration is long but does not freeze the health probe in the non-mistral case is because mistral tokenizers get handled differently during pre-processing than non-mistral tokenizers, due to some special logic the mistral ones do in their library. The non-mistral code path does proper async handling of the request (async as in async Python with coroutines, async/await keywords). This means the non-mistral tokenizers can do their work without blocking other requests.

However, the mistral tokenizer code path does its long-running operations in a synchronous way that blocks the event loop, instead of putting those synchronous operations in a background thread to keep the event loop moving. That means it ends up blocking other requests while doing its work.

These two different code paths can be seen starting here in serving_engine.py:

vllm/vllm/entrypoints/openai/serving_engine.py

Line 736 in be22bb6

async def _preprocess_chat(

I'm leaving more details and notes here in case someone else wants to jump in and get this fixed up. Otherwise, I'll see what I can do after finishing some other things up I currently have in-progress.

[bbrowning]

I had some spare time, so took a pass at fixing this in the auto-linked PR above this comment.