[Bug]: gemma3 shows degraded accuracy in vLLM v0.8.4

[llsj14]

Your current environment

• vLLM version: v0.8.4
• docker image: nvcr.io/nvidia/pytorch/23.10-py3
• install script

pip install -r ./requirements/build.txt
pip install -r ./requirements/common.txt
pip install -r ./requirements/cuda.txt
pip install flash_attn==2.7.4.post1
export VLLM_COMMIT=dc1b4a6f1300003ae27f033afbdff5e2683721ce
export VLLM_PRECOMPILED_WHEEL_LOCATION=https://wheels.vllm.ai/${VLLM_COMMIT}/vllm-1.0.0.dev-cp38-abi3-manylinux1_x86_64.whl
pip install -e .
pip install -U pynvml

Click to expand environment info

Collecting environment information... PyTorch version: 2.6.0+cu124 Is debug build: False CUDA used to build PyTorch: 12.4 ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 4.0.0
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-4.19.93-1.nbp.el7.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.2.140
CUDA_MODULE_LOADING set to: LAZY

Nvidia driver version: 535.129.03
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.5
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==2.2.5
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-dali-cuda120==1.30.0
[pip3] nvidia-ml-py==12.570.86
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] nvidia-pyindex==1.0.9
[pip3] onnx==1.14.0
[pip3] pynvml==12.0.0
[pip3] pytorch-quantization==2.1.2
[pip3] pyzmq==26.4.0
[pip3] torch==2.6.0
[pip3] torch-tensorrt==0.0.0
[pip3] torchaudio==2.6.0
[pip3] torchdata==0.7.0a0
[pip3] torchtext==0.16.0a0
[pip3] torchvision==0.21.0
[pip3] transformers==4.51.3
[pip3] triton==3.2.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.4
vLLM Build Flags:
CUDA Archs: 5.2 6.0 6.1 7.0 7.2 7.5 8.0 8.6 8.7 9.0+PTX; ROCm: Disabled; Neuron: Disabled

NVIDIA_VISIBLE_DEVICES=all
CUBLAS_VERSION=12.2.5.6
NVIDIA_REQUIRE_CUDA=cuda>=9.0
CUDA_CACHE_DISABLE=1
TORCH_CUDA_ARCH_LIST=5.2 6.0 6.1 7.0 7.2 7.5 8.0 8.6 8.7 9.0+PTX
NCCL_VERSION=2.19.3
NVIDIA_DRIVER_CAPABILITIES=compute,utility,video
NVIDIA_PRODUCT_NAME=PyTorch
CUDA_VERSION=12.2.2.009
PYTORCH_VERSION=2.1.0a0+32f93b1
PYTORCH_BUILD_NUMBER=0
CUDNN_VERSION=8.9.5.29
PYTORCH_HOME=/opt/pytorch/pytorch
LD_LIBRARY_PATH=/usr/local/lib/python3.10/dist-packages/torch/lib:/usr/local/lib/python3.10/dist-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
NVIDIA_BUILD_ID=71422337
CUDA_DRIVER_VERSION=535.104.05
PYTORCH_BUILD_VERSION=2.1.0a0+32f93b1
CUDA_HOME=/usr/local/cuda
CUDA_HOME=/usr/local/cuda
CUDA_MODULE_LOADING=LAZY
NVIDIA_REQUIRE_JETPACK_HOST_MOUNTS=
NVIDIA_PYTORCH_VERSION=23.10
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE=1
NCCL_CUMEM_ENABLE=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

🐛 Describe the bug

I'm testing open-source models using vLLM v0.8.4 and lm-evaluation-harness. However, this version shows degraded accuracy when deploying the Gemma 3 series. There were no issues when testing the Qwen 2.5 model. I'm comparing results from vLLM against those from Hugging Face.

Qwen2.5-72B-Instruct

vLLM result

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.7042	±	0.0046
		none	0	acc_norm	↑	0.8741	±	0.0033
lambada_openai	1	none	0	acc	↑	0.7613	±	0.0059
		none	0	perplexity	↓	2.7697	±	0.0560

HF result

gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 4

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.7031	±	0.0046
		none	0	acc_norm	↑	0.8735	±	0.0033
lambada_openai	1	none	0	acc	↑	0.7518	±	0.0060
		none	0	perplexity	↓	2.7677	±	0.0559

gemma-3-27b-it

vLLM result

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.3863	±	0.0049
		none	0	acc_norm	↑	0.4862	±	0.0050
lambada_openai	1	none	0	acc	↑	0.2251	±	0.0058
		none	0	perplexity	↓	956.6927	±	77.7910

HF resuit

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.6508	±	0.0048
		none	0	acc_norm	↑	0.8405	±	0.0037
lambada_openai	1	none	0	acc	↑	0.6942	±	0.0064
		none	0	perplexity	↓	3.7705	±	0.1152

gemma-3-12b-it

vLLM result

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.3351	±	0.0047
		none	0	acc_norm	↑	0.3999	±	0.0049
lambada_openai	1	none	0	acc	↑	0.2424	±	0.0060
		none	0	perplexity	↓	7298.6355	±	885.6837

HF result

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.6285	±	0.0048
		none	0	acc_norm	↑	0.8187	±	0.0038
lambada_openai	1	none	0	acc	↑	0.6775	±	0.0065
		none	0	perplexity	↓	4.1643	±	0.1289

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Are you using V1 Engine? As mentioned on this page, the attention mask of this model is simplified when using V1 Engine which may lead to some inaccuracies.

[llsj14]

@DarkLight1337
Thank you for your quick advice. I actually tested with the v0 option as well, but it showed the same degraded results.
Below is the script I used to run lm_eval.

export HF_MODEL_PATH="/hugginface/gemma-3-27b-it"
export TP_SIZE=4
export TASKS=lambada_openai,hellaswag
export DTYPE=auto
export BATCH_SIZE=4

# VLLM
VLLM_USE_V1=0 VLLM_WORKER_MULTIPROC_METHOD=spawn CUDA_VISIBLE_DEVICES=0,1,2,3 python -m lm_eval --model=vllm \
     --model_args=pretrained=$HF_MODEL_PATH,tensor_parallel_size=$TP_SIZE,gpu_memory_utilization=0.75,enforce_eager=True,trust_remote_code=True,max_gen_toks=256 \
     --tasks=$TASKS \
     --batch_size=$BATCH_SIZE \
     --output_path=./outputs

[DarkLight1337]

@WoosukKwon any idea?

[airlsyn]

We encountered the same degraded results (almost zero) on gemma-3-27b-it using vLLM version 0.8.5.

[ashgold]

Is this okay for versions of VLLM below V0.8.4?

[airlsyn]

Is this okay for versions of VLLM below V0.8.4?

v0.8.3 works fine for me when evaluating gemma-3-27b-it.

[DarkLight1337]

Can you try whether setting dtype="bfloat16" explicitly fixes the issue?

[llsj14]

I ran the test using the bfloat16 option(gemma-3-27b-it model), but the results appear to be the same.

vllm (pretrained=/hugginface/gemma-3-27b-it,tensor_parallel_size=4,gpu_memory_utilization=0.75,enforce_eager=True,trust_remote_code=True,max_gen_toks=256,dtype=bfloat16), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 4

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.3859	±	0.0049
		none	0	acc_norm	↑	0.4859	±	0.0050
lambada_openai	1	none	0	acc	↑	0.2267	±	0.0058
		none	0	perplexity	↓	947.2092	±	77.1147

[DarkLight1337]

Hmm, can you test the text-only (1B) version of the model? It would greatly narrow down the scope of investigation.

[llsj14]

yeah, I tested gemma-3-1b-it model and the results were as follows.

vLLM result

dtype auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.3512	±	0.0048
		none	0	acc_norm	↑	0.4345	±	0.0049
lambada_openai	1	none	0	acc	↑	0.2307	±	0.0059
		none	0	perplexity	↓	226.0222	±	14.3887

dtype bfloat16

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.3512	±	0.0048
		none	0	acc_norm	↑	0.4345	±	0.0049
lambada_openai	1	none	0	acc	↑	0.2307	±	0.0059
		none	0	perplexity	↓	226.0222	±	14.3887

HF result

dtype auto

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.4348	±	0.0049
		none	0	acc_norm	↑	0.5782	±	0.0049
lambada_openai	1	none	0	acc	↑	0.4343	±	0.0069
		none	0	perplexity	↓	28.4342	±	1.4753

[DarkLight1337]

We didn't change Gemma3 code much recently apart from #17180... @LucasWilkinson any idea what might cause this?

[llsj14]

hello, @airlsyn

We encountered the same degraded results (almost zero) on gemma-3-27b-it using vLLM version 0.8.5.

I'm considering the possibility that my test script might be incorrect, so I wanted to check if you also encountered this issue while using lm-evaluation-harness.

v0.8.3 works fine for me when evaluating gemma-3-27b-it.

I tried to test with v0.8.3, but encountered following issue while running gemma 3 model.

[rank0]: AttributeError: 'Gemma3TextConfig' object has no attribute 'interleaved_sliding_window'
[rank0]:[W507 17:21:20.233029347 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

[LucasWilkinson]

this appears to be ok on main, @llsj14 do you mine testing to see if the issue still exists on the nightly?

|    Tasks     |Version|Filter|n-shot|  Metric  |   |Value |   |Stderr|
|--------------|------:|------|-----:|----------|---|-----:|---|-----:|
|hellaswag     |      1|none  |     0|acc       |↑  |0.6511|±  |0.0048|
|              |       |none  |     0|acc_norm  |↑  |0.8401|±  |0.0037|
|lambada_openai|      1|none  |     0|acc       |↑  |0.7046|±  |0.0064|
|              |       |none  |     0|perplexity|↓  |3.7754|±  |0.1154|

[llsj14]

@LucasWilkinson Thank you for sharing.
I tested again with the most recent vLLM main (last commit ID: 0c0fdae) and the latest lm-evaluation-harness, but the results were still the same, as follows:

vllm (pretrained=/checkpoints/hugginface/gemma-3-27b-it,tensor_parallel_size=4,gpu_memory_utilization=0.8,enforce_eager=True,max_gen_toks=256,dtype=auto), gen_kwargs: (None), limit: None, num_fewshot: None, batch_size: 4

Tasks	Version	Filter	n-shot	Metric		Value		Stderr
hellaswag	1	none	0	acc	↑	0.3862	±	0.0049
		none	0	acc_norm	↑	0.4863	±	0.0050
lambada_openai	1	none	0	acc	↑	0.2249	±	0.0058
		none	0	perplexity	↓	955.5915	±	77.6979

Could you share the version of your evaluation code and the version of flash attention? I updated my lm-evaluation-harness and added the following code to avoid additional errors.

index 76bbe81d..f100ca54 100644
--- a/lm_eval/models/vllm_causallms.py
+++ b/lm_eval/models/vllm_causallms.py
@@ -138,7 +138,14 @@ class VLLM(TemplateLM):
                 "Found 'gemma' in model name, a BOS token will be used as Gemma series models underperform without it."
             )
 
-        if parse_version(version("vllm")) >= parse_version("0.8.3"):
+        if parse_version(version("vllm")) >= parse_version("0.8.4"):
+            self.hf_chat_template = resolve_hf_chat_template(
+                model_config=self._config,
+                tokenizer=self.tokenizer,
+                chat_template=None,
+                tools=None,
+            )
+        elif parse_version(version("vllm")) >= parse_version("0.8.3"):
             self.hf_chat_template = resolve_hf_chat_template(
                 tokenizer=self.tokenizer,
                 chat_template=None,
@@ -486,7 +493,7 @@ class VLLM(TemplateLM):
             inputs = []
             ctxlens = []
             for cache_key, context_enc, continuation_enc in chunk:
-                if full_length := (context_enc + continuation_enc) >= self.max_length:
+                if full_length := len(context_enc + continuation_enc) >= self.max_length:
                     eval_logger.warning(
                         f"Context length {full_length} exceeds max length ({self.max_length}). Truncating context."
                     )