Skip to content

[Fix] Strip use_vllm/use_verifier eval flags from judge kwargs (OpenAI 400) - #1611

Open
cdpark92 wants to merge 1 commit into
open-compass:mainfrom
cdpark92:fix-judge-kwargs-leak
Open

[Fix] Strip use_vllm/use_verifier eval flags from judge kwargs (OpenAI 400)#1611
cdpark92 wants to merge 1 commit into
open-compass:mainfrom
cdpark92:fix-judge-kwargs-leak

Conversation

@cdpark92

Copy link
Copy Markdown

Bug

run.py forwards the --use-vllm and --use-verifier CLI flags into
judge_kwargs:

# run.py, get_judge_kwargs
if args.use_verifier:
    judge_kwargs['use_verifier'] = True
if args.use_vllm:
    judge_kwargs['use_vllm'] = True

build_judge(**judge_kwargs) only pops model and nproc; everything else
reaches the API wrapper, and OpenAIWrapper.generate_inner places unknown
kwargs directly into the request body:

payload = dict(model=self.model, messages=input_msgs, n=1,
               temperature=temperature, **kwargs)

The real OpenAI API rejects unknown body fields:

400 - Unrecognized request argument supplied: use_vllm

Since even the working() health probe sends this payload, every judge-scored
evaluation fails up front (MMVet crashes on its assert model.working();
MathVista reports 100% judge failure; MCQ answer extraction degrades) whenever
a model is evaluated with --use-vllm and an OpenAI judge.

vLLM-served local judges ignore unknown request fields, which is why the bug
is easy to miss when developing against a self-hosted judge.

Fix

Pop the two eval-only flags in build_judge() before constructing the API
wrapper (+3 lines). This covers every build_judge(**judge_kwargs) call site
at once. The flags remain visible to the dataset evaluate() dispatch, which
reads them from judge_kwargs before build_judge is called — ** expansion
passes a copy, so callers are unaffected.

Reproduction

# any judge-scored benchmark + --use-vllm + a real OpenAI key
python run.py --data MMVet --model <any-vllm-capable-model> --use-vllm
# before: AssertionError: MMVet evaluation requires a working OPENAI API
#         (judge probe rejected with 400 'Unrecognized request argument supplied: use_vllm')
# after: judge calls succeed

run.py forwards --use-vllm and --use-verifier into judge_kwargs; build_judge
passed them through to the API wrappers, which place unknown kwargs in the
request body. The real OpenAI API rejects such payloads with 400
'Unrecognized request argument supplied: use_vllm', so any judge-scored
benchmark (MMVet, MathVista, MCQ answer extraction, ...) fails when a model
is evaluated with --use-vllm against an OpenAI judge. Local vLLM judges
ignore unknown fields, which is why this went unnoticed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant