[Fix] Strip use_vllm/use_verifier eval flags from judge kwargs (OpenAI 400) - #1611
Open
cdpark92 wants to merge 1 commit into
Open
[Fix] Strip use_vllm/use_verifier eval flags from judge kwargs (OpenAI 400)#1611cdpark92 wants to merge 1 commit into
cdpark92 wants to merge 1 commit into
Conversation
run.py forwards --use-vllm and --use-verifier into judge_kwargs; build_judge passed them through to the API wrappers, which place unknown kwargs in the request body. The real OpenAI API rejects such payloads with 400 'Unrecognized request argument supplied: use_vllm', so any judge-scored benchmark (MMVet, MathVista, MCQ answer extraction, ...) fails when a model is evaluated with --use-vllm against an OpenAI judge. Local vLLM judges ignore unknown fields, which is why this went unnoticed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
run.pyforwards the--use-vllmand--use-verifierCLI flags intojudge_kwargs:build_judge(**judge_kwargs)only popsmodelandnproc; everything elsereaches the API wrapper, and
OpenAIWrapper.generate_innerplaces unknownkwargs directly into the request body:
The real OpenAI API rejects unknown body fields:
Since even the
working()health probe sends this payload, every judge-scoredevaluation fails up front (MMVet crashes on its
assert model.working();MathVista reports 100% judge failure; MCQ answer extraction degrades) whenever
a model is evaluated with
--use-vllmand an OpenAI judge.vLLM-served local judges ignore unknown request fields, which is why the bug
is easy to miss when developing against a self-hosted judge.
Fix
Pop the two eval-only flags in
build_judge()before constructing the APIwrapper (+3 lines). This covers every
build_judge(**judge_kwargs)call siteat once. The flags remain visible to the dataset
evaluate()dispatch, whichreads them from
judge_kwargsbeforebuild_judgeis called —**expansionpasses a copy, so callers are unaffected.
Reproduction