[Feature] Opt-in LLM-judge rescoring for string-metric VQA benchmarks (ChartQA/DocVQA/InfoVQA/TextVQA/OCRBench)

[cdpark92]

Motivation

ChartQA / DocVQA / InfoVQA / TextVQA / OCRBench are scored with strict string /
numeric metrics (relaxed accuracy, ANLS, VQA score, substring matching). These
metrics mark answers wrong when the model's answer is semantically correct but
formatted differently from the ground truth. Real cases from a
LiquidAI/LFM2.5-VL-450M run with this repo (ground truth vs model answer, all
scored 0 by the strict metric):

Benchmark	Ground truth	Model answer	Failure kind
ChartQA	977633	977,633	thousands separator
ChartQA	1.4	1.4 billion	magnitude word appended
ChartQA	1	1 GPI	unit suffix from the chart
InfoVQA	27%	27 percent	percent sign vs word
TextVQA	10 years	10	unit word
OCRBench	begins.	begins	trailing punctuation
OCRBench	$161,886	161,886	currency symbol

How many points this costs depends on each model's answering style, so it
distorts cross-model comparisons rather than adding a constant offset.

MMVet and MathVista already rely on LLM judges. This PR adds an optional
semantic-equivalence pass to the five string-metric benchmarks — without
changing any default behavior or any existing number.

Design

Opt-in gate. These datasets are assigned no judge model by default
(get_judge_kwargs), so without --judge <model> nothing changes — the
evaluation path and outputs are identical to main. The rescoring runs only
when a judge is explicitly configured:

if (listinstr(['TextVQA', 'ChartQA', 'DocVQA', 'InfoVQA'], self.dataset_name)
        and judge_kwargs.get('model')
        and judge_kwargs.get('model') != 'exact_matching'):
    return self.evaluate_with_llm_judge(eval_file, **judge_kwargs)
return self.evaluate_heuristic(eval_file, **judge_kwargs)

Strict scores are reported unchanged. The strict metric is computed with
the exact same process_line / hit_calculate aggregation as
evaluate_heuristic, and the judge can never demote a strict-passed sample.
Judged results are appended as <split>_judge columns (OCRBench:
<category>_judge keys and Final Score_judge in the score JSON), so the
output is a strict superset:

# acc.csv on main, or on this branch without --judge
"test_augmented","test_human","Overall"
"90.16","59.92","75.04"

# this branch, with --judge
"test_augmented","test_human","Overall","test_augmented_judge","test_human_judge","Overall_judge"
"90.16","59.92","75.04","90.4","62.88","76.64"

Judge cost scales with strict failures only. Only strict-failed samples
(per-sample score < 0.5) are sent to the judge; verdicts are cached in a
resumable pkl like the other judge-based evaluators.

Model-agnostic, dataset-specific prompts (utils/llm_judge_vqa.py): the
equivalence rules are written down per dataset — percent/decimal and
magnitude-word variants, unit suffixes, thousands separators, ~5% numeric
tolerance for ChartQA (consistent with relaxed accuracy), answer-containment
for OCRBench. The judge replies with a single 1/0 token; a judge parsing
failure counts as wrong, so judge flakiness can never inflate scores.

Results on a public model

LiquidAI/LFM2.5-VL-450M, predictions generated with this repo, judge =
Qwen3.6-27B-FP8 served locally (thinking disabled):

Benchmark	Strict (unchanged)	+LLM judge	Δ
ChartQA_TEST	75.04	76.64	+1.60
DocVQA_VAL	78.34	82.73	+4.39
InfoVQA_VAL	43.30	48.80	+5.50
TextVQA_VAL	69.82	78.64	+8.82
OCRBench	679	710	+31 (/1000)

Verification

• Without --judge: the evaluate path is unchanged, outputs are identical to
  main for all five benchmarks.
• With --judge: strict columns match the heuristic run exactly — verified on
  the LFM2.5-VL-450M outputs above at full float precision (e.g. DocVQA
  78.33964128700651); only the *_judge columns are added.
• pre-commit run passes on the changed files.

Notes

Judged numbers naturally depend on the chosen judge model, so the strict
columns remain the primary metric and the judge columns are supplemental.
Happy to adjust the gating (e.g. a dedicated flag instead of judge presence),
column naming, or prompt wording if maintainers prefer.