Skip to content

fix(scorers): report an unsupported expected_format as unassessed - #74

Merged
knisar merged 1 commit into
wandb:mainfrom
KunyangZhang:fix/output-format-unsupported
Sep 16, 2026
Merged

knisar merged 1 commit into
wandb:mainfrom
KunyangZhang:fix/output-format-unsupported

Conversation

@KunyangZhang

Copy link
Copy Markdown
Contributor

What changed

OutputFormatScorer reports an unsupported expected_format as an unassessed
row instead of a full pass. Adds offline tests for the scorer.

Why

expected_format is compared case-sensitively against json and xml, and
every other value took this branch:

else:
    return ScorerResult(
        score=1.0,
        passed=True,
        category=self.category,
        explanation=f"Unknown format '{self.expected_format}', skipping check",
    )

So a configuration mistake - expected_format="JSON", "yaml", or a typo -
scored a full pass for any output, including empty and malformed ones, with
assessed=True:

>>> OutputFormatScorer(expected_format="JSON").score("not json at all")
ScorerResult(score=1.0, passed=True, ..., assessed=True)

A row that was never checked then counts as a measured pass in aggregates, which
is the opposite of what the result contract asks for - assessed is "True only
when the scorer produced a real signal ... surface the gap honestly instead of
inflating with neutral defaults". The scorer-authoring guide specifies the
unassessed shape: score=0.0, passed=False, assessed=False, and a stable
details["skipped"] reason. The branch now returns that, and records
scorer_name in details as the guide also asks. The json and xml paths are
unchanged, so no correctly configured scorer changes behaviour.

How it was checked

python -m pytest -q                                         # 332 passed, 15 skipped
python -m pytest tests/test_output_format_scorer.py -q       # 11 passed
python -m ruff check --select E9,F63,F7,F82 .                # All checks passed!
git diff --check                                            # clean

The new tests cover valid JSON, invalid JSON, a missing required key, valid and
invalid XML, and the unsupported-format row including its coverage-gap effect.
Six of them fail on the current main.

Notes:

  • reuse --no-multiprocessing lint was not run: this environment is offline and
    reuse==6.2.0 is not installable from it. The new test file carries the SPDX
    header in the same form as the other test files, with my own name as the
    copyright holder. The ruff available here is 0.16.7 rather than the 0.16.6
    pinned in CI, on the E9,F63,F7,F82 selection.
  • This is independent of fix(scorers): match KeywordToxicityScorer keywords outside other words #73, which fixes false keyword hits in
    KeywordToxicityScorer; both touch programmatic.py, in different places. If
    you would rather review a single change for the module, I can fold them
    together.

This contribution was prepared with AI assistance, as the contributing guide
asks contributors to disclose.

…sessed

`expected_format` is compared case-sensitively against `json` and `xml`, and
every other value returned `score=1.0, passed=True, assessed=True` with the
explanation "Unknown format, skipping check". A configuration mistake -
`expected_format="JSON"`, `"yaml"`, or a typo - therefore scored a full pass for
any output, including empty or malformed ones, and counted as a measured row in
aggregates.

The result contract rules that out ("assessed: True only when the scorer
produced a real signal ... surface the gap honestly instead of inflating with
neutral defaults"), and the scorer-authoring guide asks for `score=0.0`,
`passed=False`, `assessed=False` and a stable `details["skipped"]` reason when a
row cannot be evaluated. The unsupported-format branch now does that, and also
records `scorer_name` in `details` as the contract asks.

The `json` and `xml` paths are unchanged.

Adds offline tests for the scorer: valid and invalid JSON, a missing required
key, valid and invalid XML, and the unsupported-format and coverage-gap
behaviour. Six of them fail on the current main.
@knisar knisar added the status: in review Pull request is ready for maintainer review label Sep 16, 2026

@knisar knisar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 21ee7ea against main 5439934. The unsupported-format branch now returns the documented unassessed result with a stable reason, and the JSON and XML paths are unchanged.

Validation: the core suite passed with 332 passed, 15 skipped. Another 11 review checks passed, covering unsupported formats at thresholds 0, 0.5, and 1 across empty, malformed, and valid JSON output, plus the actual evaluation pipeline. In the pipeline, an unsupported format is counted as a coverage gap and excluded from the measured average; a run containing only unsupported formats has no measured pass rate. All seven CI jobs and both security checks passed.

No blocking findings. Thanks @KunyangZhang for the focused fix and regression coverage.

@knisar
knisar merged commit 9f7091f into wandb:main Sep 16, 2026
9 checks passed
@knisar knisar removed the status: in review Pull request is ready for maintainer review label Sep 16, 2026
KunyangZhang added a commit to KunyangZhang/rai-toolkit that referenced this pull request Sep 20, 2026
The first commit on this branch rewrote `programmatic.py` from a local checkout
that predated wandb#74, so it silently restored the old behaviour where an unsupported
`expected_format` returned `score=1.0`, `passed=True` and `assessed=True`. CI
caught it: six cases in `tests/test_output_format_scorer.py` failed.

This rebuilds the file from current `main` and re-applies only the wandb#79 change, so
the wandb#74 contract (an unsupported format is un-assessed) is intact. The keyword
tests are unchanged and still fail on `main` before the fix.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants