Repository navigation
fix(scorers): report an unsupported expected_format as unassessed - #74
Conversation
…sessed
`expected_format` is compared case-sensitively against `json` and `xml`, and
every other value returned `score=1.0, passed=True, assessed=True` with the
explanation "Unknown format, skipping check". A configuration mistake -
`expected_format="JSON"`, `"yaml"`, or a typo - therefore scored a full pass for
any output, including empty or malformed ones, and counted as a measured row in
aggregates.
The result contract rules that out ("assessed: True only when the scorer
produced a real signal ... surface the gap honestly instead of inflating with
neutral defaults"), and the scorer-authoring guide asks for `score=0.0`,
`passed=False`, `assessed=False` and a stable `details["skipped"]` reason when a
row cannot be evaluated. The unsupported-format branch now does that, and also
records `scorer_name` in `details` as the contract asks.
The `json` and `xml` paths are unchanged.
Adds offline tests for the scorer: valid and invalid JSON, a missing required
key, valid and invalid XML, and the unsupported-format and coverage-gap
behaviour. Six of them fail on the current main.
knisar
left a comment
There was a problem hiding this comment.
Reviewed 21ee7ea against main 5439934. The unsupported-format branch now returns the documented unassessed result with a stable reason, and the JSON and XML paths are unchanged.
Validation: the core suite passed with 332 passed, 15 skipped. Another 11 review checks passed, covering unsupported formats at thresholds 0, 0.5, and 1 across empty, malformed, and valid JSON output, plus the actual evaluation pipeline. In the pipeline, an unsupported format is counted as a coverage gap and excluded from the measured average; a run containing only unsupported formats has no measured pass rate. All seven CI jobs and both security checks passed.
No blocking findings. Thanks @KunyangZhang for the focused fix and regression coverage.
The first commit on this branch rewrote `programmatic.py` from a local checkout that predated wandb#74, so it silently restored the old behaviour where an unsupported `expected_format` returned `score=1.0`, `passed=True` and `assessed=True`. CI caught it: six cases in `tests/test_output_format_scorer.py` failed. This rebuilds the file from current `main` and re-applies only the wandb#79 change, so the wandb#74 contract (an unsupported format is un-assessed) is intact. The keyword tests are unchanged and still fail on `main` before the fix.
What changed
OutputFormatScorerreports an unsupportedexpected_formatas an unassessedrow instead of a full pass. Adds offline tests for the scorer.
Why
expected_formatis compared case-sensitively againstjsonandxml, andevery other value took this branch:
So a configuration mistake -
expected_format="JSON","yaml", or a typo -scored a full pass for any output, including empty and malformed ones, with
assessed=True:A row that was never checked then counts as a measured pass in aggregates, which
is the opposite of what the result contract asks for -
assessedis "True onlywhen the scorer produced a real signal ... surface the gap honestly instead of
inflating with neutral defaults". The scorer-authoring guide specifies the
unassessed shape:
score=0.0,passed=False,assessed=False, and a stabledetails["skipped"]reason. The branch now returns that, and recordsscorer_nameindetailsas the guide also asks. Thejsonandxmlpaths areunchanged, so no correctly configured scorer changes behaviour.
How it was checked
The new tests cover valid JSON, invalid JSON, a missing required key, valid and
invalid XML, and the unsupported-format row including its coverage-gap effect.
Six of them fail on the current
main.Notes:
reuse --no-multiprocessing lintwas not run: this environment is offline andreuse==6.2.0is not installable from it. The new test file carries the SPDXheader in the same form as the other test files, with my own name as the
copyright holder. The
ruffavailable here is 0.16.7 rather than the 0.16.6pinned in CI, on the
E9,F63,F7,F82selection.KeywordToxicityScorer; both touchprogrammatic.py, in different places. Ifyou would rather review a single change for the module, I can fold them
together.
This contribution was prepared with AI assistance, as the contributing guide
asks contributors to disclose.