Skip to content

eval: broaden the golden corpus on a measured backlog, including the third the measure cannot classify #16159

Description

@mrveiss

Parent: #16108 · third of the three steps derived on #16115

What exists

autobot-backend/eval/golden/ holds three trajectories covering three classes:

code_fix_null_guard.json      task_class=code_fix        expected_tools=read_file,edit_file,run_tests
qa_deploy_role.json           task_class=knowledge_qa    expected_tools=search_knowledge_base
triage_502_websocket.json     task_class=troubleshooting expected_tools=search_knowledge_base,read_file

task_class is free-form — TrajectoryReplayer only carries it through into per-class reporting — so growing the corpus needs no schema change.

What the backlog looks like, and what that measure cannot see

Open issues classified by the repo's own conventional-commit vocabulary, across 400:

bug 65 · tech-debt 29 · ci 26 · docs 18 · test 17 · feat 17
security 16 · fix 16 · refactor 12 · chore 12 · guard 14 · perf 5

129 of the 400 carry no conventional prefix. So this classifies about two-thirds of the backlog and the remaining third is unmeasured — not empty, unmeasured. Proportioning a corpus on this alone would proportion it on the part that happened to be easy to count, which is the failure mode this whole umbrella is about.

On the part that is visible, bug + fix (81) maps to the single class already covered, and the largest uncovered neighbours are test authoring, documentation, refactoring, security review and CI configuration.

Acceptance criteria

  • A stated method for choosing task classes, with its coverage of the backlog measured and named — including what the method cannot classify
  • The unprefixed third sampled rather than ignored, so the corpus is not proportioned on convenience
  • Corpus covers the recurring classes that method identifies, with the count per class justified by the measure rather than chosen
  • Per-task-class pass/regress reporting remains meaningful at that breadth
  • The measurement is re-runnable, so a later claim of "the corpus is representative" can be rechecked rather than believed

Blocked

Blocked by #16157. A corpus is worthless while the candidate echoes each golden back: every trajectory added is another comparison of a file with itself, and a broader corpus would make the resulting green look better-founded while asserting exactly as much as it does now — nothing. This is a genuine dependency rather than a preference, which is why it carries the edge and #16158 does not.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions