Parent: #16108 · third of the three steps derived on #16115
What exists
autobot-backend/eval/golden/ holds three trajectories covering three classes:
code_fix_null_guard.json task_class=code_fix expected_tools=read_file,edit_file,run_tests
qa_deploy_role.json task_class=knowledge_qa expected_tools=search_knowledge_base
triage_502_websocket.json task_class=troubleshooting expected_tools=search_knowledge_base,read_file
task_class is free-form — TrajectoryReplayer only carries it through into per-class reporting — so growing the corpus needs no schema change.
What the backlog looks like, and what that measure cannot see
Open issues classified by the repo's own conventional-commit vocabulary, across 400:
bug 65 · tech-debt 29 · ci 26 · docs 18 · test 17 · feat 17
security 16 · fix 16 · refactor 12 · chore 12 · guard 14 · perf 5
129 of the 400 carry no conventional prefix. So this classifies about two-thirds of the backlog and the remaining third is unmeasured — not empty, unmeasured. Proportioning a corpus on this alone would proportion it on the part that happened to be easy to count, which is the failure mode this whole umbrella is about.
On the part that is visible, bug + fix (81) maps to the single class already covered, and the largest uncovered neighbours are test authoring, documentation, refactoring, security review and CI configuration.
Acceptance criteria
Blocked
Blocked by #16157. A corpus is worthless while the candidate echoes each golden back: every trajectory added is another comparison of a file with itself, and a broader corpus would make the resulting green look better-founded while asserting exactly as much as it does now — nothing. This is a genuine dependency rather than a preference, which is why it carries the edge and #16158 does not.
Parent: #16108 · third of the three steps derived on #16115
What exists
autobot-backend/eval/golden/holds three trajectories covering three classes:task_classis free-form —TrajectoryReplayeronly carries it through into per-class reporting — so growing the corpus needs no schema change.What the backlog looks like, and what that measure cannot see
Open issues classified by the repo's own conventional-commit vocabulary, across 400:
129 of the 400 carry no conventional prefix. So this classifies about two-thirds of the backlog and the remaining third is unmeasured — not empty, unmeasured. Proportioning a corpus on this alone would proportion it on the part that happened to be easy to count, which is the failure mode this whole umbrella is about.
On the part that is visible,
bug+fix(81) maps to the single class already covered, and the largest uncovered neighbours are test authoring, documentation, refactoring, security review and CI configuration.Acceptance criteria
Blocked
Blocked by #16157. A corpus is worthless while the candidate echoes each golden back: every trajectory added is another comparison of a file with itself, and a broader corpus would make the resulting green look better-founded while asserting exactly as much as it does now — nothing. This is a genuine dependency rather than a preference, which is why it carries the edge and #16158 does not.