eval: the trajectory-eval gate is non-blocking with 3 goldens — broaden the corpus so review can safely shorten #16115

Activity

1. mrveiss

   added

   backend

   testing

   benchmarking

   on Sep 9, 2026
2. mrveiss

   added a parent issue on Sep 9, 2026

   • Umbrella: self-correction, compaction-fidelity and measurement gaps from a comparative prompt/skill-library audit #16108
3. mrveiss commented on Sep 9, 2026

   

   mrveiss

   on Sep 9, 2026

   OwnerAuthor

   More actions

   Took this as "answer how many goldens and chosen how, before make it blocking". The measurements say the corpus is not the first problem, and broadening it first would make things worse rather than better.

   The gate cannot fail, at any corpus size

   CI runs the harness with no injected runner:

   .github/workflows/trajectory-eval.yml     python -m eval.run --json trajectory-eval-report.json
   eval/run.py:55                            report = await replayer.run(goldens, candidate or baseline_candidate)
   eval/candidates.py:33                     """Echo the golden's own recorded outcome (self-consistency check)."""
                                             tool_sequence=list(golden.expected_tools)
   

   So _check_tools(golden.expected_tools, actual) compares the golden against itself. The response text and final status are echoed from the same file. There is no candidate to drift from the baseline, so no input can turn this red.

   The real path exists but is a seam, not an implementation:

   eval/candidates.py:53   async def live_replay_candidate(...)  # pragma: no cover - wiring seam
                           raise NotImplementedError(...)
   

   This is the finding that reorders the issue. Adding goldens multiplies a check that always passes; promoting it to blocking adds a gate that cannot fire while looking like a quality floor. A green that asserts nothing is worse than an absent check, because the absent one does not get cited as evidence.

   None of that is a criticism of the harness — baseline_candidate documents itself honestly as a self-consistency check, and the seam is labelled. The defect is that CI runs the self-check and the workflow header describes the result as drift detection.

   It also almost never runs

   Trigger paths are autobot-backend/eval/**, rlm/**, llc/**/replay*, and the workflow file.

   Measurement	Result
   Last 60 merged PRs that would have triggered it	0
   Entire run history	12 runs, all 2026-09-06, all on issue-15812-budget-scope + one push

   So its trigger population is the harness itself, not the behaviour the harness exists to protect. Nothing that changes agent behaviour under agents/ (97 files) or the wider llc/ tree (379 files) brings the eval with it. Promoted today, it would gate 0% of merges — and the corpus question would still be unanswered, because breadth of goldens does not widen when a gate runs.

   What the corpus question actually needs

   The three goldens cover code_fix, knowledge_qa, troubleshooting. task_class is free-form — the runner only carries it into reporting — so growth needs no schema change.

   Measuring the backlog by the repo's own work-type vocabulary, across 400 open issues:

   bug 65 · tech-debt 29 · ci 26 · docs 18 · test 17 · feat 17
   security 16 · fix 16 · refactor 12 · chore 12 · guard 14 · perf 5
   

   Stated as a limit rather than buried: 129 of those issues carry no conventional prefix, so this classifies about two-thirds of the backlog and the tail is unmeasured. A corpus proportioned on this alone would be proportioned on the part that was easy to count.

   On that partial view, bug+fix (81) maps to the one class already covered, and the largest uncovered neighbours are test authoring, docs, refactor, security review and CI configuration.

   Suggested ordering, and the promotion criterion

   1. Implement a real candidate (or wire live_replay_candidate). Until then every other criterion here is unfalsifiable.
   2. Widen the trigger to the behaviour the gate protects, and state the intended population — a gate whose reach is its own source tree measures itself.
   3. Then broaden the corpus, proportioned to a backlog measure that accounts for the unprefixed third.
   4. Then promote, on a criterion of the form: N consecutive runs against real candidates on real PRs, with a measured false-positive rate below X — which cannot be evaluated at all until (1), and is what AC 4 is already asking for.

   Happy to file 1–3 as children of this issue and take the first. Flagging rather than proceeding, because this changes the issue's premise and that is a call for whoever owns #16108's shape rather than mine to make unilaterally.
4. mrveiss

   marked this as blocked by 4 issues on Sep 10, 2026

   • eval: the trajectory gate compares every golden against itself, so it cannot fail #16157
   • eval: the trajectory gate has never run on a behaviour change — 0 of the last 60 merged PRs would trigger it #16158
   • eval: broaden the golden corpus on a measured backlog, including the third the measure cannot classify #16159
   • A replay timeout is reported as a regression, which is a verdict about a candidate that was never scored #16192
5. mrveiss

   added a commit that references this issue on Sep 10, 2026

   fix(eval): give the trajectory gate a candidate that can disagree wit…

   ...

   d49cd1b
6. mrveiss

   modified the milestones: Backlog, v0.16.0 on Sep 12, 2026