Skip to content

eval: the trajectory gate has never run on a behaviour change — 0 of the last 60 merged PRs would trigger it #16158

Open

Description

@mrveiss

Parent: #16108 · second of the three steps derived on #16115

The gate's trigger population is its own source tree

.github/workflows/trajectory-eval.yml fires only on:

autobot-backend/eval/**
autobot-backend/rlm/**
autobot-backend/llc/**/replay*
.github/workflows/trajectory-eval.yml

Two independent measurements of what that means in practice:

Measurement Result
Of the last 60 merged PRs, how many would have triggered it 0
Its entire run history 12 runs, all on 2026-09-06, all from issue-15812-budget-scope plus one push

So it has only ever run on the branch that built it. Nothing that changes agent behaviour under autobot-backend/agents/ (97 files) or the wider autobot-backend/llc/ tree (379 files) brings the eval along with it.

Promoted to blocking today, it would gate 0% of merges. A gate whose reach is its own source tree measures itself.

Acceptance criteria

  • Trigger paths cover the code whose behaviour the goldens assert — the agent and tool-dispatch surface, not just the harness
  • The intended population is stated in the workflow, so a later narrowing is visibly a change of scope rather than a tidy-up
  • Verified by measurement, not by reading the glob: re-run the "how many of the last N merged PRs would trigger this" count and record the number
  • The paths are not widened to everything — a gate that runs on every PR is a cost with no more signal than one that runs on the right ones

That last criterion is a real tension rather than a formality. This workflow installs dependencies and replays a corpus; running it on all ~60 PRs a day is not free, and the fix is to name the surface that actually changes trajectories.

Ordering

Not blocked by #16157, deliberately — widening the trigger is independent work and the edge would be a preference rather than a dependency. It is worth doing after it, though: widening the trigger first means running a check that cannot fail, more often. Sequence by judgement, not by a blocked_by edge that would misdescribe the relationship.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions