Trajectory-level eval & regression harness

Part of umbrella #10542 · Theme A (collaboration core) · trust/safety enabler

Problem

There is no trajectory-level eval/regression suite. RLM judges single responses and LLC has replay, but nothing catches "a model upgrade or prompt change silently made the agent worse at task-class Y." A collaboration platform's whole value is trust — and without this, every provider/prompt/model change is a gamble. This is also what lets us safely adopt newer/better models for everyone.

What exists today

• rlm/evaluator.py + rlm/benchmark.py — response-level judging + benchmark scaffolding.
• llc/api/replay.py + llc/models/replay_log.py + llc/tests/test_replay.py — replay primitives.
• performance_benchmarks_test.py, system_benchmarks_performance_test.py — perf (not quality) benches.
• Observability: Langfuse / LangSmith / OTel / Prometheus observers.

What to build

• A golden-trajectory harness: record canonical task runs (input + tool sequence + final outcome), replay them against a candidate model/prompt/skill set, and score drift with the RLM evaluator + deterministic checks.
• A small eval set seeded from real trajectories (per task-class), runnable in CI as a non-blocking signal first, then gated.
• A diff report: "candidate vs baseline — N regressions, M improvements," surfaced per task-class.
• Wire to the provider-swap path so changing the default model runs the suite.

Acceptance criteria

• eval command/CI job replays N golden trajectories and emits a pass/regress report.
• A deliberately worse prompt is caught as a regression (test).
• Report distinguishes regressions vs improvements per task-class.
• Reuses RLM evaluator + LLC replay rather than a new judging stack.

Entry points

autobot-backend/rlm/ (evaluator, benchmark) · autobot-backend/llc/api/replay.py + llc/models/replay_log.py · autobot-backend/llm_shared/observability/ · a new eval/ harness + a CI workflow under .github/workflows/.