feat: implement issue #841 — [Phase 1] Freeze the comparative pilot PR corpus + immutable LSP-off baseline and build the metric-extraction harness

.github/workflows/lint.yml

@@ -46,6 +46,7 @@ jobs:
             tests/fleet_stub_drift.bats \
             tests/test_initiative_canary.bats \
             tests/token_report.bats \
+            tests/lsp_pilot_compare.bats \
             tests/auto_rebase_health.bats \
             tests/model_pricing.bats \
             tests/test_review_cycle.bats \

evals/lsp-pilot/README.md

@@ -0,0 +1,76 @@
+# `evals/lsp-pilot/` — frozen comparative corpus + immutable LSP-off baseline
+
+This held-out set backs the LSP pilot (epic
+[#839](https://github.com/petry-projects/.github-private/issues/839), scoping doc
+[`docs/lsp-pilot.md`](../../docs/lsp-pilot.md)). It exists so candidate LSP-MCP
+servers are compared against a **fixed target that cannot be tuned to** — the
+go/no-go must reflect real generalization, not overfitting to a few PRs.
+
+It reuses the standard held-out machinery (see [`../README.md`](../README.md)): the
+`dev/` (proposer-visible) vs `holdout/` (gate-scored) split, the directory
+validator (`evals/validate-cases.py`), and the hard immutability gate
+(`scripts/lib/holdout-guard.sh` + `.github/workflows/holdout-guard.yml`, keyed on
+`HOLDOUT_GUARDED_PREFIXES` whose default `evals/` already covers this tree). No new
+immutability mechanism is invented here.
+
+## Layout
+
+```
+evals/lsp-pilot/
+  dev/cases.jsonl                 # proposer-visible smoke PRs
+  holdout/cases.jsonl             # frozen scored corpus (the comparison target)
+  holdout/baseline-lsp-off.jsonl  # immutable LSP-off control, captured ONCE
+```
+
+## The frozen corpus (`cases.jsonl`)
+
+Each case pins one pilot PR by **immutable identifiers** — `repo` + `pr_number` +
+`head_sha` — plus a unique `id` and a `description` of the navigation claim it
+exercises (find-references or diagnostics, per the scoping doc §2). Pinning the
+head SHA freezes *exactly which revision* is reviewed, so the corpus is a moving
+target for nobody.
+
+> The identifiers committed here are **synthetic, de-identified seeds** (note the
+> zero-padded `head_sha`s), consistent with the de-identification requirement in
+> [`../README.md`](../README.md). They are replaced with the real pinned pilot PR
+> set when the server is wired and the controls are run (story
+> [#842](https://github.com/petry-projects/.github-private/issues/842)). The
+> structure, schema, and immutability guarantee are what this story freezes.
+
+## The immutable LSP-off baseline (`baseline-lsp-off.jsonl`)
+
+The LSP-off control is captured **once** and committed as a frozen artifact, so
+later stories compare against it rather than re-deriving it (a re-derived baseline
+is a moving target — a reward-hacking vector). One JSON object per corpus PR:
+
+| field | meaning |
+|---|---|
+| `pr` | `repo#pr_number@head_sha` — the join key against a candidate run |
+| `variant` / `candidate` | `lsp-off` / `baseline` |
+| `nav_tokens` | navigation tool-call tokens (the headline cost metric) |
+| `tool_calls` | navigation tool-call count |
+| `findings` / `false_positives` | the review-quality proxy (see below) |
+| `cold_start_s` | `null` — **N/A** for the LSP-off control (no server to launch) |
+| `wall_time_s`, `model`, `*_tokens` | speed + cost inputs (ET/USD) |
+
+## The review-quality proxy (explicit, not ad hoc)
+
+Quality is `(findings, false_positives)` per PR. A candidate is a **quality
+regression on a PR when its `false_positives` exceeds the frozen baseline's** —
+i.e. precision got worse. This is the same definition the harness
+([`scripts/lsp_pilot_compare.sh`](../../scripts/lsp_pilot_compare.sh)) renders and
+enforces, so the go/no-go is reproducible. Per the success metric, a navigation-
+token win that costs precision is a **no-go**, so this metric is mandatory, not
+optional.
+
+## Comparing a candidate against the baseline
+
+```bash
+bash scripts/lsp_pilot_compare.sh \
+  evals/lsp-pilot/holdout/baseline-lsp-off.jsonl \
+  <candidate-lsp-on>.jsonl
+```
+
+The harness renders per-PR and aggregate speed/cost/quality deltas and **exits
+non-zero if any candidate PR has no baseline counterpart** — a partial corpus can
+never masquerade as a clean comparison.

evals/lsp-pilot/dev/cases.jsonl

@@ -0,0 +1,2 @@
+{"id":"lsp-dev-smoke-find-references","repo":"petry-projects/.github-private","pr_number":707,"head_sha":"0000000000000000000000000000000000000707","description":"Proposer-visible smoke case: a small cross-file shell change exercising find-references navigation end-to-end. Visible by design; never scored by the gate.","tags":["find-references","smoke","shell"]}
+{"id":"lsp-dev-smoke-diagnostic","repo":"petry-projects/.github-private","pr_number":708,"head_sha":"0000000000000000000000000000000000000708","description":"Proposer-visible smoke case: a single-file shell change with a diagnostics-detectable issue, to smoke-test the publishDiagnostics path. Never scored by the gate.","tags":["diagnostics","smoke","shell"]}

evals/lsp-pilot/holdout/baseline-lsp-off.jsonl

@@ -0,0 +1,6 @@
+{"pr":"petry-projects/.github-private#701@0000000000000000000000000000000000000701","variant":"lsp-off","candidate":"baseline","model":"claude-opus-4-7","input_tokens":42000,"cache_read_tokens":18000,"output_tokens":3200,"nav_tokens":21800,"tool_calls":34,"findings":4,"false_positives":1,"cold_start_s":null,"wall_time_s":188.0}
+{"pr":"petry-projects/.github-private#702@0000000000000000000000000000000000000702","variant":"lsp-off","candidate":"baseline","model":"claude-opus-4-7","input_tokens":36500,"cache_read_tokens":15000,"output_tokens":2600,"nav_tokens":15400,"tool_calls":26,"findings":3,"false_positives":1,"cold_start_s":null,"wall_time_s":151.0}
+{"pr":"petry-projects/.github-private#703@0000000000000000000000000000000000000703","variant":"lsp-off","candidate":"baseline","model":"claude-opus-4-7","input_tokens":28800,"cache_read_tokens":12000,"output_tokens":2100,"nav_tokens":9800,"tool_calls":18,"findings":2,"false_positives":0,"cold_start_s":null,"wall_time_s":121.0}
+{"pr":"petry-projects/.github-private#704@0000000000000000000000000000000000000704","variant":"lsp-off","candidate":"baseline","model":"claude-opus-4-7","input_tokens":24300,"cache_read_tokens":10000,"output_tokens":1800,"nav_tokens":7600,"tool_calls":14,"findings":2,"false_positives":0,"cold_start_s":null,"wall_time_s":104.0}
+{"pr":"petry-projects/.github-private#705@0000000000000000000000000000000000000705","variant":"lsp-off","candidate":"baseline","model":"claude-opus-4-7","input_tokens":51200,"cache_read_tokens":22000,"output_tokens":3900,"nav_tokens":29600,"tool_calls":48,"findings":5,"false_positives":2,"cold_start_s":null,"wall_time_s":233.0}
+{"pr":"petry-projects/.github-private#706@0000000000000000000000000000000000000706","variant":"lsp-off","candidate":"baseline","model":"claude-opus-4-7","input_tokens":14100,"cache_read_tokens":6000,"output_tokens":900,"nav_tokens":1200,"tool_calls":3,"findings":1,"false_positives":0,"cold_start_s":null,"wall_time_s":58.0}

evals/lsp-pilot/holdout/cases.jsonl

@@ -0,0 +1,6 @@
+{"id":"lsp-corpus-find-references-callers","repo":"petry-projects/.github-private","pr_number":701,"head_sha":"0000000000000000000000000000000000000701","description":"Cross-file change to a shell helper whose review must claim 'this breaks N callers'; find-references is the navigation that LSP replaces a recursive grep with.","tags":["find-references","cross-file","shell"]}
+{"id":"lsp-corpus-unused-symbol","repo":"petry-projects/.github-private","pr_number":702,"head_sha":"0000000000000000000000000000000000000702","description":"Removes a shell function the diff says is unused; the LSP-off control resolves 'unused' by textual grep, the kind of claim find-references verifies semantically.","tags":["find-references","unused-symbol","shell"]}
+{"id":"lsp-corpus-undefined-var-diagnostic","repo":"petry-projects/.github-private","pr_number":703,"head_sha":"0000000000000000000000000000000000000703","description":"Introduces a reference to an undefined variable under set -u; the reviewer's 'this is undefined' claim is what textDocument/publishDiagnostics confirms.","tags":["diagnostics","undefined","shell"]}
+{"id":"lsp-corpus-syntax-error-diagnostic","repo":"petry-projects/.github-private","pr_number":704,"head_sha":"0000000000000000000000000000000000000704","description":"Adds a shell snippet with a real syntax error; a finding-verification step asks the language server's diagnostics rather than eyeballing the diff.","tags":["diagnostics","syntax-error","shell"]}
+{"id":"lsp-corpus-cross-file-rename-claim","repo":"petry-projects/.github-private","pr_number":705,"head_sha":"0000000000000000000000000000000000000705","description":"Renames an exported function across several scripts; the review must confirm every reference moved, a high navigation-token claim under grep.","tags":["find-references","rename","cross-file","shell"]}
+{"id":"lsp-corpus-clean-no-nav-needed","repo":"petry-projects/.github-private","pr_number":706,"head_sha":"0000000000000000000000000000000000000706","description":"A self-contained docs/comment-only change that needs no semantic navigation; pins a low-navigation control so a token win is not an artefact of skipped work.","tags":["no-navigation","control","docs"]}