Initiative (placeholder): Per-language LSP re-pilot — follow-up to #839 (Shell = NO-GO)

[don-petry]

Why this exists

Epic #839 piloted LSP (language-server) code intelligence in the pr-review harness for exactly one language/server: Shell via bash-language-server behind agent-lsp. The measured result was a NO-GO (decision: docs/lsp-pilot.md §7, recorded on #845):

Run	Model	Agg nav-token ratio	High-nav stratum
28718240665	sonnet-4-6	1.26×	1.01×
28940391169	opus-4-8 (prod)	1.05×	1.04×

The ≥2× navigation-token target was missed decisively; on shell diffs already cheap to grep, agent-lsp's MCP tool-result payloads cost about as much context as the reads they replace. That result does not generalize — it says nothing about languages with deeper cross-file type resolution, which is exactly where find-references is expected to beat grep.

This is a holding issue (inert: initiative:hold, no initiative:auto) to keep the open door tracked. Promote a child to a real pilot by removing the hold and planning it; do not auto-release.

The reusable harness is already in place (dormant, cost-free)

A per-language re-pilot is a restart, not a rebuild — reuse:

• scripts/lsp_pilot_run.sh — live A/B runner (LSP-off vs LSP-on on the same PRs)
• scripts/lsp_pilot_measure.sh / scripts/lsp_pilot_compare.sh — nav-token / tool-call / cold-start extraction + rendering
• .github/workflows/lsp-pilot-run.yml — on-demand workflow_dispatch driver
• evals/lsp-pilot/ — the frozen-corpus pattern (repin per language)

Per-language deltas needed each time: (a) pick + version-pin the LSP server, (b) point .github/mcp/lsp.json at it (or a per-lang variant) and update setup-lsp-pilot.sh install, (c) repin evals/lsp-pilot/holdout/cases.jsonl to real cross-file PRs in that language's repos, (d) dispatch the A/B.

Method fixes carried over from the Shell pilot (do these before re-running)

1. Gate on tokens first. The success metric is a conjunction (≥2× nav-token reduction AND no precision regression). Precision cannot rescue a GO once tokens miss — so run the token A/B first and only build a precision harness if tokens clear ~2×.
2. [Phase 2] Add the LSP finding-verification step to the deep/audit pr-review tiers #843 verification can't measure precision symmetrically. scripts/lib/lsp-verification.sh only emits finding_verification records on the LSP-on leg (it needs mcp__lsp__* in the allowlist), so it structurally cannot produce a false-positive delta vs LSP-off. Measuring the precision half needs a neutral judge (same scorer over both legs) — reuse the eval score_llm_judge path, not [Phase 2] Add the LSP finding-verification step to the deep/audit pr-review tiers #843.
3. N≥3 median per PR to beat the run-to-run navigation-strategy variance seen in the Shell controls.

Candidate children (the languages we actually use)

• [LSP re-pilot placeholder] Python (pyright) — broodminder-export / TalkTerm #1125 — Python re-pilot (pyright) — broodminder-export (96% Py), TalkTerm (34% Py). Highest a-priori odds: real cross-file imports + type resolution.
• [LSP re-pilot placeholder] TypeScript / JavaScript (typescript-language-server / tsgo) — broodly / google-app-scripts / markets #1126 — TypeScript / JavaScript re-pilot (typescript-language-server, or tsgo when GA) — broodly (TS), google-app-scripts (Apps Script JS), embedded JS in markets. The vendor-cited strong case.

Lower-priority / not currently scoped: HTML, Groovy, HCL/Terraform (small footprint; weak LSP-vs-grep case for review navigation).

Refs: discussion #578 · epic #839 · decision #845 · docs/lsp-pilot.md.

[don-petry]

📌 Rebuild pointer — how to restart this without keeping the harness in-tree

The Shell pilot (NO-GO, #845) is being removed from the repo (retire PR — the ~3,800 LOC harness + Shell corpus + docs + the production-integration hooks). Everything below is pinned here so a future per-language re-pilot is a cherry-pick + edit 3 files, not a rebuild.

Stable git ref

The complete, working harness lives at petry-projects/.github-private@f882266373a1c16ba18760c1ef57753652f9cf7d (origin/main immediately before the retire PR). Built by PRs #848 · #853 · #852 · #861 · #922 · #931 · #952 · #961 · #1034 (stories #840–846). Restore with git checkout f882266 -- <path>.

What to cherry-pick (the self-contained token-A/B harness — language-agnostic)

scripts/lsp_pilot_run.sh        # the A/B runner (LSP-off vs LSP-on on the same PRs)
scripts/lsp_pilot_measure.sh    # nav-token / tool-call extraction from the stream-json transcript
scripts/lsp_pilot_compare.sh    # renders baseline-vs-candidate verdict
scripts/lsp_pilot_report.sh     # optional richer report
scripts/setup-lsp-pilot.sh      # installs the LSP server + prints the allowed-tools set
.github/workflows/lsp-pilot-run.yml   # on-demand workflow_dispatch driver
.github/mcp/lsp.json                  # the agent-lsp MCP server config
evals/lsp-pilot/README.md + dir scaffold  # frozen-corpus conventions + validator
tests/lsp_pilot_compare.bats, tests/lsp_pilot_report.bats,
tests/dev-lead/unit/test_lsp_pilot_run.bats, .../test_lsp_pilot_measure.bats

Per-language changes needed (this is the entire delta)

1. .github/mcp/lsp.json — one line: bash:bash-language-server,start → python:pyright ([LSP re-pilot placeholder] Python (pyright) — broodminder-export / TalkTerm #1125) or typescript:typescript-language-server ([LSP re-pilot placeholder] TypeScript / JavaScript (typescript-language-server / tsgo) — broodly / google-app-scripts / markets #1126).
2. scripts/setup-lsp-pilot.sh — install that server instead of bash-language-server (agent-lsp resolves mcp__lsp__* generically, so the allowed-tools set is unchanged).
3. .github/workflows/lsp-pilot-run.yml — swap the BASH_LANGUAGE_SERVER_VERSION env for the per-language server-version pin.
4. Repin evals/lsp-pilot/holdout/cases.jsonl to real cross-file PRs in that language's repos (the actual research each time).

What is NOT needed to test another language (deliberately removed, do NOT resurrect for a test)

The production-integration arm — scripts/lsp_pilot_emit.sh, scripts/lib/lsp-verification.sh, and the hooks in pr-review.yml (LSP_PILOT_ENABLED gate), review-one-pr.sh (apply_lsp_verification), engine.sh, token-metrics.sh, and the lsp_verification annotation blocks in prompts/deep-review.md + security-audit.md. The standalone A/B runner does not touch any of it. (It's also the asymmetric-verification path that can't measure a symmetric precision delta — see method note above; use a neutral judge instead.)

The Shell go/no-go corpus (for reproducibility — repin per language, don't reuse)

The 20 real .github-private PRs the Shell pilot scored, pinned by head SHA:

PR	head SHA	stratum
#528	5bcfcb49379a	high-nav
#928	6def6d0ed9f9	high-nav
#453	e76120d25aa2	high-nav
#379	15b82baf1d13	high-nav
#468	a1ac99f3c094	high-nav
#649	3d374b3bc084	high-nav
#720	56a68897cab6	high-nav
#784	61ce4a49fc7d	high-nav
#592	15adf90cadd2	high-nav
#658	61c5ae9a60ce	high-nav
#609	6255bea2a327	high-nav
#1012	e1b425b28993	high-nav
#460	a77aa42f0f80	medium-nav
#1011	382b898eb14a	medium-nav
#494	a699132666e7	medium-nav
#196	aeb0582dd862	diagnostics
#1065	90009abebd56	control
#953	f48671a753ba	control
#867	4d772cfbcc91	control
#847	5dd92e6b81f4	control