feat: implement issue #841 — [Phase 1] Freeze the comparative pilot PR corpus + immutable LSP-off baseline and build the metric-extraction harness

[don-petry]

Closes #841

Implemented by dev-lead agent. Please review.

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

Warning

Review limit reached

@don-petry, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 59 minutes and 59 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: ea9138d7-bd1d-44a5-b462-dc86fbb8f0fa

📥 Commits

Reviewing files that changed from the base of the PR and between b3f862c and e18b13b.

📒 Files selected for processing (8)

• .github/workflows/lint.yml
• evals/lsp-pilot/README.md
• evals/lsp-pilot/dev/cases.jsonl
• evals/lsp-pilot/holdout/baseline-lsp-off.jsonl
• evals/lsp-pilot/holdout/cases.jsonl
• scripts/lsp_pilot_compare.sh
• tests/lsp_pilot_compare.bats
• tests/test_holdout_guard.bats

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch dev-lead/issue-841-20260621-0147

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[don-petry]

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

[donpetry-bot]

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-21T03:04:35Z.

[gemini-code-assist]

Code Review

This pull request introduces the LSP pilot evaluation infrastructure, which includes a frozen comparative corpus, an immutable LSP-off baseline, and a comparison harness script (scripts/lsp_pilot_compare.sh) along with its unit tests. The review feedback highlights critical performance bottlenecks in the comparison script, specifically pointing out inefficient subprocess spawns (such as calling awk inside loops) and the use of temporary files, and provides detailed suggestions to consolidate these operations into single, native awk executions.

[don-petry]

Dev-Lead — fix-reviews (applied)

Changes committed and pushed.

[don-petry]

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #853
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-06-21T02:43:19Z

[don-petry]

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-06-21T02:43:19Z

[don-petry]

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

[donpetry-bot]

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-21T03:17:55Z.

[don-petry]

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

[donpetry-bot]

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 71ec45eaa0ceb3a6f64e237f72d699403a554bbb
Review mode: triage-approved (single reviewer)

Summary

Phase 1 of the LSP pilot (#841): adds a frozen held-out corpus + dev smoke set under evals/lsp-pilot/, an immutable LSP-off baseline artifact, and a pure, unit-tested comparison harness (scripts/lsp_pilot_compare.sh). All five acceptance criteria are substantively met. Risk is LOW — eval infrastructure over synthetic, de-identified seed data (zero-padded SHAs), no auth/secrets/migrations/network, pure shell helpers with bats coverage. Triage's low-risk assessment is confirmed.

Linked issue analysis

Closes #841. AC1: evals/lsp-pilot/{dev,holdout} layout with PRs pinned by repo+pr_number+head_sha ✓. AC2: holdout immutability guard coverage asserted by new tests in tests/test_holdout_guard.bats (default evals/ prefix already covers the tree) ✓. AC3: baseline-lsp-off.jsonl committed as a frozen one-time artifact ✓. AC4: pure render_lsp_comparison renders per-PR + aggregate speed/cost/quality deltas with an explicitly-defined (findings, false_positives) quality proxy, unit-tested ✓. AC5: lp_missing_baselines / render fail loud (non-zero) when a candidate PR lacks a baseline counterpart, covered by tests ✓.

Findings

No blocking findings.

• Advisory (non-blocking): gemini-code-assist flagged per-row awk subprocess spawns and a mktemp temp file in scripts/lsp_pilot_compare.sh. These targeted the initial commit (aedde23); the current HEAD revision eliminates the temp file and consolidates _lp_aggregate into a single awk. Remaining per-row helper calls are immaterial on a fixed 6-PR corpus and reuse the org-mandated pricing helpers (AGENTS.md). dev-lead recorded fix-reviews (applied) followed by review-changes (no-changes) at the reviewed SHA.
• coderabbit review was DISMISSED; sonarqubecloud Quality Gate passed (0 new issues, 0 security hotspots).
• Secret scan: the run_secret_scanning MCP tool was not available in this run; the gitleaks CI check passed and the added data is synthetic/de-identified, so no secret exposure.

CI status

All checks green or appropriately skipped: Lint, ShellCheck, bats/unit-tests, holdout-guard, CodeQL (actions+python), AgentShield, Agent Security Scan, Secret scan (gitleaks), SonarCloud, CodeRabbit, dependency-audit (language jobs skipped — no matching ecosystems). No failing or cancelled checks. mergeStateStatus is BLOCKED only on the pending org-leads review this verdict addresses.

---

Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[donpetry-bot]

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-21T08:19:16Z.