feat: implement issue #926 — Runaway PR #860: agents self-triggering — add per-PR churn breaker, human-gated cap resets, and self-trigger guards - #928
Merged
Conversation
…— add per-PR churn breaker, human-gated cap resets, and self-trigger guards
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
More reviews will be available in 59 minutes and 55 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (10)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Collaborator
Author
Dev-Lead — review-changes (applied)Changes committed and pushed. |
don-petry
enabled auto-merge (squash)
June 25, 2026 03:35
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a per-PR automated-churn circuit breaker (pr-automation-budget.sh) and updates the review cycle cap logic to prevent runaway agent loops by gating resets on human interactions. Feedback on these changes highlights several important robustness improvements: slurping paginated JSON arrays from gh api to prevent syntax errors, using optional chaining in jq to avoid crashes on null author objects, checking both dry-run environment variables, and robustly parsing the PR number from URLs.
don-petry
disabled auto-merge
June 25, 2026 03:35
Collaborator
Author
Dev-Lead — review-changes (applied)Changes committed and pushed. |
don-petry
enabled auto-merge (squash)
June 25, 2026 03:38
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
don-petry
disabled auto-merge
June 25, 2026 03:39
Collaborator
Author
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
don-petry
enabled auto-merge (squash)
June 25, 2026 03:41
don-petry
disabled auto-merge
June 25, 2026 05:52
Collaborator
Author
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
|
don-petry
enabled auto-merge (squash)
June 25, 2026 05:53
donpetry-bot
approved these changes
Jun 25, 2026
donpetry-bot
left a comment
donpetry-bot
left a comment
Contributor
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: MEDIUM
Reviewed commit: d7119be1accfb8066cc332cc7c3df79bfc664e12
Review mode: triage-approved (single reviewer)
Summary
Implements the in-repo (P0) half of #926 — the PR-#860 runaway fix. Two mechanisms: (1) review-cycle cap reset is now human-gated (only a non-bot state=APPROVED review resets; the cascade's own approval marker and machine approvals like repair-pr-approvals no longer re-arm the cap), and (2) a new per-PR automation budget (scripts/lib/pr-automation-budget.sh) that bounds TOTAL agent activity (commits + comments + reviews) since the last human interaction at MAX_PR_AUTOMATION_CYCLES=10, escalating once (deduped comment + needs-human-review + auto-merge off) on exhaustion. FORCE_REVIEW deliberately does NOT bypass the budget. Wired into review-one-pr.sh and both dev-lead-fix-* scripts before any automated write. The org-repo half (kill the self-mention) is correctly scoped to a separate PR per the postmortem. Net +715/-30 across 10 files, 3 of them new test files.
Linked issue analysis
Closes #926 ("Runaway PR #860: agents self-triggering"). Substantively addresses the repo-scoped P0 items: human-gated cap reset and the per-PR churn breaker. The org-repo self-trigger kill is explicitly deferred to a separate PR in petry-projects/.github (documented in the new postmortem and consistent with the thin-caller stub constraint in AGENTS.md). Direction of the change is fail-safe: it makes automation strictly more conservative and escalates to humans rather than looping.
Findings
No blocking findings.
- Logic verified: compute_pr_automation_cycles takes max() of non-bot event timestamps as the human reset point and counts only bot events strictly after it; empty/null login is conservatively treated as a bot (cannot reset). Malformed/empty input degrades to 0 so the integer gate never breaks. is_human_approval requires state=APPROVED AND non-bot author.
- Good correctness fix in review-one-pr.sh: PR_ITEMS now retains bodyless reviews via 'select(.body != null or .state != null)', so a human approval with an empty body still resets the cap (previously would have been dropped).
- Minor (non-blocking): event ordering relies on string comparison of ISO-8601 timestamps. Agent activity is UTC 'Z' so this is sound; a human git commit authored with a non-UTC offset could compare slightly off, but this is a heuristic breaker with a conservative default and the failure mode is benign (at worst counts one extra/fewer event).
- Secret scan: GitHub Secret Protection MCP tool (run_secret_scanning) is not exposed in this run; relied on the gitleaks CI check, which passed. No secrets, auth, crypto, or migration surface in the diff.
- Downstream: scripts are part of the shared pr-review surface consumed by 5 repos, but review-cycle.sh and review-one-pr.sh ship together as one pinned bundle, so the added author/state item fields cannot be seen partially-upgraded. No compat break.
- Test coverage is strong: 176-line unit suite for the budget lib, 142-line E2E regression reproducing both #860 loops (ack storm + review/fix ping-pong) asserting halt + exactly-once escalation, and updated cycle-cap tests proving machine approvals no longer reset.
CI status
All required checks green: Lint/ShellCheck, bats, unit-tests, CodeQL (actions+python), SonarCloud (Quality Gate passed, 0 new issues), gitleaks secret scan, AgentShield, Holdout Guard, Test-Deletion Guard, agent-profile/stub validations. Advisory bots: coderabbitai APPROVED at head SHA; gemini-code-assist left non-blocking comments that the dev-lead fixups already applied (final dev-lead status: no-changes). The two CANCELLED dev-lead dispatch/ci-relay checks are the expected self-trigger/[skip ci-relay] guards, not failures. Codex advisory hit a usage limit (informational). mergeStateStatus is BEHIND (branch needs a rebase before merge) — not a review blocker.
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
Contributor
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-25T06:57:06Z. |
This was referenced Jun 26, 2026
Closed
don-petry
pushed a commit
that referenced
this pull request
Jun 26, 2026
Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). The producer that scripts/lsp_pilot_compare.sh consumes. - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off Grep/Glob/Read/Bash vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison into the job summary. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse the repo's vetted pins; both legs gated to a modest model to bound cost. - tests: 17 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline; quality proxy is populated only when the verification step runs. shellcheck clean; bats green; workflow YAML validates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr
don-petry
pushed a commit
that referenced
this pull request
Jun 26, 2026
Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr
don-petry
added a commit
that referenced
this pull request
Jun 27, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
This was referenced Jul 4, 2026
This was referenced Jun 2, 2026
don-petry
added a commit
that referenced
this pull request
Aug 2, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 3, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 3, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 3, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 7, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 7, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 7, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 7, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 7, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 7, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 7, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 7, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 7, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 7, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 8, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 8, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 8, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
don-petry
added a commit
that referenced
this pull request
Aug 8, 2026
…tor (#952) Adds the missing #844 execution path: a self-contained, dispatchable workflow that measures both review variants LIVE on real PRs, so the comparison no longer depends on the frozen synthetic baseline. - scripts/lsp_pilot_measure.sh: turns one review's stream-json transcript + Token Cost Observatory log into one pilot-schema JSONL record (nav_tokens + tool_calls from the transcript; findings/false_positives from #843 verification records; cold_start_s from the #846 lsp_cold_start record; real reported usage). jq pre-flight in main(); value-flags guard their argument via ${2?…} (set -u safe). - scripts/lsp_pilot_run.sh: per PR, runs the same navigation-heavy review twice (LSP-off vs LSP-on + the pinned agent-lsp nav allowlist), measures each, and renders the comparison; the harness exit code is preserved (FAIL LOUD on a missing baseline), not masked. - .github/workflows/lsp-pilot-run.yml: workflow_dispatch driver. Installs agent-lsp + bash-language-server (cold-start record), runs the A/B over a real smoke corpus (#928, #899), uploads JSONL artifacts. Action SHAs reuse vetted pins. - tests: 18 unit cases over the pure extractor/driver helpers. Honest scope: a navigation-COST probe (nav_tokens/tool_calls/cold-start/ET/USD), not the full review pipeline. shellcheck clean; bats green; workflow YAML validates. Claude-Session: https://claude.ai/code/session_016Y3xW3cnB5wscv4SmjSNEr Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #926
Implemented by dev-lead agent. Please review.