PR Review Agent — failures detected 2026-05-12

[github-actions]

1. Executive Summary

Status: DEGRADED
Period: 2026-05-11T22:16Z – 2026-05-12T07:52Z
Result: 6 of 15 runs failed (40%), including 5 consecutive failures from 01:35Z onward

Key findings:

• gh copilot suggest -p command format is broken — "Invalid command format" on every Copilot invocation; the fallback engine is completely non-functional
• Claude (Anthropic) rate limits hit regularly, especially on claude-opus-4-7; every rate-limit event now triggers the broken Copilot fallback, guaranteeing session abort
• GOOGLE_API_KEY is unset, removing Gemini as a buffer between Claude exhaustion and Copilot

Action required: Fix the gh copilot CLI invocation syntax in review-one-pr.sh / review-batch.sh immediately — this is the single change that restores the fallback chain and prevents guaranteed session aborts on Claude rate-limit events.

---

2. Failure Breakdown

Failure Category	Affected Runs	Example Error Message
Copilot CLI breaking change	#708, #711, #712, #713, #714, #715 (all 6)	error: Invalid command format. Did you mean: copilot -i "suggest -p ..."
Claude engine rate limit	#713, #714, #715 (primary trigger); #711, #712 (contributing)	rate limit detected — exiting with code 2 for engine fallback / You've hit your limit · resets 5am (UTC)
Per-PR session-abort bug (old code — now fixed by #133)	#708, #711, #712	Session aborted early after failure on ... (exit code 1). Skipped N remaining candidate(s)
Missing Gemini fallback (configuration gap)	All 6 (no Gemini buffer available)	Claude rate limit hit but Gemini fallback unavailable (CLI not installed or GOOGLE_API_KEY missing)

---

3. Error Patterns

Pattern A — Copilot CLI invalid command format (all 6 runs)

Exact error (representative from run #715, step "Review each PR (cascade)"):

error: Invalid command format.

Did you mean: copilot -i "suggest -p # Tier 1: Triage
...
For non-interactive mode, use the -p or --prompt option.
Try 'copilot --help' for more information.

• Where: Every call through the copilot engine path — triage (o4-mini via copilot), rubber duck, and single-review fallback
• Root cause: The gh copilot extension's suggest subcommand syntax has changed. The script invokes gh copilot suggest -p "PROMPT" (or equivalent), but the CLI now expects gh copilot -i "suggest -p PROMPT" (non-interactive flag before the subcommand). The full prompt is being echoed back in the error's "Did you mean:" suggestion, confirming the prompt itself is fine — only the invocation format is wrong.
• Impact: 100% of Copilot engine attempts fail. Since Copilot is the last fallback, every Claude rate-limit event results in session abort with no recovery.

Pattern B — Claude rate limits triggering the broken fallback (runs #713, #714, #715)

Exact error (run #714, step "Review each PR (cascade)"):

[tier1] usage/rate limit detected — exiting with code 2 for engine fallback
    limit message: You've hit your limit · resets 5am (UTC)
##[warning]Claude rate limit hit but Gemini fallback unavailable (CLI not installed or GOOGLE_API_KEY missing) — falling through to Copilot
##[warning]Gemini rate limit hit — switching to Copilot engine for remaining PRs
    [tier1] triage (o4-mini)
    [tier1] usage/rate limit detected — exiting with code 2 for engine fallback
    limit message:  (stderr: error: Invalid command format.
##[error]Rate limit hit on copilot engine, no fallback available for ...
##[error]Session aborted early after failure on ... (rate-limit on fallback engine).

• Where: review-batch.sh engine fallback logic; review-one-pr.sh run_agentic / triage tier
• Root cause: Claude hourly quotas are exhausted (especially claude-opus-4-7 in single-review/audit; claude-haiku-4-5 in run docs(CLAUDE): record dev-lead/pr-review automation rules #714 at 04:46Z near the 5am UTC reset). Gemini is not installed/configured. Copilot is invoked as fallback but immediately fails with the CLI format error (Pattern A). The script correctly detects the non-zero exit as a rate-limit signal and reaches "no fallback available" → exit 2 → session abort.

Pattern C — Per-PR session abort on old code (runs #708, #711, #712 — FIXED)

Exact error (run #711, step "Review each PR (cascade)"):

##[error]single-review did not produce valid JSON
##[error]Review failed for https://github.com/petry-projects/.github-private/pull/129 (exit code 1)
##[error]Session aborted early after failure on .../pull/129 (exit code 1). Skipped 35 remaining candidate(s); will retry on next scheduled run.
Summary: 0 reviews posted, 0 no-ops skipped, 1 failures (processed 1/36 candidates) [SESSION ABORTED EARLY]

• Where: review-batch.sh exit-code handler (old *) case); review-one-pr.sh single-review validation
• Root cause: Before PR fix: per-PR isolation and single-review retry (closes #132) #133, any per-PR exit code 1 set session_aborted=1 and broke the batch loop. claude-opus-4-7 returned verbose non-JSON for the single confirmation step (likely a truncated/rate-limited response). The single error aborted 35 candidates.
• Status: Fixed in current main (89549a2b) via PR fix: per-PR isolation and single-review retry (closes #132) #133. The *) case in review-batch.sh now increments $failed and continues.

---

4. Token Scope Analysis

Scopes currently present (from gh auth status in all logs):

notifications, read:audit_log, read:discussion, read:org, read:project, repo, workflow

Scopes that appear missing or insufficient:
None — all observed failures are caused by CLI format errors and rate limits, not permission/scope errors. The repo scope covers PR read/write; workflow covers reading workflow status; read:org covers org membership checks for delegation logic.

Recommendation: No token scope changes required. The DON_PETRY_BOT_GH_PAT scopes are sufficient for all operations the workflow performs.

Note on COPILOT_GITHUB_TOKEN: The gh copilot --version check in "Install review engine CLIs" passes silently in all runs (no warning emitted), confirming the Copilot extension is installed and the token has access. The failure is CLI invocation syntax, not auth.

---

5. Recommendations

1. Fix gh copilot invocation syntax [CRITICAL]

• What: Locate every call to gh copilot suggest in scripts/review-one-pr.sh and scripts/review-batch.sh. The current call is gh copilot suggest -p "PROMPT" (or equivalent). Change to use the non-interactive flag pattern: gh copilot -i "suggest -p PROMPT" or pass the prompt via stdin / a temp file. Verify by running gh copilot --help on the runner to confirm current supported syntax.
• Why: Every Copilot invocation fails with "Invalid command format"; the fallback engine is completely dead.
• Expected impact: Restores the fallback chain. Claude rate-limit events will fall to Copilot rather than aborting the session.
• Urgency: CRITICAL — without this fix, every Claude rate-limit event (which occurs regularly, see below) causes a full session abort.

2. Set GOOGLE_API_KEY to enable Gemini as intermediate fallback [HIGH]

• What: Add GOOGLE_API_KEY as a repository/organization secret and ensure it is populated in the workflow env block (already wired: GOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}). The @google/gemini-cli is already installed (best-effort) in the Claude engine path.
• Why: Gemini would absorb Claude rate-limit events before falling to Copilot, reducing Copilot invocations and providing a more reliable secondary engine. Currently GOOGLE_API_KEY: is blank in all runs.
• Expected impact: Reduces frequency of Copilot fallback invocations; provides a working intermediate layer.
• Urgency: HIGH — critical after fixing recommendation test issue from agent #1, but test issue from agent #1 must come first.

3. Tune Claude opus-4-7 usage to reduce hourly rate-limit frequency [MEDIUM]

• What: In review-one-pr.sh, the single-review confirmation step always uses claude-opus-4-7 (ENGINE_SINGLE_MODEL). Consider gating opus use to HIGH-risk PRs only and using sonnet-4-6 for LOW/MEDIUM confirmations. Alternatively, stagger scheduled runs (currently 7 * * * *) to avoid batching heavy opus calls in a short window.
• Why: Runs feat: implement issue #677 — [Phase 1] Thread an optional MCP config into the Claude agentic review tiers (engine.sh) #713–docs(AGENTS): record dev-lead/pr-review automation rules (hands-off label; pr-review is CODEOWNER approver) #715 show rate limits hitting within the same UTC hour window (01:35–07:52). Opus-4-7 is expensive per-token and triggers the fallback chain when quota is exhausted.
• Expected impact: Fewer rate-limit fallback events; more runs complete without triggering the fallback chain.
• Urgency: MEDIUM — less urgent once fallback chain is repaired.

4. Upgrade actions/cache@v4 to Node.js 24-compatible version [LOW]

• What: In .github/workflows/pr-review.yml, update actions/cache@v4 (SHA 0057852b...) to the latest version that supports Node.js 24. Check actions/cache releases for a Node 24-compatible tag.
• Why: All runs emit: ##[warning]Node.js 20 actions are deprecated... Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026. The current pin will break on June 2, 2026.
• Expected impact: Eliminates deprecation warnings; prevents future runner breakage.
• Urgency: LOW — no current failures; deadline is 2026-06-02.

5. Add explicit Gemini CLI version check to install step [LOW]

• What: In the "Install review engine CLIs" step, add a post-install check: if ! command -v gemini >/dev/null 2>&1; then echo "::warning::Gemini CLI install failed — GOOGLE_API_KEY fallback unavailable"; fi. Currently failure is silent until fallback is attempted.
• Why: The warning Gemini fallback unavailable (CLI not installed or GOOGLE_API_KEY missing) appears at review time, not install time, making it harder to diagnose in the logs.
• Expected impact: Earlier visibility on missing Gemini configuration.
• Urgency: LOW.

---

6. Health Score

Health: 3/10 — Copilot fallback is fully broken and GOOGLE_API_KEY is unset, so every Claude rate-limit event guarantees a session abort; the most recent 5 consecutive runs all failed for this reason.