PR Review Agent — failures detected 2026-05-27

[github-actions]

1. Executive Summary

Status: BLOCKING
Period: 2026-05-26 17:07Z – 2026-05-26 17:52Z
Result: 2 of 3 actual run attempts failed (67%); 3 additional runs cancelled by concurrency queue churn

Key findings:

• All three review engines are non-functional simultaneously: Claude rate-limited, Gemini billing exhausted, Copilot triage exiting code 1
• Gemini prepaid credits are fully depleted — a billing issue (not a transient rate limit), persistent until manually resolved
• Copilot fallback fails silently — gh copilot exits 1 at triage with no diagnostic beyond calling gh copilot (model=openai/o4-mini, ...), indicating auth or subscription failure
• 3 cancelled runs (#2252–#2254) are expected concurrency-queue behaviour (not a bug) given 4 runs triggered within 38 seconds for the same PR

Action required: Top up Gemini prepaid credits immediately, then diagnose the Copilot token; without at least one working fallback engine, 100% of reviews fail whenever Claude is rate-limited.

---

2. Failure Breakdown

Failure Category	Affected Runs	Example Error Message
Engine rate limit — Claude	#2251, #2255	You've hit your limit · resets 9:40pm (UTC)
Engine billing exhausted — Gemini	#2251, #2255	Your prepayment credits are depleted. Please go to AI Studio at https://ai.studio/projects to manage your project and billing. (HTTP 429 RESOURCE_EXHAUSTED)
Engine failure — Copilot (exit 1)	#2251, #2255	triage exited with code 1 / [copilot] calling gh copilot (model=openai/o4-mini, timeout=300s, flags=--deny-tool *)
Concurrency queue cancellation	#2252, #2253, #2254	(No error — expected GitHub Actions concurrency behaviour)

---

3. Error Patterns

Pattern A: Claude OAuth Token Rate-Limited

[tier1] triage (claude-haiku-4-5-20251001)
[tier1] usage/rate limit detected on stdout — exiting with code 2 for engine fallback
limit message: You've hit your limit · resets 9:40pm (UTC)
##[warning]Claude rate limit hit — switching to Gemini engine for remaining PRs

Step: Review each PR (cascade) → scripts/review-batch.sh → review-one-pr.sh tier 1 triage
Root cause: The CLAUDE_CODE_OAUTH_TOKEN has exhausted its daily usage allowance. The reset time is 9:40 PM UTC, meaning runs triggered between the limit hit and reset will always cascade to fallback engines. The token's quota is shared across all concurrent runs; rapid re-triggers of the same PR (#343) consumed it quickly.

---

Pattern B: Gemini Prepaid Credits Depleted

Attempt 1 failed with status 429. Retrying with backoff...
"message": "Your prepayment credits are depleted. Please go to AI Studio at
https://ai.studio/projects to manage your project and billing.",
"status": "RESOURCE_EXHAUSTED"
Attempt 10 failed: Your prepayment credits are depleted. Max attempts reached
RetryableQuotaError: Your prepayment credits are depleted.

Step: Review each PR (cascade) — Gemini fallback, tier 1 triage (gemini-2.0-flash)
Root cause: This is not a transient rate limit. The error RESOURCE_EXHAUSTED with "prepayment credits are depleted" means the Google Cloud billing account attached to GOOGLE_API_KEY has run out of prepaid credits. The Gemini CLI retries 10 times with backoff (consuming ~4 minutes per run attempt) before giving up. This error will persist across all runs until credits are replenished.

---

Pattern C: Copilot Engine Triage Exit Code 1

[tier1] triage (o4-mini)
##[warning]triage exited with code 1
    triage stderr:     [copilot] calling gh copilot (model=openai/o4-mini, timeout=300s, flags=--deny-tool *)
##[error]cascade failed at tier 1 (triage process exit 1) for https://github.com/petry-projects/.github/pull/343

Step: Review each PR (cascade) — Copilot fallback (third-tier), tier 1 triage
Root cause: gh copilot exits immediately with code 1. The only stderr output is the invocation line itself — no API error, no auth failure message. This indicates one of:

1. COPILOT_GITHUB_TOKEN (GH_PAT) does not have an active GitHub Copilot subscription or the token lacks copilot scope
2. The --deny-tool * flag or model=openai/o4-mini is not supported by this token's Copilot plan
3. gh copilot extension is installed but the underlying model endpoint is returning an unlogged error

Notably, the install step did not emit the ::warning::gh copilot built-in unavailable with user token guard, meaning gh copilot --version passed — but that check only verifies the CLI exists, not that it can actually call the API successfully.

---

Pattern D: Concurrency Queue Churn (Informational)

Runs #2252, #2253, #2254 were triggered within 38 seconds of each other for the same PR URL. With cancel-in-progress: false, GitHub keeps at most one pending run per concurrency group; each subsequent trigger evicts the previously-queued pending run. This is correct behaviour — #2255 was the last one queued and ran after #2251 completed.

---

4. Token Scope Analysis

From gh auth status in logs:

✓ Logged in to github.com account donpetry-bot (GH_TOKEN)
- Token: github_pat_11CDFSYKQ0w50IHawIgb0w_***

Token	Scopes Currently Evident	Issue
DON_PETRY_BOT_GH_PAT (GH_TOKEN)	Authenticated; pull-requests: write, contents: read, checks: read confirmed via GITHUB_TOKEN permissions block	No issue detected for repo operations
CLAUDE_CODE_OAUTH_TOKEN	Present; rate-limited (daily quota exhausted at time of runs)	Quota exhaustion — not a scope issue
GOOGLE_API_KEY / GEMINI_API_KEY	Present; billing exhausted	Not a scope/auth issue — billing account out of prepaid credits
COPILOT_GITHUB_TOKEN (GH_PAT)	Present; passes gh copilot --version check but fails at runtime	Missing or inactive: Copilot API access. The PAT likely lacks copilot scope or the associated account has no active Copilot for Business/Enterprise seat

Missing / insufficient:

• COPILOT_GITHUB_TOKEN needs either: a GitHub account with an active Copilot subscription, or the copilot OAuth scope on the PAT if using GitHub Models API mode
• No token scope issue for GH_TOKEN — repo auth is working correctly

---

5. Recommendations

1. Top up Gemini prepaid credits [CRITICAL]

• What: Log into Google AI Studio at the billing URL from the error and add prepaid credits to the project associated with GOOGLE_API_KEY
• Why: Until credits are added, Gemini is permanently unavailable as a fallback engine. Every run that hits the Claude rate limit will cascade to Copilot and fail
• Expected impact: Restores the second-tier fallback engine, bringing resilience back to 2-of-3 engines
• Urgency: CRITICAL — this is a persistent outage, not transient

2. Diagnose and fix COPILOT_GITHUB_TOKEN [CRITICAL]

• What: Test the PAT manually: GH_TOKEN=<value of GH_PAT> gh copilot suggest "hello" — verify the account has an active Copilot seat and the token has copilot scope. If the account lacks a seat, assign one or switch to an account that has one
• Why: Copilot is the last-resort engine. With Claude rate-limited and Gemini billing dead, only Copilot can prevent total failure — and it's also failing
• Expected impact: Restores third-tier fallback, making the cascade fully functional when Claude and Gemini are both unavailable
• Urgency: CRITICAL — the entire cascade fails when Claude is rate-limited without a working fallback

3. Consider switching Gemini billing to pay-as-you-go / increase prepaid threshold [HIGH]

• What: In Google AI Studio, switch from a prepaid model to automatic billing or set a higher top-up threshold to prevent credit exhaustion mid-operation
• Why: Prepaid credits depleting mid-run causes 4-minute waste per run (10 retries with backoff) before the cascade moves on. Recurring depletion will continue causing this pattern
• Expected impact: Eliminates the silent 4-minute delay and prevents repeated billing-exhaustion failures
• Urgency: HIGH — recurring cost and latency issue

4. Add Gemini billing depletion to the rate-limit detection heuristic [MEDIUM]

• What: In the script that detects Gemini rate limits and triggers the engine fallback, also match on prepayment credits are depleted / RESOURCE_EXHAUSTED in stderr (it appears this already triggers a fallback to Copilot, but the Gemini CLI retries 10 times first, wasting ~4 min per run)
• Why: The 10-retry backoff loop in the Gemini CLI cannot be bypassed, but if the detection triggers earlier (on first attempt's stderr rather than waiting for all 10), the overall run time improves. Alternatively, detect the billing-depleted pattern before invoking Gemini at all
• Expected impact: Reduces wasted runner minutes when Gemini billing is depleted; faster cascade to Copilot
• Urgency: MEDIUM — optimization, not a blocker once billing is restored

5. Monitor Claude quota reset and consider a Anthropic API key with higher tier limits [LOW]

• What: The CLAUDE_CODE_OAUTH_TOKEN reset time is 9:40pm UTC — if mention-triggered reviews are likely to fire during business hours when quota is exhausted, consider an Anthropic API key with a usage plan rather than an OAuth token with a fixed daily cap
• Why: Multiple rapid re-triggers of the same PR (feat(fleet-monitor): summarize token usage by workflow #343) consumed the daily Claude allowance, forcing cascade to fallback engines
• Expected impact: Reduces frequency of fallback cascades during peak hours
• Urgency: LOW — this is an optimization; the cascade exists for this reason

---

6. Health Score

Health: 2/10 — all three review engines are currently non-functional simultaneously (Claude rate-limited, Gemini billing exhausted, Copilot failing on invoke), making 100% of review attempts fail until Gemini credits are replenished and the Copilot token is fixed.

[don-petry]

All engines are currently rate-limited. Please re-apply the `dev-lead` label when the rate limit clears to retry.