PR Review Agent — failures detected 2026-06-17

[github-actions]

1. Executive Summary

Status: DEGRADED
Period: 2026-06-17T05:31:01Z – 2026-06-17T05:36:29Z
Result: 2 of 5 completed runs failed (40%); 95 of 100 runs cancelled (expected — concurrency group behaviour)

Key findings:

• claude-fable-5 model throttled on every invocation (rc=1), produces conversational output instead of JSON, causing 100% failure of the single-reviewer confirm path
• GitHub GraphQL rate limit was submitted too quickly (addPullRequestReview) triggered when concurrent runs race to post a review for the same PR
• 95 runs cancelled in a ~5-minute window — all targeting PR feat: implement issue #143 — Compliance: secret_scanning_non_provider_patterns #518, indicating event-storm triggering (check_suite + pull_request + pull_request_review firing simultaneously on the same commit)
• Gemini fallback permanently unavailable (prepayment credits depleted), leaving no rate-limit escape hatch

Action required: Replace or validate the claude-fable-5 model identifier in the review cascade config — it is throttled/non-functional and blocks every approve-path review.

---

2. Failure Breakdown

Failure Category	Affected Runs	Example Error Message
Engine rate limit / invalid model	#22802	[claude] model claude-fable-5 throttled (rc=1) — trying next in chain
Engine rate limit / invalid model	#22802	[approve] single-review attempt 1/2 produced no valid JSON (exit 0)
GitHub API rate limit (GraphQL)	#22772	failed to create review: GraphQL: was submitted too quickly (addPullRequestReview)
Gemini fallback unavailable (warning, not hard failure)	#22802, #22772	Gemini fallback unavailable — prepayment credits depleted
Excessive trigger / event storm (cancellations)	#22703–#22801 (95 runs)	N/A — cancelled before execution

---

3. Error Patterns

Pattern A — claude-fable-5 throttled / non-functional

Exact error (run #22802, review-batch.sh → single-reviewer cascade):

[approve] single-review attempt 1/2 produced no valid JSON (exit 0)
I don't see a specific request yet. The session is set up in single-reviewer PR mode, but no `$PR_URL` or task has been given to me.
stderr: ::warning::[claude] model claude-fable-5 throttled (rc=1) — trying next in chain
[approve] retrying in 15s
[approve] single-review attempt 2/2 produced no valid JSON (exit 0)
::warning::single-review failed after 2 attempts — flagging for manual review

Root cause: The model ID claude-fable-5 is either invalid (no such model exists in the Claude lineup as of the current stable release), throttled at quota-zero, or the model returns an interactive greeting instead of structured JSON output. Both attempts exit 0 (no hard crash) but the model ignores the structured-review prompt entirely, outputting conversational text. The cascade treats exit-0-with-no-JSON as a throttle signal and warns, but the fallback chain is exhausted after two attempts, causing hard failure.

Pattern B — GitHub GraphQL addPullRequestReview rate limit

Exact error (run #22772, review-batch.sh submission step):

Posting APPROVED review...
failed to create review: GraphQL: was submitted too quickly (addPullRequestReview)
ERROR: gh pr review failed with exit code 1

Root cause: Multiple concurrent runs for PR #518 progressed past triage independently (the concurrency cancel didn't fire in time, or runs were on separate queues). When run #22772 reached the review-post step, the GitHub GraphQL API rejected the submission because another review had been posted within the API's enforced submission interval for the same PR. No retry logic exists at the gh pr review call site.

Pattern C — Gemini credits depleted (warning)

Exact warning (both failure runs, review-batch.sh startup):

Gemini fallback unavailable — prepayment credits depleted — replenish via Google AI Studio (https://aistudio.google.com/billing). When Claude is rate-limited, runs will fall through directly to Copilot.

Root cause: GOOGLE_API_KEY/GEMINI_API_KEY are set but the account has no remaining prepaid credits. This is a degraded-fallback state: when Claude is rate-limited the engine skips Gemini and falls directly to Copilot, reducing resilience.

Pattern D — Event storm producing 95 cancelled runs

All 100 runs target PR #518 and are triggered within a 5-minute 28-second window (05:31:01–05:36:29). The workflow triggers on check_suite, pull_request (synchronize/ready_for_review/reopened), and pull_request_review (submitted/dismissed). A single push + CI completion + advisor-bot comments on PR #518 would fire all three event types in rapid succession, each spawning a new run and cancelling the previous one via the concurrency group. The 3 successful runs (#22720, #22800, #22801) correspond to the runs that reached completion before being cancelled.

---

4. Token Scope Analysis

From gh auth status output in both failure run logs:

Scope	Present	Status
repo	✓	Sufficient for PR operations
read:org	✓	Required for team-based reviewer queries
workflow	✓	Required for workflow dispatch
notifications	✓	Not required, harmless
read:audit_log	✓	Not required, harmless
read:discussion	✓	Not required, harmless
read:project	✓	Not required, harmless

Scope validation step exits cleanly in both failure runs — token scopes are not a contributing factor to the current failures. The classic PAT has repo + read:org which satisfies the workflow's scope guard.

No missing scopes. No action required on the token.

---

5. Recommendations

1. [CRITICAL] Fix or replace the claude-fable-5 model identifier in the review cascade config

   • What: Locate the cascade model config (likely scripts/review-one-pr.sh or equivalent config file at the pr-review/stable tag) and replace claude-fable-5 with a valid current model (e.g. claude-opus-4-7 or claude-sonnet-4-6) for the single-reviewer confirm/audit step
   • Why: claude-fable-5 is throttled or non-existent; it returns conversational output on every call, breaking the entire approve path — 100% of runs that reach the confirm step will fail
   • Expected impact: Restores the single-reviewer confirm path; runs that pass triage will be able to post reviews
   • Urgency: CRITICAL — blocks every approval in the current stable release

2. [HIGH] Add retry-with-backoff to gh pr review submission

   • What: In the script that calls gh pr review --approve (review-one-pr.sh or equivalent), wrap the call in a retry loop (e.g. 3 attempts, 10–30s exponential backoff) when the GraphQL error was submitted too quickly is detected
   • Why: GitHub enforces a per-PR submission interval; concurrent runs racing to post the same review will hit this. Run #22772 shows a completed, correct review discarded due to no retry
   • Expected impact: Eliminates transient GraphQL rate-limit failures on the review post step
   • Urgency: HIGH — causes valid completed reviews to be silently lost

3. [HIGH] Add a concurrency group to pr-review-trigger.yml to prevent the event storm

   • What: Add a concurrency: block to the trigger workflow (or confirm it is already set in pr-review.yml) scoped to ${{ github.repository }}-${{ github.event.pull_request.number || github.event.check_suite.id }} with cancel-in-progress: true
   • Why: 95 of 100 runs in this window are cancelled — each new event for PR feat: implement issue #143 — Compliance: secret_scanning_non_provider_patterns #518 spawns a full runner and pays queue/startup overhead before being cancelled. A concurrency group collapses these at queue time before a runner is provisioned
   • Expected impact: Reduces wasted runner-minutes by ~95% for active PRs; reduces the likelihood of the was submitted too quickly race
   • Urgency: HIGH — waste and race condition source

4. [MEDIUM] Replenish Gemini API prepaid credits

   • What: Add credits to the Google AI Studio account associated with GOOGLE_API_KEY/GEMINI_API_KEY via https://aistudio.google.com/billing
   • Why: Gemini is the first fallback when Claude is rate-limited; without it, runs fall directly to Copilot, which may have lower capacity or different rate limits. Both failure runs show this warning
   • Expected impact: Restores three-tier fallback resilience (Claude → Gemini → Copilot)
   • Urgency: MEDIUM — not causing current failures but degrades fault tolerance

5. [LOW] Investigate the 4th expected advisory bot that never submits

   • What: Both runs show Only 3/4 bots submitted and fall through on timeout. Identify which 4th bot is expected (configuration likely in the advisory-gate config at pr-review/stable) and confirm whether it is still active or should be removed from the expected-count
   • Why: The timeout fallback works correctly, but if the 4th bot has been removed or is consistently late, the advisory gate is always burning its full timeout window before proceeding
   • Expected impact: Faster advisory gate completion if the expected count is reduced to match actual active bots
   • Urgency: LOW — timeout fallback handles it; cosmetic/efficiency concern

---

6. Health Score

Health: 3/10 — approve-path is completely broken due to an invalid/throttled model identifier, and 95% of runs are wasted by an event storm with no concurrency guard.

[don-petry]

Closing stale daily digest — superseded by the newest snapshot. Digest steps now close prior days automatically (#1204 / #1212).