PR Review Agent — failures detected 2026-05-02

[petry-projects-pr-review-agent]

1. Executive Summary

Over the last 7 days, 9 of 10 runs succeeded and 1 failed (run #503, 2026-05-01T16:29Z). The failure was a two-stage cascade: Claude Haiku 4.5 hit its hourly quota on the 5th candidate PR, correctly triggering the designed Copilot fallback — but the fallback itself immediately crashed with exit 127 (copilot: No such file or directory) because gh extension install github/gh-copilot silently failed during setup. The extension installer refused to register the name copilot because it conflicts with a gh built-in alias, and the || true guard suppressed the error. This left the fallback engine path permanently broken, causing a session abort that skipped 27 remaining candidates. Urgency level: DEGRADED — day-to-day operation is healthy, but the rate-limit recovery path is non-functional and will session-abort any run that hits Claude quota.

---

2. Failure Breakdown

Failure Category	Affected Runs	Example Error Message
Engine rate limit (Claude quota exhausted)	#503	[tier1] rate limit detected — exiting with code 2 for engine fallback / You're out of extra usage · resets 5pm (UTC)
Copilot CLI not installed — fallback engine broken	#503	timeout: failed to run command 'copilot': No such file or directory (exit 127)
Stale git worktree artifact (post-job warning)	#503	fatal: No url found for submodule path '.claude/worktrees/adoring-chaplygin-ad821c' in .gitmodules
Node.js 20 deprecation (advisory)	All 10	actions/create-github-app-token@v1 running on deprecated Node.js 20

---

3. Error Patterns

Pattern 1 — Claude rate limit (tier 1 triage)

[tier1] triage (claude-haiku-4-5-20251001)
[tier1] rate limit detected — exiting with code 2 for engine fallback
rate limit message: You're out of extra usage · resets 5pm (UTC)

• Step: "Review each PR (cascade)"
• Script: scripts/review-one-pr.sh (Claude engine, tier-1 triage path)
• Root cause: The Claude Haiku 4.5 subscription quota was exhausted mid-batch. This is expected transient behavior; the workflow is explicitly designed to catch exit code 2 and fall over to Copilot. The rate limit itself is not a bug — the broken fallback that follows is.

---

Pattern 2 — Copilot CLI missing (exit 127), blocking fallback

##[warning]triage exited with code 127
timeout: failed to run command 'copilot': No such file or directory
##[error]cascade failed at tier 1 (triage process exit 127) for https://github.com/petry-projects/ContentTwin/pull/104
##[error]Review failed for https://github.com/petry-projects/ContentTwin/pull/104 (exit code 1)

• Step: "Review each PR (cascade)" → Copilot fallback retry

• Script: scripts/review-one-pr.sh (Copilot engine, tier-1 triage path)

• Root cause: During "Install review engine CLIs", the install step ran:

  gh extension install github/gh-copilot || true
  

  The runner printed "copilot" matches the name of a built-in command or alias, indicating gh refused to register the extension under that name due to a naming conflict with an internal alias. Because || true suppresses all non-zero exits, this failure was invisible. When the fallback engine subsequently tried to invoke copilot, the binary did not exist, producing exit 127. This is the direct cause of the session abort and the loss of 27 un-reviewed PRs.

---

Pattern 3 — Stale Claude Code worktree artifact (post-job warning, non-blocking)

fatal: No url found for submodule path '.claude/worktrees/adoring-chaplygin-ad821c' in .gitmodules
##[warning]The process '/usr/bin/git' failed with exit code 128

• Step: Post-job cleanup (actions/checkout cleanup)
• Root cause: A Claude Code git worktree directory (/.claude/worktrees/adoring-chaplygin-ad821c) is committed or present in the repository but is not registered in .gitmodules. The checkout post-job sweep tries to iterate submodules and trips on it. This does not affect the workflow outcome but generates a warning on every run.

---

Pattern 4 — Node.js 20 deprecation (advisory, non-blocking)

Node.js 20 actions are deprecated. The following actions are running on Node.js 20 ... actions/create-github-app-token@v1.
Actions will be forced to run with Node.js 24 by default starting June 2nd, 2026.

• Step: Post-job cleanup
• Root cause: actions/create-github-app-token@v1 pins to a Node.js 20 runtime. This becomes a hard break on 2026-06-02 when GitHub forces Node.js 24 as default.

---

4. Token Scope Analysis

Scopes currently present

From gh auth status and the workflow permissions block:

Source	Scope / Permission	Value
GITHUB_TOKEN (workflow)	contents	read
GITHUB_TOKEN (workflow)	pull-requests	write
GITHUB_TOKEN (workflow)	checks	read
GitHub App token (GH_TOKEN)	Installation token for petry-projects org	Present — confirmed logged in as petry-projects-pr-review-agent[bot]
COPILOT_GITHUB_TOKEN	PAT with Copilot scope	Empty — shown as blank in env dump

Scopes that appear missing or insufficient

Missing Scope / Secret	Evidence	Impact
COPILOT_GITHUB_TOKEN	Env dump shows COPILOT_GITHUB_TOKEN: (blank)	Even if the Copilot CLI were installed, the Copilot engine would fail auth when making API calls — no PAT is wired up
Copilot API access	Blank token + install failure	The entire Copilot fallback path is inoperable

Recommendation per gap

• COPILOT_GITHUB_TOKEN: Set a GitHub PAT with copilot scope as a repository secret. Command: gh secret set COPILOT_GITHUB_TOKEN --body <token> --repo don-petry/pr-review-agent.
• No missing scopes were detected on the GitHub App token side — the auth step passed cleanly and all PR enumeration / review posting completed successfully on prior runs.

---

5. Recommendations

1. Fix gh extension install github/gh-copilot silent failure [CRITICAL]

• What: In .github/workflows/pr-review.yml, the claude) branch of the install step uses gh extension install github/gh-copilot || true. The || true masks the install failure caused by the copilot built-in alias conflict.
• Fix options (pick one):
  • Use --force if supported: gh extension install github/gh-copilot --force || true
  • Capture and explicitly check: replace || true with || echo "::warning::gh-copilot extension install failed — rate-limit fallback will be unavailable" so the failure is visible in logs
  • Use a different invocation path that avoids the alias conflict (check gh extension list after attempted install and gate fallback on presence)
• Why: The current || true makes the CI appear to succeed while leaving the fallback engine broken. Any Claude quota event will cause a session abort instead of a graceful handoff.
• Expected impact: Rate-limit events will recover via Copilot instead of aborting the session and dropping 20–30 unreviewed PRs.
• Urgency: CRITICAL — this is the direct cause of the only failure in the window, and Claude quota events are recurring.

---

2. Set COPILOT_GITHUB_TOKEN secret [HIGH]

• What: COPILOT_GITHUB_TOKEN is empty. Add a GitHub PAT with copilot (and repo) scope.
• Command: gh secret set COPILOT_GITHUB_TOKEN --body <pat> --repo don-petry/pr-review-agent
• Why: Even after fixing the install, the Copilot engine will fail API auth without a valid token. The workflow comment already calls this out (# Store as repo secret: gh secret set COPILOT_GITHUB_TOKEN), so this was likely an oversight during initial setup.
• Expected impact: Copilot fallback will be fully functional end-to-end.
• Urgency: HIGH — prerequisite for the fallback to actually work after fixing item 1.

---

3. Add a pre-flight check that aborts gracefully if fallback engine is unavailable [HIGH]

• What: In scripts/review-one-pr.sh or in the workflow's install step, verify gh extension list | grep -q copilot (or which copilot) after installation. If the Copilot engine is configured as fallback but unavailable, degrade to logging a warning rather than session-aborting.
• Why: The current design session-aborts on any non-zero, non-100, non-2 exit code from review-one-pr.sh. Exit 127 from a missing binary is treated identically to a review logic error, which is overly aggressive. A missing-CLI condition should be detectable and non-fatal (skip the PR, warn, continue).
• Expected impact: Resilience against partial install failures; remaining PRs are processed even when one engine is broken.
• Urgency: HIGH — defensive fix to prevent recurrence of the 27-PR drop.

---

4. Update actions/create-github-app-token to a Node.js 24-compatible version [MEDIUM]

• What: In .github/workflows/pr-review.yml, change:

  uses: actions/create-github-app-token@v1

  to the latest release that supports Node.js 24 (check https://github.com/actions/create-github-app-token/releases).
• Why: GitHub will force Node.js 24 as the default runtime on 2026-06-02. After that date, @v1 will break if it hasn't been updated by the action author.
• Expected impact: Avoids a hard workflow break approximately 30 days from the report date.
• Urgency: MEDIUM — not failing today, but has a hard deadline.

---

5. Remove stale Claude Code worktree from the repository [LOW]

• What: Remove the directory .claude/worktrees/adoring-chaplygin-ad821c from the repository (or add it to .gitignore). Verify .gitmodules does not contain a stale reference to it.
• Why: The actions/checkout post-job cleanup iterates git submodules and trips on this path every run, emitting an exit-128 warning. It's cosmetic today but could mask real submodule errors in future.
• Expected impact: Clean post-job logs; eliminates confusing warning noise.
• Urgency: LOW — no functional impact.

---

6. Health Score

Health: 8/10 — The core review loop is reliable (9/10 runs pass cleanly), but the Claude→Copilot rate-limit failover path is entirely broken due to a silent install failure and missing COPILOT_GITHUB_TOKEN, meaning any Claude quota event will session-abort and drop the remaining candidate queue.

[don-petry]

@claude - Fix these findings.

[claude]

Claude finished @don-petry's task in 6m 11s —— View job • claude/issue-24-20260503-0154 • [Create PR ➔](https://github.com/don-petry/pr-review-agent/compare/main...claude/issue-24-20260503-0154?quick_pull=1&title=Issue%20%2324%3A%20Changes%20from%20Claude&body=This%20PR%20addresses%20issue%20%2324%0A%0AGenerated%20with%20%5BClaude%20Code%5D(https%3A%2F%2Fclaude.ai%2Fcode)

---

Fixing PR Review Agent failures (Issue #24)

• Read CLAUDE.md and understand repo structure
• Read current workflow and scripts
• Fix 4 (LOW): Remove 6 stale Claude Code worktrees from git tracking + add .gitignore
• Commit and push changes
• Create pull request → PR #25

---

What was done

Committed and pushed (non-workflow changes, which the GitHub App can push):

• Removed 6 stale Claude Code worktree entries (.claude/worktrees/*) from git — these triggered fatal: No url found for submodule path warnings on every actions/checkout post-job cleanup
• Added .gitignore entry for .claude/worktrees/ to prevent future worktrees from being committed

---

Workflow changes — apply manually to .github/workflows/pr-review.yml

The GitHub App lacks workflows permission, so these three critical fixes must be applied by a human:

Fix 1 (CRITICAL) — Silent Copilot install failure (line ~73)

-              gh extension install github/gh-copilot || true
+              gh extension install github/gh-copilot --force 2>&1 || true
+              if ! gh extension list 2>/dev/null | grep -q copilot; then
+                echo "::warning::gh-copilot extension install failed — rate-limit fallback will be unavailable. Check for naming conflict with gh built-in alias."
+              fi

Fix 2 (HIGH) — Pre-flight check before fallback (line ~153)

             if [ "$rc" -eq 2 ] && [ "${REVIEW_ENGINE:-claude}" = "claude" ]; then
+              if ! gh extension list 2>/dev/null | grep -q copilot; then
+                echo "::warning::Copilot fallback engine unavailable (not installed) — skipping $pr_url and continuing batch"
+                failed=$((failed + 1))
+                echo "::endgroup::"
+                continue
+              fi
               echo "::warning::Claude rate limit hit — switching to Copilot engine for remaining PRs"
               export REVIEW_ENGINE=copilot

Fix 3 (MEDIUM) — Pin actions/create-github-app-token to Node.js 24-compatible SHA (line ~56)

-        uses: actions/create-github-app-token@v1
+        uses: actions/create-github-app-token@1b10c78c7865c340bc4f6099eb2f838309f1e8c3 # v3.1.1

---

Out of scope (requires human action)

Fix 5 (HIGH) — Set COPILOT_GITHUB_TOKEN secret:

gh secret set COPILOT_GITHUB_TOKEN --body <pat-with-copilot-and-repo-scopes> --repo don-petry/pr-review-agent

To grant the GitHub App workflows permission so future Claude runs can push workflow changes directly, see: FAQ

PR #25 ·

[don-petry]

@claude - The INSTALLATION instructions does not seem to contain directions for a human setup of COPILOT_GITHUB_TOKEN. Enhance the directions to make it clear HOW to create the token including SCOPES and WHICH repo/context should contain the secret.