Skip to content

PR Review Agent: make failure runs diagnosable and harden transient setup failures - #353

Closed
don-petry with Copilot wants to merge 15 commits into
mainfrom
copilot/investigate-github-log-retention
Closed

PR Review Agent: make failure runs diagnosable and harden transient setup failures#353
don-petry with Copilot wants to merge 15 commits into
mainfrom
copilot/investigate-github-log-retention

Conversation

Copilot AI commented May 21, 2026

Copy link
Copy Markdown
Contributor

Recent PR Review Agent failures were non-diagnosable because affected runs had no retrievable logs/jobs, and failures appeared transient in a narrow time window. This PR improves observability for failed runs and reduces setup fragility in the CLI install phase.

  • Auth visibility + scope enforcement

    • Adds an explicit early Verify auth scopes step in pr-review.yml.
    • Emits gh auth status output and fails fast if required scopes are missing (repo, read:org), so auth regressions are visible at the top of the run.
  • Failure diagnostics artifact

    • Adds a dedicated if: failure() artifact upload step after cascade review.
    • Captures candidate/debug outputs (prs.txt, cascade logs, legacy review logs) from both /tmp and ${{ runner.temp }} paths with 14-day retention.
  • Transient npm install resilience

    • Introduces a small retry wrapper for global npm CLI installs used by review engines.
    • Applies retry behavior to Claude/Gemini installs to reduce one-off network/registry failures from failing the entire run.
- name: Upload debug log on failure
  if: failure()
  uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
  with:
    name: pr-review-debug-${{ github.run_number }}
    path: |
      prs.txt
      /tmp/cascade/**
      /tmp/review-*.log
      ${{ runner.temp }}/cascade/**
      ${{ runner.temp }}/review-*.log
    if-no-files-found: ignore
    retention-days: 14

Copilot AI linked an issue May 21, 2026 that may be closed by this pull request
Copilot AI changed the title [WIP] Investigate GitHub log retention issues for PR Review Agent PR Review Agent: make failure runs diagnosable and harden transient setup failures May 21, 2026
Copilot AI requested a review from don-petry May 21, 2026 20:37
@don-petry
don-petry marked this pull request as ready for review May 21, 2026 20:39
Copilot AI review requested due to automatic review settings May 21, 2026 20:39
@don-petry

Copy link
Copy Markdown
Collaborator

Dev-Lead — rate-limited (intent: fix-bot-comment)

PR: #353
Please re-trigger manually (re-mention @dev-lead) when the rate limit clears — the original request cannot be reconstructed automatically.

@gemini-code-assist

Copy link
Copy Markdown

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@don-petry

Copy link
Copy Markdown
Collaborator

Dev-Lead — rate-limited (intent: human)

PR: #353
Please re-trigger manually (re-mention @dev-lead) when the rate limit clears — the original request cannot be reconstructed automatically.

@don-petry

Copy link
Copy Markdown
Collaborator

Note

@don-petry I received your request but all AI engines are currently rate-limited. Please re-mention @dev-lead when the rate limit clears (estimated: unknown) — I cannot reconstruct the original instruction automatically.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves the diagnosability and resilience of the PR Review Agent workflow by making authentication problems visible early, retrying transient npm install failures, and uploading failure-only debug artifacts.

Changes:

  • Add an early “Verify auth scopes” step that prints gh auth status output and fails fast when required scopes are missing.
  • Add a small npm_install_retry wrapper and use it for Claude/Gemini CLI installs.
  • Upload a debug artifact on workflow failure containing candidate lists and cascade/legacy logs from /tmp and ${{ runner.temp }}.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread .github/workflows/pr-review.yml Outdated
Comment on lines +183 to +184
auth_status="$(gh auth status 2>&1)"
echo "$auth_status"
Comment thread .github/workflows/pr-review.yml Outdated
Comment on lines +300 to +305
# Current scripts emit diagnostics under /tmp/cascade.
prs.txt
/tmp/cascade/**
# Legacy/supplemental review logs from earlier review scripts.
/tmp/review-*.log
# Keep runner.temp paths too for cross-runner compatibility.
@don-petry

Copy link
Copy Markdown
Collaborator

@copilot - review, analyze, address and resolve the comments. Ensure CI is passing.

Copilot AI commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

@copilot - review, analyze, address and resolve the comments. Ensure CI is passing.

Addressed and pushed in 7c1af5e. I fixed both actionable workflow review issues: (1) gh auth status output is now captured and echoed even on failure before exiting, and (2) the artifact path literal now contains only real paths (no inline comment entries). I also rechecked CI logs via GitHub Actions MCP: prior failures were from the old commit state, and the latest pushed changes passed local targeted validations plus parallel Code Review/CodeQL.

@don-petry

Copy link
Copy Markdown
Collaborator

Dev-Lead — rate-limited (intent: human-pr)

PR: #353
The retry cron will re-attempt automatically.

@don-petry

Copy link
Copy Markdown
Collaborator

Note

I received your request but all AI engines are currently rate-limited. I'll retry automatically once the rate limit clears.
Rate limit resets at: unknown

@don-petry

Copy link
Copy Markdown
Collaborator

Dev-Lead — rate-limited (intent: fix-bot-comment)

PR: #353
Please re-trigger manually (re-mention @dev-lead) when the rate limit clears — the original request cannot be reconstructed automatically.

@don-petry

Copy link
Copy Markdown
Collaborator

Dev-Lead — rate-limited (intent: human)

PR: #353
Please re-trigger manually (re-mention @dev-lead) when the rate limit clears — the original request cannot be reconstructed automatically.

@don-petry

Copy link
Copy Markdown
Collaborator

Note

@don-petry I received your request but all AI engines are currently rate-limited. Please re-mention @dev-lead when the rate limit clears (estimated: unknown) — I cannot reconstruct the original instruction automatically.

@don-petry

Copy link
Copy Markdown
Collaborator

Dev-Lead — human-pr (no-changes)

No changes were needed for this PR.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 64a7ac5e27

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/pr-review.yml Outdated
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
# Collect both current cascade logs and legacy/supplemental log paths.
with:
name: pr-review-debug-${{ github.run_number }}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Make failure artifact name unique per rerun attempt

Using pr-review-debug-${{ github.run_number }} can make this step fail on workflow re-runs, because github.run_number is stable across re-run attempts while upload-artifact@v4+ fails when an artifact with the same name already exists unless overwrite: true is set. In a fail-then-rerun scenario, the second failed attempt may not upload diagnostics at all due to a 409 conflict, which undermines the new observability path this change is adding.

Useful? React with 👍 / 👎.

@don-petry

Copy link
Copy Markdown
Collaborator

Dev-Lead — human-pr (applied)

Changes committed and pushed.

@don-petry

Copy link
Copy Markdown
Collaborator

No description provided.

don-petry pushed a commit that referenced this pull request Jun 15, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 18, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 21, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 23, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 23, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 23, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 23, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 25, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 25, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 25, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 25, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Jun 25, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 2, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 3, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 3, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 7, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 7, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 7, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 7, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 7, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 7, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 7, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 8, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 8, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 8, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
don-petry pushed a commit that referenced this pull request Aug 8, 2026
…rdening (resolve conflicts)

- Improve pr-review workflow failure diagnosability
- Add early auth scope verification
- Retry transient npm install failures
- Upload debug artifacts on failure
- Harden setup resilience

Closes #353

Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PR Review Agent — failures detected 2026-05-21

4 participants