ci: Make Claude Code Review check advisory - #24306
Open
MartinZikmund wants to merge 3 commits into
Open
Conversation
The review job failing turned the whole PR red even when every build and test job was green — most often for transient API errors that happen before the agent reads any code. Mark the job continue-on-error so its check reports success, and emit a warning annotation on failure so a genuinely broken workflow still surfaces in the run summary. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019enMfAJBfq8XCHVeRFDWXj
github-actions
Bot
added
the
area/build
Contributor
|
Claude encountered an error after 0s —— View job I'll analyze this and get back to you. |
Contributor
There was a problem hiding this comment.
Pull request overview
Makes the Claude Code Review GitHub Actions workflow advisory so transient agent/action failures no longer mark the overall PR checks as failed, aligning with the intent of issue #24305 (CI signal should reflect build/test health, not agent flakiness).
Changes:
- Marks the
reviewjob as non-blocking viacontinue-on-error: true. - Adds a final conditional step that emits a
::warning::annotation when the job has failed, preserving visibility in the run summary.
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
MartinZikmund
enabled auto-merge
August 31, 2026 07:43
Job-level continue-on-error only greens the workflow run — the job's own check run keeps reporting failure, and that is what the PR renders. Move the flag onto the steps, which is what actually makes the job conclude successfully. Skip the agent when the prompt fetch failed rather than reviewing against a truncated prompt, and key the warning annotation off the step outcome since failure() no longer trips. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019enMfAJBfq8XCHVeRFDWXj
Contributor
|
Claude encountered an error after 0s —— View job I'll analyze this and get back to you. |
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019enMfAJBfq8XCHVeRFDWXj
Contributor
|
Claude encountered an error after 0s —— View job I'll analyze this and get back to you. |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.
Suppressed comments (1)
.github/workflows/claude-code-review.yml:121
- The warning step keys off
steps.review.outcome, so when the prompt fetch fails thereviewstep is skipped and the warning message reportsskippedwithout indicating the real cause (prompt load failure). Consider warning on either a prompt-load failure or a review failure, and tailor the message so failures are actionable from the run summary.
This was referenced Aug 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
GitHub Issue: closes #24305
PR Type:
🏗️ Build or CI related changes
What changed? 🚀
Current behavior: when the
Claude Code Reviewjob fails, the pull request's check rollup turns red — even when every Azure Pipelines build and test job passed. The job fails for reasons unrelated to the PR's content: transient API errors, rate limiting, or timeouts. PR #24194 is a recent example — fully green CI, but thereviewjob died after 213 ms withis_error: true,num_turns: 1,total_cost_usd: 0, before the agent looked at a single line of the diff.This PR makes the review advisory:
continue-on-error: trueon the steps, so a failed agent run doesn't fail the job — and therefore doesn't fail thereviewcheck that the PR renders.continue-on-error: trueon the job as well, which keeps the workflow run itself green in the Actions tab. Worth being explicit about the difference, because it is not obvious: the job-level flag alone does not green the PR check — a job's check run still reports its real conclusion, and only the run's rollup is affected. The first commit here got that wrong; the second fixes it.REVIEW.mdfetch failed, rather than reviewing against a truncated prompt.::warning::annotation whenever the review step didn't succeed, so a genuinely broken workflow (bad action pin, missingREVIEW.md, expired key) still surfaces in the run summary rather than failing silently. It keys offsteps.review.outcomebecausefailure()no longer trips once the step is marked continue-on-error.The review comments the agent posts on the PR are unaffected — only the check status changes.
Tradeoff: the check can no longer block a merge, even for a real configuration error. The warning annotation and the job log are the remaining signal. One deliberate hole is left: a failure of the
Checkout repositorystep is still red, since that means the runner itself is broken rather than the review being flaky.Security posture unchanged: none of the five defenses documented in the workflow header are touched — the trigger gate, tool allowlist, absence of
id-token: write, the pinned action SHA, and the base-branch prompt all stay exactly as they were.PR Checklist ✅
Screenshots Compare Test Runresults.