Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
152 changes: 152 additions & 0 deletions .github/workflows/ci-failure-analyst-reusable.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
# This file is the source of truth. Callers: .github/workflows/ci-failure-analyst.lock.yml (this repo)
# and any repo deploying templates/ci-failure-analyst.yml.
name: "CI Failure Analyst — Reusable"

on:
workflow_call:
secrets:
CLAUDE_CODE_OAUTH_TOKEN:
required: true

jobs:
analyze:
runs-on: ubuntu-latest
timeout-minutes: 15
env:
CLAUDE_CODE_OAUTH_TOKEN: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
Comment thread
don-petry marked this conversation as resolved.

steps:
- name: Resolve non-fork PR
id: pr
env:
HEAD_SHA: ${{ github.event.check_run.head_sha }}
CHECK_RUN_PRS: ${{ toJson(github.event.check_run.pull_requests) }}
run: |
set -euo pipefail
# Prefer the pull_requests array embedded in the check_run payload;
# filter to non-fork PRs (head repo must equal base repo).
pr_number=$(echo "$CHECK_RUN_PRS" \
| jq -r '.[] | select(.head.repo.id == .base.repo.id) | .number' \
| head -n 1)
if [ -z "$pr_number" ]; then
# Fallback: commits-to-pulls API handles edge cases (e.g. a second
# push arrived before this check_run completed).
# shellcheck disable=SC2016 # $ENV.REPO is jq built-in syntax, not a shell variable
pr_number=$(gh api "repos/$REPO/commits/$HEAD_SHA/pulls" \
--jq '.[] | select(.head.repo.full_name == $ENV.REPO) | .number' \
2>/dev/null | head -n 1 || true)
fi
if [ -z "$pr_number" ]; then
echo "::notice::No non-fork PR for SHA $HEAD_SHA — skipping"
echo "skip=true" >> "$GITHUB_OUTPUT"
else
echo "skip=false" >> "$GITHUB_OUTPUT"
echo "pr_number=$pr_number" >> "$GITHUB_OUTPUT"
fi

- name: Check idempotency
if: steps.pr.outputs.skip == 'false'
id: idem
env:
PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
HEAD_SHA: ${{ github.event.check_run.head_sha }}
run: |
set -euo pipefail
existing=$(gh api "repos/$REPO/issues/$PR_NUMBER/comments" \
--paginate \
--jq ".[] | select(.body | startswith(\"<!-- ci-analyst sha=$HEAD_SHA -->\")) | .id" \
2>/dev/null | head -1 || true)
if [ -n "$existing" ]; then
echo "::notice::Diagnostic comment already posted (id=$existing) — idempotency skip"
echo "exists=true" >> "$GITHUB_OUTPUT"
else
echo "exists=false" >> "$GITHUB_OUTPUT"
fi

- name: Install Claude Code
if: steps.pr.outputs.skip == 'false' && steps.idem.outputs.exists == 'false'
env:
CLAUDE_CODE_VERSION: ${{ vars.CLAUDE_CODE_VERSION || 'latest' }}
run: |
mkdir -p "$HOME/.npm-global"
npm config set prefix "$HOME/.npm-global"
echo "$HOME/.npm-global/bin" >> "$GITHUB_PATH"
npm install -g "@anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}"

- name: Analyze failure and post comment
if: steps.pr.outputs.skip == 'false' && steps.idem.outputs.exists == 'false'
env:
PR_NUMBER: ${{ steps.pr.outputs.pr_number }}
HEAD_SHA: ${{ github.event.check_run.head_sha }}
WORKFLOW_NAME: ${{ github.event.check_run.name }}
DETAILS_URL: ${{ github.event.check_run.details_url }}
CHECK_RUN_ID: ${{ github.event.check_run.id }}
run: |
set -euo pipefail
export PATH="$HOME/.npm-global/bin:$PATH"

cat > /tmp/prompt-template.txt << 'TMPL'
A CI check named "$WORKFLOW_NAME" just failed on PR #$PR_NUMBER in repository $REPO.

Key facts:
- PR number: $PR_NUMBER
- Head SHA: $HEAD_SHA
- Workflow/check name: $WORKFLOW_NAME
- Details URL: $DETAILS_URL
- Check run ID: $CHECK_RUN_ID

Follow these steps in order:

### 1. Fetch failure logs
Extract the GitHub Actions run ID from the details URL:
run_id=$(echo "$DETAILS_URL" | grep -oP '(?<=runs/)\d+' || true)
If a run_id is found, fetch the failed-step logs:
gh run view "$run_id" --log-failed --repo $REPO 2>/dev/null | tail -200
If no run_id (e.g. external quality gate like SonarCloud), use the PR diff:
gh pr diff $PR_NUMBER --repo $REPO 2>/dev/null | head -200

### 2. Identify the failing step and error
From the logs, extract:
- The specific step name that failed
- The key error message or assertion

### 3. Classify the root cause into exactly one category:
| Category | Indicators |
|---|---|
| Test failure | Assertion failed, FAIL, AssertionError, Expected...got... |
| Env issue | Missing secret/env var, command not found, permission denied, auth error |
| Flaky test | Timeout, ECONNREFUSED, connection reset, intermittent HTTP error |
| Config error | Malformed YAML, missing required field, invalid syntax, wrong runner |
| Lint/style | eslint, shellcheck, markdownlint, yamllint, ruff, golangci-lint errors |
| Build error | Compilation error, npm ERR!, cargo build failed, dependency resolution failure |

If the workflow name contains lint, shellcheck, markdownlint, yamllint, or similar -> must classify as Lint/style.

### 4. Post a comment on PR #$PR_NUMBER using bash:
gh api "repos/$REPO/issues/$PR_NUMBER/comments" \
-f body='<!-- ci-analyst sha=$HEAD_SHA -->
## CI Failure: $WORKFLOW_NAME

**Step:** FAILING_STEP_NAME
**Root cause:** ROOT_CAUSE_CATEGORY

BRIEF_EXPLANATION (2-3 sentences: what failed, why it likely failed, what the error means)

**Suggested fix:** ONE_CONCRETE_ACTION

[View run logs]($DETAILS_URL)'

Rules:
- The <!-- ci-analyst sha=$HEAD_SHA --> marker MUST be the very first line of the comment body
- Root cause must be one of the six categories above
- Suggested fix must be one specific, actionable step (not "fix the tests")
- Keep the entire comment under 300 words
TMPL

PROMPT=$(envsubst < /tmp/prompt-template.txt)
claude --print \
--permission-mode bypassPermissions \
--allowedTools Bash \
-p "$PROMPT"
156 changes: 15 additions & 141 deletions .github/workflows/ci-failure-analyst.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading