[CI Failure Doctor] Recurring CI failure: Check Go formatting step fails - 5th occurrence today

[github-actions]

🏥 CI Failure Investigation - Run #29602

Summary

⚠️ RECURRING PATTERN - This is the 5th occurrence of this failure type today.

The CI workflow failed because Go code was not properly formatted before committing. The "Check Go formatting" step in the lint-go job detected unformatted Go files.

Failure Details

• Run: #21012388902 (Run #29602)
• Commit: 030f506d2354f66024247585b91e19301f9ddbf6
• PR: Change codespace installation callout from CAUTION to TIP in quickstart #9997 - "Change codespace installation callout from CAUTION to TIP in quickstart"
• Trigger: push to main branch
• Author: Copilot (AI agent)
• Failed Job: lint-go (Job #60410207744)
• Failed Step: "Check Go formatting" (step 5)
• Time: 2026-01-14 22:37:59 UTC

Root Cause Analysis

Primary Cause

The AI agent (Copilot) committed code changes without running the mandatory pre-commit command: make agent-finish (or at minimum make fmt).

Pattern Recognition

This failure matches the known pattern GO_FORMAT_CHECK_FAILED with high confidence:

• Pattern first seen: 2026-01-14 06:08:42 UTC
• Last occurrence (before this): 2026-01-14 07:51:41 UTC
• Total occurrences today: 5
• Pattern signature: lint-go job → Check Go formatting step failure

Why This Keeps Happening

1. AI agents are not consistently following the documented workflow that requires running make agent-finish before every commit
2. No automated enforcement - there are no pre-commit hooks or branch protection rules that enforce formatting
3. False sense of completion - the AI agent successfully completes its task (e.g., doc changes) but doesn't realize it needs to format code

Suspicious Commit Characteristics

The PR #9997 only changed 2 files:

• docs/src/content/docs/setup/quick-start.md (the intended documentation change)
• .github/workflows/glossary-maintainer.lock.yml (regenerated workflow file)

However, the commit statistics show hundreds of Go files modified. This suggests either:

• A merge commit that brought in unformatted changes from another branch
• A grafted commit in a shallow clone

The formatting check caught the issue regardless of the source.

Failed Jobs and Errors

Job: lint-go

Status: ❌ Failed
Duration: ~17 seconds (22:38:03 - 22:38:20)
Failed Step: Check Go formatting (step 5)

Step Flow:

✅ 1. Set up job
✅ 2. Checkout code
✅ 3. Set up Go
✅ 4. Report Go cache status
❌ 5. Check Go formatting (FAILED)
⏭️ 6. Install golangci-lint (SKIPPED)
⏭️ 7. Run golangci-lint (SKIPPED)
⏭️ 8. Lint error messages (SKIPPED)

Expected Error (based on pattern):

❌ Code is not formatted. Run 'make fmt' to fix.
Unformatted files:
[list of files]

To fix this locally, run:
  make fmt

Investigation Findings

Historical Context

This is part of a systematic issue where AI agents repeatedly commit code without proper formatting:

Date	Run ID	Commit	PR	Author
2026-01-14 06:08	20984232736	b1b5f4b	#9903	Copilot
2026-01-14 06:14	20984290612	99c636f	#9894	Copilot
2026-01-14 06:28	20984405453	(unknown)	(unknown)	Copilot
2026-01-14 07:51	20986559929	(unknown)	(unknown)	Copilot
2026-01-14 22:37	21012388902	030f506	#9997	Copilot

Pattern: All failures are from Copilot (AI agent) commits.

Impact Assessment

• Severity: Medium (doesn't break functionality, but blocks CI)
• Impact: High (all CI checks fail, blocks other PRs, wastes CI resources)
• Urgency: High (recurring issue needs systematic fix)
• Time to fix: ~5 minutes (run make fmt and push)

Recommended Actions

Immediate Fix (For This Specific Failure)

# Fix the formatting issue
make fmt
git add .
git commit -m "Fix Go formatting"
git push

Short-term Prevention (Reduce Recurrence)

• Strengthen AI agent instructions: Add more prominent warnings in AGENTS.md about mandatory make agent-finish
• Add validation check: Create a reminder in the AI agent prompt about pre-commit requirements
• Update PR template: Add a checklist item for make agent-finish

Long-term Solutions (Eliminate Recurrence)

• Add pre-commit hooks: Install Git hooks that automatically run make fmt before commits

  # .git/hooks/pre-commit
  #!/bin/sh
  make fmt
  git add .

• Auto-formatting PR workflow: Add a GitHub Action that automatically formats code and commits it

  name: Auto-format
  on: pull_request
  jobs:
    format:
      runs-on: ubuntu-latest
      steps:
        - uses: actions/checkout@v4
        - name: Format code
          run: make fmt
        - name: Commit changes
          run: |
            git config user.name "github-actions[bot]"
            git config user.email "github-actions[bot]@users.noreply.github.com"
            git add .
            git diff-index --quiet HEAD || git commit -m "Auto-format code"
            git push

• Branch protection: Require the lint-go check to pass before allowing merges

• AI agent improvements: Modify the AI agent's workflow to automatically run make agent-finish before creating PRs

Prevention Strategies

AI Team Self-Improvement

Add the following to AI agent instructions:

## MANDATORY PRE-COMMIT CHECKLIST

Before EVERY commit that modifies Go files (.go, go.mod, go.sum):

1. ✅ Run `make agent-finish` (runs build, test, recompile, fmt, lint)
2. ✅ Verify no errors from the above command
3. ✅ Only then commit and push

**CRITICAL**: This is NOT optional. CI will fail if you skip this step.

**Why this matters**: The formatting check is automatic and strict. Unformatted
code will cause CI to fail, blocking all other work and wasting resources.

**Remember**: `make agent-finish` is your friend - it catches issues BEFORE CI does.

Monitoring and Alerts

• Alert threshold: If this pattern occurs more than 2 times per day, escalate to team
• Current status: ⚠️ THRESHOLD EXCEEDED (5 occurrences today)
• Recommendation: Implement automated fixes immediately

Historical Context

Similar past failures and their resolutions:

• [CI Failure Doctor] Check Go formatting step failed - code not properly formatted #9913: [CI Failure Doctor] Check Go formatting step failed (closed)
• [ca] Fix test expectations and recompile workflows #9184: Fix test expectations and recompile workflows (closed)
• [ca] Fix test failures after code formatting #7545: Fix test failures after code formatting (closed)

Lesson: This is not a new problem. Manual fixes are not sufficient - we need systematic prevention.

Related Files and Patterns

• Pattern database: /tmp/gh-aw/cache-memory/patterns/go-formatting-failure.json
• Investigation reports:
  • 2026-01-14-20984232736.json
  • 2026-01-14-20984290612.json
  • 2026-01-14-20984405453.json
  • 2026-01-14-20986559929.json
  • 2026-01-14-21012388902.json (this failure)

Conclusion

This failure is 100% preventable and is part of a recurring pattern. The root cause is clear: AI agents are not following the documented pre-commit workflow.

While individual fixes are quick (5 minutes), the recurring nature means we're wasting significant time and CI resources. Systematic prevention is urgently needed.

---

Investigation completed by CI Failure Doctor
Pattern: GO_FORMAT_CHECK_FAILED (confidence: high)
Auto-generated: 2026-01-14T22:45:00Z

AI generated by CI Failure Doctor

To add this workflow in your repository, run gh aw add githubnext/agentics/workflows/ci-doctor.md@ea350161ad5dcc9624cf510f134c6a9e39a6f94d. See usage guide.

[github-actions]

🔁 Duplicate Failure Detected - Run #29605

Another occurrence of this same formatting failure pattern has been detected:

Run Details

• Run: #21012664370 (Run #29605)
• Commit: 04dcdf20f394e65b4b870c635304adcc3724592b
• PR: Fix read-only filesystem errors in check_workflow_recompile_needed tests #9979 - "Fix read-only filesystem errors in check_workflow_recompile_needed tests"
• Time: 2026-01-14 22:48:53 UTC
• Failed Job: lint-go → Failed Step: "Check Go formatting"

Analysis

This failure is identical to the pattern documented in this issue:

• ✅ Same failed job: lint-go
• ✅ Same failed step: "Check Go formatting" (step 5)
• ✅ Same root cause: Code committed without running make fmt

Key Observation

The commit only changed JavaScript files:

• actions/setup/js/check_workflow_recompile_needed.cjs
• actions/setup/js/check_workflow_recompile_needed.test.cjs

But the Go formatting check still failed, confirming that unformatted Go code already existed in the repository and was not caught by previous commits.

Occurrence Count

This is now the 6th confirmed occurrence of this exact failure pattern. The issue is clearly systematic and requires the automated prevention measures described in this issue.

Recommendation

The recurring nature of this failure confirms that manual reminders are insufficient. The long-term solutions outlined in this issue (pre-commit hooks, auto-formatting workflows, or branch protection requiring lint-go to pass) should be prioritized for implementation.

---

Auto-detected by CI Failure Doctor - Run 21012803437

AI generated by CI Failure Doctor

[github-actions]

🔄 6th Recurrence Detected - Run #29613

⚠️ The pattern continues: Another Go formatting failure just occurred less than 1 hour after run #29602.

This Failure

• Run: #21013538352
• Commit: ea482165ec4cff339d3991928309f53bf6a4a94a
• PR: Fix TOCTOU vulnerability in changeset.js file operations #10001 - "Fix TOCTOU vulnerability in changeset.js file operations"
• Author: Copilot (AI agent)
• Time: 2026-01-14 23:24:41 UTC
• Pattern Match: ✅ 100% match to GO_FORMAT_CHECK_FAILED

Key Finding

This PR only modified JavaScript (scripts/changeset.js), yet the Go formatting check failed. This suggests:

1. The commit was grafted/merged with pre-existing unformatted Go files
2. The AI agent did not run make agent-finish before the merge
3. The formatting issues were already on main or in the PR branch

Escalating Problem

Timeline of failures today:

1. Run #20984232736 (06:08 UTC) - Initial occurrence
2. Run #20984290612 (06:14 UTC) - 6 minutes later
3. Run #20984405453 (06:28 UTC) - 14 minutes later
4. Run #20986559929 (07:51 UTC) - 1 hour 23 minutes later
5. Run #21012388902 (22:37 UTC) - 14+ hours gap
6. Run #21013538352 (23:24 UTC) - Less than 1 hour later ⚠️

The problem is accelerating in the evening hours.

Immediate Action Needed

Someone needs to run make fmt on main and push the formatting fixes NOW to break this cycle. Each subsequent PR is likely carrying forward these unformatted files.

---

See full investigation in attached memory cache: 2026-01-14-21013538352.json

AI generated by CI Failure Doctor

[github-actions]

🍪 Issue Monster has assigned this to Copilot!

I've identified this issue as a good candidate for automated resolution and assigned it to the Copilot agent.

The Copilot agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

AI generated by Issue Monster

[github-actions]

🔴 Issue NOT Resolved - 7th Consecutive Failure

⚠️ Critical: This issue was closed prematurely. The formatting failures are continuing after the fix from PR #10008 was merged.

Latest Failure

• Run: #21015057323 (Run #29627)
• Commit: 1c862ded0b7a1b70110c44a7e0607cb1f9769194
• PR: Add agentic workflow to auto-fix high severity code scanning alerts every 30 minutes #10005 - "Add agentic workflow to auto-fix high severity code scanning alerts every 30 minutes"
• Time: 2026-01-15 00:31:13 UTC
• Failed Job: lint-go → Failed Step: "Check Go formatting"
• Pattern Match: ✅ 100% match to GO_FORMAT_CHECK_FAILED

Timeline of Consecutive Failures

All failures occurred AFTER PR #10008 was merged:

Run #	Time (UTC)	Commit	PR #	Status
29623	00:16:46	df2c140	#9962	❌ Failed
29624	00:18:20	66259c3	#10008	❌ Failed (this was the "fix" PR)
29625	00:20:15	48fb2b5	#9978	❌ Failed
29626	00:30:16	fd6a8a2	#10010	❌ Failed
29627	00:31:13	1c862de	#10005	❌ Failed (this run)

Root Cause - Why the Fix Didn't Work

The fix in PR #10008 enhanced the warning text in AGENTS.md, but:

1. Documentation changes alone are insufficient - AI agents need automated enforcement
2. No pre-commit hooks - Nothing prevents unformatted code from being committed
3. No workflow validation - The compiled .lock.yml files can be committed without formatting checks
4. Immediate failure of the "fix" PR - Even PR Enhance AGENTS.md formatting warnings to prevent recurring CI failures #10008 itself failed the formatting check (run #29624)

Evidence This Is Systematic

The pattern shows:

• ✅ All failures are from Copilot AI agent commits
• ✅ Consistent failure point: lint-go job, "Check Go formatting" step
• ✅ 100% preventable: All failures could have been avoided by running make fmt
• ✅ Continues after attempted fix: 4 more failures after PR Enhance AGENTS.md formatting warnings to prevent recurring CI failures #10008

Required Actions

Immediate (Stop the Bleeding)

1. Run formatting on main branch NOW:

   git checkout main
   git pull
   make fmt
   git add .
   git commit -m "Fix Go formatting - emergency fix for CI failures"
   git push

2. Block new PRs until this is fixed - Consider requiring manual review

Short-term (Prevent Immediate Recurrence)

• Add pre-commit hooks to the repository template
• Add workflow validation that formats code before compilation
• Update AI agent workflow to always run make fmt before creating PRs

Long-term (Eliminate Root Cause)

• Automated formatting workflow: Create a GitHub Action that automatically formats code on every PR
• Pre-merge validation: Require formatting checks to pass before merge
• AI agent architecture change: Make make agent-finish part of the agent's internal workflow, not a manual step

Why This Is Critical

Cost of inaction:

• 🔥 5+ CI failures per hour during active development
• ⏱️ ~6 minutes of CI time wasted per failure
• 🚫 Blocks all other PRs from getting clean CI passes
• 😤 Developer frustration from repeatedly fixing the same issue
• 💰 GitHub Actions minutes wasted on predictable failures

Recommendation

DO NOT CLOSE this issue until:

1. ✅ Main branch passes CI (all Go files formatted)
2. ✅ Automated enforcement is implemented (pre-commit hooks or workflow)
3. ✅ At least 24 hours pass without a formatting-related CI failure

---

Investigation by CI Failure Doctor
Run: https://github.com/githubnext/gh-aw/actions/runs/21015187293
Pattern: GO_FORMAT_CHECK_FAILED (7th occurrence)

AI generated by CI Failure Doctor