[audit-workflows] Agentic Workflow Fleet Audit — 2026-08-17

[github-actions[bot]]

Window: last 24h (2026-08-16T21:19Z → 2026-08-17T21:06Z) · 364 runs across the fleet, pulled via 4 paginated batches from the agenticworkflows logs MCP tool.

⚠️ Audit gap note: repo-memory's audit-history.jsonl last entry is 2026-07-06 — this audit had not run (or hadn't recorded) for ~6 weeks. Trend charts below show a visible gap between 07-06 and today; treat the intervening weeks as unobserved rather than "healthy."

Fleet health

Metric	Value
Total runs	364
Success	307 (84.3% raw)
Failure	57
Excl. intentional-failure tests	306/362 = 84.5%
AI Credits (AIC)	15,742
Total tokens	16,151,392
GitHub API calls	6,112
Missing tools / missing data	0 / 0

Engine mix: copilot 176, pi 101, claude 65, codex 8, aider 2, crush 2, goose 2 (8 runs had no resolvable engine_id/engine field in aw_info.json/logs — left unclassified rather than inferred from .lock.yml, per audit constraints; all 8 failed, so likely represent driver-startup crashes before the engine identified itself, not a distinct engine).

Today's 84.3% raw success rate sits in the same 82–90% band as the last observed window (07-06, 86.25%) — no fleet-wide regression, but the 6-week blind spot means we can't rule out intermediate incidents.

AIC spend (15.7K) is roughly half of 07-06's 29.6K, consistent with fewer total runs (364 vs 400). Full token totals (16.15M) are available today; historically the fleet had a token-reporting gap (see recommendations.json) — this audit's token figures close that gap for today's window at least.

Top findings

1. PR Sous Chef — 14/59 runs (24%) failed; root cause identified and already being fixed

PR Sous Chef (runs every ~15 min) accounted for the largest single share of fleet failures. Traced two failing runs (32034893678, 32032749148) to full artifact logs:

• The agent completes successfully (posts nudges, creates the run-report issue).
• The safe-outputs job then fails on a single item: update_pull_request #53236 with update_branch: true — ERR_API: update pull request #53236 failed (attempt 1).
• One failed item marks the entire run as failure, even though 3 of 4 safe-outputs (2× add_comment, 1× create_issue) succeeded.
• This exactly matches known-issue safe-output-partial-failure-intolerance (first seen 2026-05, recurrence count now 9) — recurring again today, specifically via update_branch on stacked/conflicting PR Simplify GitHub access modes #53236.
• Good news: PR Treat stacked-PR update_branch API failures as non-fatal in PR Sous Chef safe outputs #53300, "Treat stacked-PR update_branch API failures as non-fatal in PR Sous Chef safe outputs," is already open and addresses this exact failure mode — the sous-chef's own run report even nudged that PR today.

Recommendation: prioritize merging #53300. Separately, consider making the safe-outputs processor treat individual item failures as partial success (report which items failed) instead of reddening the whole run — this is the second time this exact issue class has resurfaced since May.

2. CI Optimization Coach — 81% of network requests blocked (proxy.golang.org)

ci-coach.md has no network: allowlist entry for the Go module proxy, yet the workflow analyzes CI runs that apparently invoke Go tooling: 931 of 1,144 firewall requests (81%) were blocked, 100% of them to proxy.golang.org:443. The run still reported success, so this is a silent capability gap, not a hard failure — but it likely means Go-proxy-derived data is unavailable to the coach's analysis.

Recommendation: add proxy.golang.org (and likely sum.golang.org) to ci-coach.md's network allowlist, matching what Daily Go Test Parallelizer already has configured.

3. Visual Regression Checker — configured localhost allowlist isn't reaching the firewall

visual-regression-checker.md explicitly allows curl (host.docker.internal/redacted) in tools.bashand includeslocalinnetwork.allowed, yet firewall logs show host.docker.internal:4321blocked 15 times across 2 runs (9 and 6 blocked respectively) while runs still reportedsuccess. This suggests the localnetwork category isn't actually permitting egress tohost.docker.internal, so the checker may be silently unable to reach its own preview server — a functional gap masked by a passing status. (The other blocked domains — accounts.google.com, www.google.com, gstatic.com, clients2.google.com, android.clients.google.com` — are routine Chromium/Playwright telemetry and not concerning.)

Recommendation: verify whether network: local is expected to cover host.docker.internal; if not, add an explicit allow entry, then confirm the visual diff step is actually receiving screenshots rather than failing silently.

4. PR Code Quality Reviewer — moderate reliability (33% failure in one window)

Failure rate for this workflow ranged 17–33% across the day's batches (2/6 and 2/17 in different windows). This engine (Pi) matches an existing copilot-sdk-driver-failures known-issue family (long-run 0-turn CLI-exec fails), previously the dominant fleet failure class through early July. Worth continued monitoring — did not deep-dive into per-run logs this cycle given time budget, but flagging as a watch item since it's a repeat name from the known-issues list.

5. Token-usage outliers (single-run, not necessarily problems — but worth a look)

Six workflows burned 800K–2.4M tokens in a single run today: Daily Go Test Parallelizer (361K avg × 6 runs = 2.17M total), Daily Cache Strategy Analyzer (1.38M, 1 run), Issue Arborist (1.25M, 1 run), Dev (1.2M, 1 run), Schema Feature Coverage Checker (1.08M, 1 run), Daily Evals Feature Report (847K, 1 run). None of these failed, so this is a cost-efficiency note rather than a reliability one — worth a spot-check that these aren't retry-looping or re-reading large files unnecessarily.

6. Known issues checked against today's data

• Avenger (known chronic avenger-err-config issue, was 100% failing in early July): improved to 13/15 success (86.7%) today. Recommend downgrading this known-issue's severity/marking as resolved pending one more clean day.
• Smoke CI (known smoke-ci-copilot-cli-100pct-fail-on-push, was the rejig docs #1 fleet-failure driver in July): zero runs observed in today's 24h window — cannot confirm current status; likely no pushes to main during the window, or the workflow was retired/renamed. Needs a direct check next cycle.
• chroot-node-not-available (Daily Issues Report Generator): recurred again today (1/1 failed) — did not have budget to pull detailed logs to confirm same root cause; carrying forward as still-open.

Repo memory

Updated all six repo-memory files at /tmp/gh-aw/repo-memory/default/ and validated size limits via push_repo_memory (7 files, 71 KB total, well under the 60 KB patch-diff limit):

• known-issues.json — bumped safe-output-partial-failure-intolerance (recur 8→9), avenger-err-config-no-structured-logs (recur 18→19, noted recovery), chroot-node-not-available (recur 11→12), copilot-sdk-driver-failures (last-seen refresh); added 2 new issues for the CI Optimization Coach and Visual Regression Checker network/firewall gaps.
• recommendations.json — added 3 new recommendations (PR Treat stacked-PR update_branch API failures as non-fatal in PR Sous Chef safe outputs #53300 priority merge, Go-proxy allowlist, host.docker.internal investigation); reformatted to pretty-printed 2-space JSON.
• workflow-trends.json — replaced with today's per-workflow snapshot (54 workflows with any failure or AIC ≥ 150).
• audit-history.jsonl — appended the 2026-08-17 entry (raw/excl.-intentional rates, per-engine rates, AIC, tokens).
• metrics-summary.json — updated latest_window and appended to history.
• anomalies.json — closed the loop on the Avenger and Smoke CI anomalies with today's data (improvement / observation gap respectively), and added a new anomaly for the PR Sous Chef update_branch cluster, linked to safe-output-partial-failure-intolerance.

Generated by 🔍 Agentic Workflow Audit Agent · agent · 292.5 AIC · ⌖ 7.51 AIC · ⊞ 8.7K · ◷

• expires on Aug 18, 2026, 1:50 PM UTC-08:00

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-08-18T21:50:04.045Z.

Closed by Workflow