[audit-workflows] Daily Agentic Workflow Audit — 2026-08-16 #53209
Replies: 3 comments
|
Smoke test: automated interaction check from the Copilot engine validation workflow. All systems nominal! 🤖 Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "www.google.com"
- "www.gstatic.com"See Network Configuration for more information.
|
|
Smoke test says hello! Warning Firewall blocked 6 domainsThe following domains were blocked by the firewall during workflow execution:
To allow these domains, add them to the network:
allowed:
- defaults
- "accounts.google.com"
- "android.clients.google.com"
- "clients2.google.com"
- "contentautofill.googleapis.com"
- "www.google.com"
- "www.gstatic.com"See Network Configuration for more information.
|
|
This discussion has been marked as outdated by Agentic Workflow Audit Agent. A newer discussion is available at Discussion #53499. |
Uh oh!
There was an error while loading. Please reload this page.
Overview
Fleet health is markedly better than the last known baseline from 2026-07-06 (86.25% raw / 82.4% prod-main), but because of the 41-day blind spot, some of that apparent improvement is unverifiable — see below.
Key issues (Clustering)
1. Avenger — chronic
driver_exit, unresolved 2+ months (recurrence #19)Same root cause tracked since ~2026-06-08 (
err-config-no-structured-logs): process exits withTurns=0/ErrorCount=0, no structured logs, no firewall blocking involved. Today: 4/8 Avenger runs failed (50%) in a tight ~1h cluster (04:37–06:47Z) — §31922664876, §31927076597, §31929386213, §31931891565 — interleaved with successful runs immediately before/after. Root cause still not fixed after 2+ months chronic.2.
copilot-sdk-driver-failures— longrun 0-turn family, rotated workflow set (recurrence #25)The chronic "agent does real work for 20–50+ min then job fails 0-turn" pattern persists, but this cycle it hit a different set of workflows than previously logged: Daily Container Image Security Scan (35.8m, aic 438.5), Daily Firewall Logs Collector and Reporter (52.1m, aic 98.4), Daily Regression Audit (29.2m, aic 31.2), Code Scanning Fixer (26.7m,
agent_logic), Linter Miner (10.7m,agent_logic). Some of today's instances classify asagent_logicrather than puredriver_exit— a minor signature drift worth watching.3. Smoke CI (100%-fail-on-push) &
chroot-node-not-available— status unverified, not confirmed fixedBoth were top chronic offenders as of 2026-07-06 but produced zero triggering runs in today's window, so their status cannot be confirmed either way. Absence of failures ≠ confirmed fix.
4. Process finding: 41-day audit cadence gap
This audit last ran 2026-07-06; today is 2026-08-16. A gap this long means any regression or fix that occurred entirely within the gap is invisible to this history, which directly weakens confidence in item #3 above.
Collection phase detail
copilot-sdk-driver-failureslongrun cluster (items above); no other cost anomalies stood out in this window.logsbatches,start_date: -1d.Recommendations
avenger-err-config-no-structured-logs) — unfixed for 2+ months across 19 recorded recurrences. Given the persistence, recommend the owning team consider disabling Avenger's schedule until the err-config root cause is addressed, rather than continuing to burn compute on a ~50%-fail workflow.copilot-sdk-driver-failuresroot cause — still open; the affected-workflow set keeps rotating (recurrence Logs revamp. #25 introduces 5 new names), suggesting this is a shared driver/runtime issue rather than a per-workflow bug. Continue root-cause investigation at the copilot engine/driver level.Trend charts (last ~30 days, with the 41-day gap shown honestly)
Workflow health
Success/failure counts and success-rate line through 2026-07-06, then a shaded gap through 2026-08-16 where no data exists — no line is drawn across it. Today's isolated point (398 success / 22 fail, 94.8%) is direct-labeled rather than connected, since a single point 41 days after the last one doesn't represent a trend. The rate line sits comfortably above the pre-gap average, consistent with the improvement noted above, but this is one data point, not a confirmed sustained trend.
Token usage
Daily token bars with a 7-day moving average computed only over the pre-gap segment (a moving average spanning the gap would be meaningless). Today's 14.3M-token bar is annotated as a single isolated day with no MA, since there aren't 7 consecutive days to average against post-gap.
Repo memory
All six repo-memory files (
audit-history.jsonl,workflow-trends.json,known-issues.json,recommendations.json,anomalies.json,metrics-summary.json) were updated and merged with prior history (not overwritten): recurrence counters incremented for the two chronic issues, two new anomalies recorded (Avenger cluster, audit-cadence gap), and one new recommendation added (restore audit cadence).References:
All reactions