🔍 Agentic Workflow Audit Report - 2026-01-23

[github-actions[bot]]

Audit Summary

⚠️ Data Collection Limitation: The gh-aw MCP server tools (status, logs) were not available during this audit. Analysis is based on cached historical data from the last available workflow runs through 2026-01-21.

Key Metrics (Last 30 Days)

Metric	Value	Trend
Total Runs	30	↔️
Success Rate	76.67%	↗️ Recovering
Total Tokens	31.1M	↘️ Normalizing
Total Cost	$3.98	↘️ Decreasing
Avg Daily Cost	$1.33	-
Data Points	3 (Jan 13, 14, 21)	Limited

Success Rate Trajectory

2026-01-13: ████████████████████                      50.0% ⚠️
2026-01-14: ████████████████████████████████████████ 100.0% ✅
2026-01-21: ████████████████████████████████          80.0% ✅

Analysis: Success rate recovered from a critical low of 50% on Jan 13 to 100% on Jan 14, stabilizing at 80% by Jan 21. This shows positive recovery but needs sustained performance at 80%+ levels.

---

🚨 Critical Issues

1. MCP Infrastructure Failure (2026-01-13)

Severity: CRITICAL

All 6 MCP servers failed simultaneously in the "Smoke Claude" workflow (run 20943805698). This represents the first complete MCP infrastructure failure observed.

Affected Servers:

• github ⚠️ NEW
• playwright ⚠️ NEW
• safeoutputs 🔴 RECURRING (7 total incidents)
• safeinputs ⚠️ NEW
• serena ⚠️ NEW
• tavily ⚠️ NEW

Impact:

• Success rate dropped to 50% on Jan 13
• Token usage spiked to 15.8M (highest in period)
• Complete loss of MCP functionality for affected runs

Status: Partially recovered by Jan 14, but root cause unclear

View MCP Failure Timeline

Date	Affected Servers	Workflows	Status
2025-12-24 to 2026-01-13	safeoutputs (7x)	Smoke Claude	Recurring
2026-01-13	All 6 servers	Smoke Claude	Mass failure event
2026-01-14+	-	-	Recovered

2. Recurring Safeoutputs Failures

Severity: HIGH

The safeoutputs MCP server has failed 7 times between 2025-12-24 and 2026-01-13, indicating a systemic reliability issue.

Recommendation: Investigate safeoutputs server initialization, configuration, and resource constraints.

---

🔧 Missing Tools

The following tools have been requested by workflows but are not currently available:

Tool	Workflows Affected	Count	Status
github-list_code_scanning_alerts	Code Scanning Fixer	1	Active
Safe-input GitHub query tools	Daily Project Performance	1	Active
make, go, build tools	Tidy	5	Recurring
playwright MCP server	Smoke Copilot Playwright	1	Recurring
safeinputs-gh	Smoke Copilot Playwright, Smoke Codex	3	Recurring
playwright_navigate	Smoke Copilot Playwright	1	Recurring
Directory creation capability	Spec-Kit Execute	2	Recurring
Pull request context	Smoke Codex	1	Active

View Missing Tool Details

github-list_code_scanning_alerts

• Workflows: Code Scanning Fixer
• Reason: Required to fetch code scanning alerts for automated fixing
• Impact: Workflow cannot access security findings
• Recommendation: Add to GitHub MCP server tool set

Build Tools (make, go fmt, golangci-lint, npm)

• Workflows: Tidy
• Count: 5 requests
• Reason: Code formatting and linting operations require build toolchain access
• Impact: Cannot perform automated code cleanup
• Recommendation: Evaluate if these tools should be available in workflow environment

Safe-Input GitHub Query Tools

• Workflows: Daily Project Performance
• Reason: Need authenticated GitHub CLI access through safe-input mechanism
• Impact: Cannot query GitHub data securely
• Recommendation: Implement safeinputs-gh tool for secure GitHub operations

---

⚠️ Error Patterns

High-Frequency Patterns

Pattern	Frequency	Severity	Affected Workflows
copilot-permission-denied	90+	Medium	AI Moderator, Issue Monster, Smoke Copilot, others (10 total)
mcp-invalid-url	36	High	Smoke Copilot variants (No Firewall, Playwright, Safe Inputs)
mcp-github-schema-error	1	High	AI Moderator

View Error Pattern Details

copilot-permission-denied

• Message: "Permission denied and could not request permission from user"
• First Seen: 2025-12-25
• Last Seen: 2025-12-26
• Analysis: Indicates workflow execution lacks necessary permissions for certain operations
• Recommendation: Review workflow permissions and permission request mechanisms

mcp-invalid-url

• Message: "Invalid URL / Failed to start MCP client"
• First Seen: 2025-12-25
• Last Seen: 2025-12-26
• Analysis: MCP client initialization failing due to malformed or inaccessible URLs
• Recommendation: Validate MCP server URL configurations and network accessibility

mcp-github-schema-error

• Details: Invalid schema for github-get_commit function
• Impact: AI Moderator workflow failed after 5 retries
• Recommendation: Fix schema definition in GitHub MCP server

---

📊 Performance Metrics

Token Usage Trends

30-Day Total: 31,132,870 tokens
Average Daily: 10,377,623 tokens

Daily Breakdown:

• 2026-01-13: 15,767,907 tokens (⚠️ 152% above average) - MCP failure day
• 2026-01-14: 10,166,416 tokens (↔️ Near average)
• 2026-01-21: 5,198,547 tokens (↘️ 50% below average) - Improving efficiency

Analysis: Token usage is normalizing after the Jan 13 spike. The downward trend suggests improved efficiency or reduced workload complexity.

Cost Analysis

30-Day Total: $3.98
Average Daily: $1.33

Daily Breakdown:

• 2026-01-13: $0.42 (↘️ Low cost despite high tokens - suggests model mix)
• 2026-01-14: $2.72 (⚠️ Peak cost - premium model usage)
• 2026-01-21: $0.84 (↔️ Normalized)

Peak Cost: $2.72 on 2026-01-14
Peak Tokens: 15.8M on 2026-01-13

Success Rate Historical Context

Period	Success Rate	Comparison
Last 3 days	76.67%	Baseline
Last 7 days	76.67%	Same
30-day average	76.67%	Same
2025-12-04	100%	Previous peak
2025-12-06	30.9%	Previous low
2025-12-19	12.0%	All-time low

Analysis: Current 76.67% success rate is above the critical lows seen in December but below the target 80%+ threshold for production reliability.

---

🎯 Affected Workflows

Based on historical audit data, the following workflows have experienced issues:

View Workflow Status (Most Recent: 2026-01-13)

Smoke Claude

• Runs: 2
• Success: 0/2 (0%)
• Status: 🔴 Critical
• Issues: All 6 MCP servers failed

Smoke Copilot

• Runs: 2
• Success: 1/2 (50%)
• Status: ⚠️ Warning
• Issues: Go not available in PATH on failure run

Smoke Codex

• Runs: 2
• Success: 1/2 (50%)
• Status: ⚠️ Warning

Changeset Generator

• Runs: 3
• Success: 2/3 (66.7%)
• Status: ⚠️ Warning
• Issues: codex agent failures

AI Moderator

• Runs: 1
• Success: 1/1 (100%)
• Status: ✅ Healthy (recent schema error noted for 2026-01-16)

Code Scanning Fixer

• Status: ⚠️ Missing Tools
• Issue: Requires github-list_code_scanning_alerts

Tidy

• Status: ⚠️ Missing Tools
• Issue: Requires build tools (make, go fmt, golangci-lint, npm)

Smoke Copilot Playwright

• Status: ⚠️ Missing Tools & MCP Failures
• Issue: Playwright MCP server unavailable, safeinputs-gh missing

---

💡 Recommendations

Immediate Actions (Priority 1)

1. 🔴 URGENT: Investigate 2026-01-13 MCP Infrastructure Failure

   • Analyze root cause of simultaneous failure across all 6 MCP servers
   • Review system logs, resource constraints, network connectivity
   • Implement monitoring to detect early signs of cascading MCP failures
   • Expected Impact: Prevent future mass outages

2. 🔴 Address Recurring Safeoutputs Failures

   • 7 incidents across multiple dates indicates systemic issue
   • Review initialization process, configuration, dependencies
   • Consider implementing retry logic with exponential backoff
   • Expected Impact: Improve reliability of safeoutputs-dependent workflows

3. 🟡 Fix MCP GitHub Schema Error

   • github-get_commit function has invalid schema
   • Caused AI Moderator failure with 5 retry attempts
   • Expected Impact: Restore AI Moderator reliability

Short-Term Improvements (Priority 2)

4. 🟡 Add Missing Critical Tools

   • github-list_code_scanning_alerts → Enable Code Scanning Fixer
   • Safe-input GitHub query tools → Enable Daily Project Performance
   • Expected Impact: Unlock 2 currently blocked workflows

5. 🟡 Resolve Permission Denied Warnings

   • 90+ occurrences of "copilot-permission-denied"
   • Affects 10 workflows including AI Moderator, Issue Monster
   • Review workflow permission scopes and request mechanisms
   • Expected Impact: Reduce warning noise, improve workflow stability

6. 🟢 Investigate MCP Invalid URL Errors

   • 36 occurrences across Smoke Copilot variants
   • Validate MCP server URL configurations
   • Test network accessibility from workflow runners
   • Expected Impact: Improve Smoke test reliability

Long-Term Monitoring (Priority 3)

7. 🟢 Monitor Success Rate Sustainability

   • Current 76.67% is below 80% target
   • Recent trajectory: 50% → 100% → 80% (volatile)
   • Establish alerting for success rate < 80% sustained over 3+ days
   • Expected Impact: Early detection of degradation

8. 🟢 Track Token Usage Trends

   • Recent normalization (15.8M → 10.2M → 5.2M) is positive
   • Monitor for anomalies or unexpected spikes
   • Investigate correlation between token usage and failure rates
   • Expected Impact: Cost optimization and efficiency insights

9. 🟢 Evaluate Build Tool Access for Tidy Workflow

   • 5 requests for make, go, npm tools
   • Decide if code formatting should be supported in agentic workflows
   • If yes, provide sandboxed build environment
   • Expected Impact: Enable automated code cleanup

---

📈 Historical Context

Audit Comparison

Date	Runs	Success Rate	Tokens	Cost	Key Issues
2026-01-23	30	76.67%	31.1M	$3.98	Data collection limitation
2026-01-13	10	50.0%	15.8M	$0.42	MCP mass failure
2025-12-25	58	87.93%	20.0M	$5.67	9 missing tools
2025-12-24	86	55.8%	-	$11.17	1021 errors
2025-12-22	65	76.92%	31.1M	$7.50	778 errors, 10 missing tools

Trend Analysis

Success Rate: Volatile with recent recovery (50% → 80%), but historically inconsistent
Token Usage: Decreasing trend after Jan 13 spike (positive)
Cost: Decreasing and stabilizing around $1-2/day
Issues: Missing tools and MCP failures are recurring themes

---

🔄 Next Steps

For Repository Maintainers

• Review MCP server infrastructure logs for 2026-01-13 failure
• Implement monitoring for MCP server health (uptime, latency, errors)
• Add github-list_code_scanning_alerts to GitHub MCP server
• Fix github-get_commit schema in GitHub MCP server
• Evaluate granting build tool access to Tidy workflow
• Set up alerts for success rate < 80% sustained over 3 days

For Audit System

• Restore gh-aw MCP server access for real-time log collection
• Implement automated daily audit scheduling
• Add chart generation capability (requires Python data science libraries)
• Enhance firewall log analysis for network-related failures
• Track workflow-level metrics (per-workflow success rates, token usage)

---

📁 Audit Data

All findings have been stored in repository memory:

• /tmp/gh-aw/repo-memory/default/audits/2026-01-23.json - Full audit results
• /tmp/gh-aw/repo-memory/default/audits/index.json - Updated audit index
• /tmp/gh-aw/python/data/trend_summary.json - 30-day trend analysis

Data Limitations

This audit was conducted without access to the gh-aw MCP server tools, which provide:

• Real-time workflow run logs
• Detailed error traces
• Firewall analysis
• Turn-by-turn execution data

Impact: Analysis is based on cached historical data (last updated 2026-01-21). Current 24-hour window data is not available.

Recommendation: Ensure gh-aw MCP server is available for future audits to enable comprehensive analysis.

---

Audit Period: Last 30 days (based on available data)
Data Sources: Cache memory (/tmp/gh-aw/cache-memory/), historical audits
Analysis Date: 2026-01-23T04:04:46Z
Next Audit: Recommended within 24 hours with gh-aw MCP access

AI generated by Agentic Workflow Audit Agent

• expires on Jan 30, 2026, 4:08 AM UTC

[github-actions[bot]]

This discussion was automatically closed because it expired on 2026-01-30T04:08:35.524Z.