[Fleet Monitor] petry-projects/google-app-scripts — .github/workflows/dev-lead.yml

[don-petry]

🟡 WARNING: petry-projects/google-app-scripts / .github/workflows/dev-lead.yml

Metric	Value
Failure Rate	12.5% (2 / 16 runs)
Status	WARNING
p50 Duration	8s
p95 Duration	52s
Successful Runs	6
Cancelled Runs	0

Workflow: .github/workflows/dev-lead.yml

Last updated by Fleet Monitor run on 2026-07-06.
Threshold: failure rate > 10% over the monitored window.

[don-petry]

Dev-Lead: Root-Cause Analysis — no in-repo code change applies

I investigated the 50% failure rate on .github/workflows/dev-lead.yml and traced every failing run to a single, repository-external cause. No source change in petry-projects/google-app-scripts can fix it; fabricating one would be out of scope and would not reduce the failure rate.

What's actually failing

The failing runs are all triggered by issues events (issue labeled dev-lead). The companion issue_comment runs are no-op skips and succeed — that pairing is exactly why the rate sits at ~50%.

Each failing run dies in the Run issue step (dev-lead-fix-issue.sh) with an identical two-part failure:

{"api_error_status":429, ... "result":"You've hit your limit · resets Jun 16, 4pm (UTC)"}
##[warning]claude unavailable (exit 2), trying next engine
Error: Model "openai/o4-mini" from --model flag is not available.
##[error]Engine failed to implement issue #NNN
##[error]Process completed with exit code 1.

Verified identical across runs 27425658399, 27425651153, 27425645070, 27425641324, 27425632042 (issues #336, #335, #334, #257, #332).

Root cause (two compounding factors, both external to this repo)

1. Claude account quota exhausted — HTTP 429, resets Jun 16, 4pm UTC. This is the CLAUDE_CODE_OAUTH_TOKEN org/repo secret, not anything in this repo. The burst is amplified by ~14 compliance issues being labeled within ~60s on 2026-06-12 15:29–15:30, all firing dev-lead simultaneously.
2. Fallback engine misconfigured — the secondary engine invokes --model openai/o4-mini, which the provider reports as not available, so the fallback can't absorb the outage. The model name is hardcoded in the private petry-projects/.github-private/.github/workflows/dev-lead-reusable.yml / engine.sh.

Why nothing in this repo fixes it

• .github/workflows/dev-lead.yml is a thin caller stub and is already correct: pinned to @dev-lead/stable, passes agent_ref, and grants statuses: read (the prior fixes in fix(dev-lead): grant statuses:read (fixes startup_failure) #370 and fix(dev-lead): pin to @dev-lead/stable channel + agent_ref (#557) #378). There is no stub-level config bug.
• All execution logic (engine.sh, dev-lead-fix-issue.sh, the exit 2 → exit 0 step wrapper) lives in the private reusable workflow and is .dev-lead/-gitignored / fetched at runtime — not editable here.
• The two failing inputs (Claude quota, fallback model name) are an org secret and a private-workflow constant, respectively.

Recommended remediation (owners outside this repo)

1. Restore Claude quota for CLAUDE_CODE_OAUTH_TOKEN (raise the limit or wait for the Jun 16 reset).
2. Fix the fallback engine in .github-private engine.sh: replace openai/o4-mini with a valid, available model — without a working fallback, any Claude outage hard-fails every issues run.
3. Optional hardening in the reusable workflow: treat "all engines unavailable / rate-limited" as a soft-skip (exit 2 → exit 0) instead of a hard exit 1, so transient quota exhaustion shows as skipped rather than failed and stops tripping the Fleet Monitor threshold.

Phase notes

• Phase 2 (TDD) intentionally skipped: no observable in-repo behavior changes — there is nothing to assert in a Jest test. Per repo guidance, behavior-preserving/no-change situations skip the test-first phase.
• No files changed; no PR will be opened. The fix must land in the org Anthropic quota and the private .github-private reusable workflow.