[Fleet Monitor] petry-projects/.github — .github/workflows/canary-rollout.yml

[don-petry]

🟡 WARNING: petry-projects/.github / .github/workflows/canary-rollout.yml

Metric	Value
Failure Rate	16.7% (1 / 6 runs)
Status	WARNING
p50 Duration	175s
p95 Duration	234s
Successful Runs	5
Cancelled Runs	0

Workflow: .github/workflows/canary-rollout.yml

Last updated by Fleet Monitor run on 2026-07-15.
Threshold: failure rate > 10% over the monitored window.

[don-petry]

Dev-Lead Implementation Plan

Issue: #738 — [Fleet Monitor] petry-projects/.github — .github/workflows/canary-rollout.yml

Root Cause

The canary-rollout.yml scheduled tick (every 4h) runs the read-only fleet-wide evaluate-all (or gated promote-all), which fans out to many gh run list calls — one per agent × per tier repo. The gate reader _run_json (scripts/canary-rollout.sh) deliberately fails closed: any non-zero gh exit returns 1, which propagates up through _frontier_state → cmd_evaluate → cmd_evaluate_all and, under the workflow step's set -euo pipefail, fails the whole run. Failing closed on a sustained outage is correct (an empty [] could green-light a bad promotion), but a single transient blip (network / secondary rate-limit / brief 5xx) among dozens of reads is enough to fail an otherwise-healthy scheduled run. That intermittent surface is consistent with the observed 16.7% (1/6) failure rate.

Scope

Add a bounded retry-with-backoff around the gh run list call in _run_json so a momentary blip is retried instead of failing the fleet sweep, while a sustained outage still exhausts the retries and fails closed (safety preserved).

Implementation Checklist

• scripts/canary-rollout.sh — wrap the gh run list fetch in _run_json with bounded retry (env-tunable attempts/backoff; retry only on non-zero exit; unchanged success + fail-closed contract)

Tests to Write

• tests/canary_rollout.bats — _run_json retries a transient failure then returns the payload (status 0)
• tests/canary_rollout.bats — _run_json still fails closed (non-zero) when gh fails on every attempt

Test Command

bats --print-output-on-failure tests/canary_rollout.bats (plus shellcheck --severity=warning -x scripts/canary-rollout.sh scripts/lib/canary-rollout.sh and bash .dev-lead/scripts/dev-lead-lint.sh)

[don-petry]

Dev-Lead: Implementation Complete

Root Cause

The scheduled 4h tick runs the fleet-wide read-only evaluate-all (or gated promote-all), which fans _run_json out across every agent × tier repo. _run_json failed closed on any non-zero gh run list exit, and that non-zero propagated up through _frontier_state → cmd_evaluate → cmd_evaluate_all and, under the workflow step's set -euo pipefail, failed the whole run. Failing closed on a sustained outage is correct, but a single transient blip (network / secondary rate-limit / brief 5xx) among dozens of reads was enough to fail an otherwise-healthy sweep — matching the observed 16.7% (1/6) intermittent failure rate.

Fix

Added a bounded retry-with-backoff around the gh run list call in _run_json (scripts/canary-rollout.sh). A transient failure is retried (default 3 attempts, 2s→4s backoff); a sustained failure still exhausts the retries and returns non-zero, so the fail-closed safety contract (an empty [] must never mask real failures and green-light a bad promotion) is fully preserved. Retry counts/backoff are env-tunable via CANARY_GH_RETRIES / CANARY_GH_RETRY_SLEEP. The success and empty-result paths are byte-identical to before.

Plan Execution

• Wrapped _run_json's gh run list fetch in bounded retry (fail-closed preserved)
• Added test: transient failure is retried, then payload returned (status 0)
• Added test: sustained failure still fails closed (non-zero) after retries exhausted

Test Results

ok 208 orchestrator: autocut detects a removed workflow_call input → MAJOR (interface break) (#712)
ok 209 orchestrator: autocut detects a newly-required workflow_call input → MAJOR (interface break) (#712)
ok 210 orchestrator: autocut detects a non-breaking feat commit → MINOR (#712)
ok 211 orchestrator: autocut detects a fix-only change → PATCH (#712)
ok 212 orchestrator: autocut fails SAFE to PATCH when the commit-signal fetch errors (#712)
ok 213 orchestrator: autocut knob override forces MAJOR over a patch-only diff (#712)

total: 213 passing, 0 failing (bats tests/canary_rollout.bats)

Also passing: shellcheck --severity=warning -x scripts/canary-rollout.sh scripts/lib/canary-rollout.sh and bash .dev-lead/scripts/dev-lead-lint.sh.

Files Changed

• scripts/canary-rollout.sh: bounded retry-with-backoff in _run_json; fail-closed contract unchanged
• tests/canary_rollout.bats: two _run_json tests (transient-retry-then-succeed; sustained-fail-closed)

Notes

Retry applies to the single gh run list reader shared by all gate-history reads (_tier_sample, _cumulative_health, _baseline_daily, …), so the whole read-only gate path is hardened, not just one call site. Worst-case added latency on a genuine outage is bounded (~6s per exhausted call, well within the job's timeout-minutes: 15). No behavioral change to the promotion/gate decision logic.