dev-lead engine repeatedly times out (engine-error) on larger issues — exhausts retries without delivering (e.g. .github#518)

[don-petry]

Summary

The dev-lead engine repeatedly fails with engine-error (timeout / transient engine error) on larger, feature-sized issues, exhausting its retry budget without ever producing a PR. The retry mechanism now works (auto re-dispatch, "attempt N/3") — but the underlying per-attempt engine budget is too small for substantial issues, so retries just repeat the same timeout. Net: bigger issues silently never get delivered, even though the agent correctly posts an implementation plan first.

Live example — petry-projects/.github#518

#518 ("add-to-project: add a reconcile/backfill mode") is a real feature build (a scan script + workflow_dispatch/cron triggers + bats + docs). dev-lead:

• posted a correct implementation plan, then
• attempt 1/3 failed engine-error (run 27908018949),
• auto-retried → attempt 2/3 failed engine-error (run 27909956774),
• ~2.5h elapsed, no PR. Status comment: status=failed attempt=2 reason=engine-error.

This matches earlier instances this session:

• #478 — single attempt, Engine failed to implement (no retry then; pre-dev-lead retry doesn't cover failed initial issue implementations (only rate-limited PRs) — issues silently stall #781-fix).
• #500 — attempt 1 failed engine-error, attempt 2 succeeded (but over-engineered the fix → introduced a CodeQL finding that needed manual rework in [Phase 2] Replace PUT-contents clobber deploy with versioned, ring-staged, health-gated promotion #501).

So the throughline: the engine times out on issues above some size/complexity, and either fails outright or produces a rushed/over-engineered result.

Why it matters

• Larger initiatives labeled dev-lead don't get delivered — they consume 3 attempts of compute and then stall, with only failure comments to show.
• It's silent: no escalation/hand-off, so a human only notices if watching.

Suggested directions

• Right-size the budget: raise the per-attempt engine timeout (and/or use a higher-capacity model/tier) for issues above a complexity threshold.
• Detect oversize + decompose or hand off: if an issue is too big for one attempt, have dev-lead split it (epic → stories, via the planner) or label dev-lead:needs-human and notify, instead of burning 3 identical timeouts.
• Escalate on retry-exhaustion: when attempt 3/3 fails, apply a clear dev-lead:failed/needs-human label + ping, so it doesn't just go quiet.

References

• Live: petry-projects/.github#518 (runs 27908018949, 27909956774).
• Related: .github-private#781 (retry gap — the N/3 retry now exists, but the engine-error root cause remains); .github#500→[Phase 2] Replace PUT-contents clobber deploy with versioned, ring-staged, health-gated promotion #501 (over-engineered success).

[github-actions]

Thanks for the detailed report and for including live run IDs — that makes diagnosis much easier. To narrow down the right fix, could you share the current per-attempt timeout (and/or token/step budget) configured for the dev-lead engine? Knowing that baseline will clarify whether a simple budget increase covers issues like #518, or whether a decompose/escalate path is needed for this size of work.

[don-petry]

Root-cause research (new evidence + code trace)

The timeouts are deterministic budget-exceeded events, mislabeled as transient engine-error and then retried ~6× — pure waste. Traced end-to-end in scripts/engine.sh + scripts/dev-lead-fix-issue.sh + dev-lead-reusable.yml:

1. The knob and the walls. Each model tier is wrapped in GNU timeout (scripts/engine.sh):

DEEP_TIMEOUT_SEC="${DEEP_TIMEOUT_SEC:-600}"   # deep implementation tier
ACTION_TIMEOUT_SEC="${ACTION_TIMEOUT_SEC:-600}"

dev-lead-reusable.yml sets DEEP_TIMEOUT_SEC: ${{ vars.DEEP_TIMEOUT_SEC || '' }} → empty falls through to the engine default 600s. So:

• petry-projects/.github: vars.DEEP_TIMEOUT_SEC unset → 600s wall.
• petry-projects/.github-private: vars.DEEP_TIMEOUT_SEC=1800 → 1800s wall.
• Job backstop: timeout-minutes: 60 (3600s).

2. Timeout (exit 124) is classified as retryable. scripts/engine.sh:

is_transient_failure() {  # 0 = retry
  case "$rc" in 124|137|143) return 0 ;; *) return 1 ;; esac
}

The comment assumes 124 means "flaky network/process state." In practice a per-tier timeout firing at exactly DEEP_TIMEOUT_SEC means the deep model genuinely needed more than the budget — deterministic, not flaky. Retrying with the same budget just repeats it.

3. Retried at two layers. Engine-level RETRY_MAX_ATTEMPTS retries the model call on 124, and the workflow-level MAX_ATTEMPTS=3 re-dispatches the whole run. A single too-big issue burns up to ~6 full-budget attempts.

4. The reason is collapsed. handle_engine_failure reads reason from the /tmp/dev-lead-failure-reason sidecar, defaulting to engine-error. The engine never writes timeout on a 124, so every timeout surfaces as reason=engine-error — indistinguishable from a genuine transient error, which is why the retry path treats them the same.

Fresh evidence from the PR-limits initiative (petry-projects/.github#505)

Issue	Repo	Wall	Runs	Outcome
#507	.github	~10m23s each	4×	all engine-error at the 600s wall; delivered only after splitting into #507+#561
#510	.github	exactly 10m	2×	same 600s wall; hand-authored (PR #573)
#1003	.github-private	exactly 30m00s	1×	engine finished + posted "Implementation Complete" with passing tests, then died at the 1800s wall before pushing → work lost; hand-authored (PR #1004)

The identical per-issue durations are the signature of a deterministic wall. #1003 also exposes a distinct failure mode: the engine can complete the implementation and still time out in the post-implementation phase (self-review / commit / PR-create), losing the finished work.

Answers to the open design questions

Does time = cost, or is system slowness a factor? Both. Wall-clock maps to (a) GitHub Actions runner-minutes and (b) LLM token spend (the deep model runs the whole window). But wall-clock is not pure "work" — it's inflated by model/API latency, provider throttling, MCP/tool round-trips, and runner slowness. So a lone timeout is ambiguous: genuinely-too-much-work vs. a transiently-slow run. The discriminator we have is empirical: repeated timeouts at ~identical durations = deterministic (too big); a one-off = possibly slowness. That argues for at most one budget-escalated retry (to absorb slowness), never same-budget retries.

Variable timeouts by complexity? Worth it, but secondary. The first-order fix is to stop same-budget retries. Two complementary levers: (a) adaptive retry — if we retry a timeout at all, give it a larger budget (e.g. 1.5×, capped by the job), not the same one that just failed; (b) complexity-scaled initial budget — size DEEP_TIMEOUT_SEC at dispatch from an issue-size heuristic (a size:* label, body length, # acceptance criteria / referenced files).

What is the maximum timeout? Hard ceiling = the job timeout-minutes (currently 60 → 3600s; GitHub's absolute max is 360 min/job). A single deep call must leave room for the other tiers in the same job — triage 300 + audit 600 + duck 300 + git/PR/overhead — so within a 60-min job the practical DEEP_TIMEOUT_SEC max is ~1800–2100s (1800 is already in use in .github-private). Going higher requires raising timeout-minutes too.

Plan

1. Classify timeouts distinctly. On a per-tier timeout (exit 124) in the deep/action tier, the engine writes reason=timeout to /tmp/dev-lead-failure-reason (mechanism already exists; just populate it) so it is no longer collapsed into engine-error.
2. Stop the wasteful retry (the core fix). Do not retry a deep/action timeout with the same budget at either layer: remove 124 from the engine model-call retry for these tiers, and add a timeout branch to handle_engine_failure that escalates immediately (mirroring the existing missing-binary immediate-escalate path) instead of the engine-error → will retry path.
3. At most one adaptive-budget retry (optional, absorbs slowness). Before escalating, allow exactly one retry at an escalated DEEP_TIMEOUT_SEC (e.g. 1.5×, only if it fits under the job cap). If it still times out → deterministic → escalate.
4. Actionable escalation. The needs-human comment states the tier, elapsed seconds, and the budget, plus guidance: "hit the {N}s deep budget — split this issue or raise vars.DEEP_TIMEOUT_SEC," not the current generic "engine-error — will retry."
5. Don't lose completed work. Address the [PR-limits Phase 3] Wire the admission gate into dev-lead-fix-issue.sh (enforce the org-wide cap) #1003 mode: if the implementation phase completed (commit exists / changes staged) but a later phase times out, prefer pushing the branch + opening the PR over discarding — or at minimum surface that the work was completed-but-not-pushed.
6. Instrument. Log per-tier elapsed seconds alongside the existing TOKEN_LOG_FILE token counts, and surface elapsed-vs-budget in the escalation, so timeout-vs-slowness can be told apart over time.
7. Sane fleet defaults. Set vars.DEEP_TIMEOUT_SEC explicitly (petry-projects/.github is currently unset → 600, too small for feature-sized stories); recommend a documented fleet default with verified job-cap headroom.

Acceptance criteria

• A deep/action timeout (exit 124) is recorded as reason=timeout, distinct from engine-error, in the status marker + escalation comment.
• A timeout is never retried with the same budget at either the engine or workflow layer.
• At most one budget-escalated retry occurs before escalation to human; if the escalated budget also times out, the run escalates (no further retries).
• Total wall-clock (and token/runner cost) spent on a deterministically-too-big issue is bounded to ≤ ~1 escalated attempt, not the current ~3–6× the budget.
• The needs-human comment includes tier, elapsed seconds, budget, and split-or-raise-budget guidance.
• Genuine transient failures (non-124, e.g. rate-limit) retry exactly as they do today (no regression).
• If the implementation completed before a later-phase timeout, the branch/PR is not silently discarded (pushed, or the completed-but-unpushed state is surfaced).
• Per-tier elapsed seconds are logged for every run.
• vars.DEEP_TIMEOUT_SEC is set explicitly per repo/fleet, documented, and verified to fit under timeout-minutes.
• Tests: a stubbed 124 → asserts reason=timeout + no same-budget retry + escalation message; a stubbed non-124 transient → still retries. (Extend tests/dev-lead/unit/.)

---

Evidence, code trace, and the design-question answers gathered while shepherding epic petry-projects/.github#505; those stories were unblocked by splitting/rescoping the oversized issues and hand-authoring two of them. Related mislabel-of-timeout note lives in the shepherd's session memory.

[don-petry]

Direction update (per maintainer) — maximize per-stage runtime; don't use timeouts for cost

Revised strategy, superseding the "tight budget + adaptive-retry" emphasis in the plan above:

Timeouts are a hang/stuck-run backstop, not a cost control. We will maximize the allowed runtime per stage so a legitimately large issue rarely hits a wall in the first place, and we will not size timeouts to bound token/runner cost. Cost, if it needs bounding, is handled elsewhere (token budgets / model tiering / issue sizing) — never by starving a stage of time.

What changes vs. the plan above

• Keep (still correct): classify a per-tier timeout (exit 124) as reason=timeout distinct from engine-error; do not retry a timeout with the same budget. But the retry framing simplifies: at a maximized budget a timeout is definitively deterministic (there is no larger budget to escalate to), so a timeout → escalate to human immediately ("this issue needs splitting"), no retry. Drop the "adaptive-budget retry" step — it's unnecessary once the initial budget is already the max.
• New primary lever: max runtime per stage.
• New: timeout observability in monitoring.

Max runtime per stage — model

Per-tier timeouts must fit inside the job timeout-minutes backstop, alongside the other tiers that run in the same job + fixed overhead (checkout, CLI install, git/PR ops). So:

DEEP budget  =  job_cap  −  (Σ other per-tier budgets)  −  fixed_overhead_margin

Goal: raise the job timeout-minutes well above today's 60, and set each stage's timeout to consume its available share — deep/action (the implementation stages) get the lion's share; triage/duck stay small (they're lightweight reads). Concrete proposal (tune during implementation):

Knob	Today	Proposed
job timeout-minutes (backstop)	60 (3600s)	120 (7200s) — raise toward the practical max; GitHub's hard ceiling is 360 min/job
DEEP_TIMEOUT_SEC	600 (.github) / 1800 (.github-private)	≈ job_cap − reserved → ~5400s at a 120-min cap
ACTION_TIMEOUT_SEC	600	same share as deep (implementation-class work)
AUDIT_TIMEOUT_SEC	600	keep modest (900–1200s)
TRIAGE_TIMEOUT_SEC / DUCK_TIMEOUT_SEC	300	keep small (lightweight read/classify)

Two ways to set the deep budget; implementation picks one:

• Static per-repo: set vars.DEEP_TIMEOUT_SEC explicitly per repo (petry-projects/.github is currently unset → 600, the root of the .github walls) to the raised value, and verify it fits under timeout-minutes.
• Derived: compute DEEP_TIMEOUT_SEC at startup as job_cap − Σ(other tiers) − overhead_margin so it always tracks the job cap and can't exceed it.

Guardrail: Σ(all per-tier budgets) + overhead_margin ≤ job_cap must hold (a CI/lint check), so the sum can never silently exceed the backstop.

Timeout observability (new)

Raise the visibility of this failure mode in the monitoring fleet so we can see whether the bigger budgets actually eliminate walls:

• Emit a timeout counter/rate for dev-lead (and the other engine callers) in scripts/fleet_monitor.sh / actions-fleet-monitor.yml — count runs whose failure is reason=timeout (keyed off the new classification), broken out from generic engine-error, over the reporting window.
• Surface it in the fleet-monitor summary/issue and (optionally) the daily status, so a rising timeout count is an actionable signal rather than buried in engine-error.

Revised acceptance criteria

• Job timeout-minutes is raised (proposed 120; document the chosen value and rationale).
• Per-tier timeouts (DEEP_/ACTION_/AUDIT_/TRIAGE_/DUCK_TIMEOUT_SEC) are set to their maximum viable share of the raised job budget; deep/action get the lion's share. Cost is explicitly not a factor in these values.
• A CI/lint guardrail asserts Σ(per-tier budgets) + overhead_margin ≤ job timeout-minutes so the tiers can never exceed the backstop.
• petry-projects/.github's vars.DEEP_TIMEOUT_SEC is no longer effectively 600 (set explicitly or derived from the job cap).
• A per-tier timeout (exit 124) is recorded as reason=timeout (distinct from engine-error) in the status marker + escalation comment.
• A timeout at the maximized budget escalates to human immediately with tier + elapsed + budget + "split this issue" guidance; it is not retried with the same budget at either the engine or workflow layer.
• Genuine transient failures (non-124, e.g. rate-limit) retry exactly as today (no regression).
• Completed-but-unpushed work is not discarded when a later phase times out (the [PR-limits Phase 3] Wire the admission gate into dev-lead-fix-issue.sh (enforce the org-wide cap) #1003 mode).
• The monitoring fleet surfaces a timeout count/rate for dev-lead, broken out from engine-error, in the fleet-monitor report.
• Per-tier elapsed seconds are logged for every run (alongside the existing token log) to confirm walls are eliminated.
• Tests (tests/dev-lead/unit/): stubbed 124 → reason=timeout + escalate + no same-budget retry; the Σ-budgets-≤-cap guardrail; non-124 transient still retries.

Cost is intentionally out of scope for timeout sizing per maintainer direction; timeouts exist only to catch genuinely hung/stuck runs, and should otherwise be as large as the job budget allows.

[don-petry]

Dev-Lead Implementation Plan

Issue: #901 — dev-lead engine repeatedly times out (engine-error) on larger issues

Root cause

run_writer in scripts/engine.sh enforces a fixed per-attempt writer timeout
(ACTION_TIMEOUT_SEC, default 600s / 10min). Feature-sized issues need longer, so
the engine is killed by GNU timeout (exit 124 → classified engine-error), and because
every retry uses the same 600s budget, retries just repeat the identical timeout — exactly
the #518 symptom (attempt 1/3 and 2/3 both engine-error, no PR).

The retry mechanism (attempt N/3) and retry-exhaustion escalation (dev-lead:needs-human
label + comment) already exist and work — the missing piece is right-sizing the per-attempt
budget.

Scope

Right-size the dev-lead writer engine timeout: scale it by issue complexity and grow it
per attempt, bounded by a cap that stays under the reusable workflow's 60-min job cap.
No change to the common small-issue path (stays 600s), and an explicit operator override
(repo var ACTION_TIMEOUT_SEC) is honored verbatim. Retry-exhaustion escalation is already
implemented, so this focuses on direction #1 ("right-size the budget").

Implementation Checklist

• scripts/dev-lead-fix-issue.sh: add pure issue_complexity_tier (0/1/2 from issue
  body length + task-list checkbox count) and compute_engine_timeout <attempt>
  (base + tier*step + (attempt-1)*step, capped).
• scripts/dev-lead-fix-issue.sh: before the run_writer_with_fallback call, set
  ACTION_TIMEOUT_SEC to the computed budget — but only when the operator did not
  set it explicitly (capture the raw env value before engine.sh collapses empty→600).
  Emit a ::notice:: recording the chosen budget/attempt/tier for observability.
• scripts/dev-lead-fix-issue.sh: convert the bottom main "$@" to the repo-standard
  source-guard so the pure functions are unit-testable.

Tests to Write

• tests/dev-lead/unit/test_fix_issue.bats: issue_complexity_tier → 0 for a small
  body, 1 for medium, 2 for a large feature body with many checkboxes.
• compute_engine_timeout: tier-0/attempt-1 == 600 (preserves default), grows with
  attempt (attempt 2 → 1200), grows with complexity (tier-2/attempt-1 → 1800), and is
  capped (tier-2/attempt-3 → cap).
• End-to-end: a large issue logs the right-sized ::notice::; an explicit
  ACTION_TIMEOUT_SEC override suppresses auto-sizing.

Test Command

bats tests/dev-lead/unit/test_fix_issue.bats (full: bats tests/**/*.bats + bash .dev-lead/scripts/dev-lead-lint.sh)

[don-petry]

Dev-Lead: issue #901 implementation failed — will retry

The engine failed (engine-error — e.g. a timeout or transient engine error). This was attempt 1 of 3; dev-lead will retry automatically. You can also re-apply the dev-lead label to retry now.

• Cause: engine-error
• Run: https://github.com/petry-projects/.github-private/actions/runs/28598293351