feat(dev-lead): retry failed initial issue implementations, never silent (#781)

[don-petry]

Summary

Failed initial issue implementations used to stall silently. When a dev-lead issues-labeled run failed at the Run issue engine step for any non-rate-limit reason, dev-lead-fix-issue.sh emitted only a generic Engine failed to implement issue #N / exit 1 — no comment, no marker, no cause — and dev-lead-retry only ever scanned PRs for status=rate-limited. So the issue kept its dev-lead label, looked "in progress", and was never retried.

This PR makes failed issue implementations never-silent, cause-aware, and auto-retried (bounded). Closes #781.

Step 0 — confirmed live root cause (petry-projects/.github#478)

Pulled the failed Run issue job log for run 27733022075. The generic exit 1 was hiding a 600-second per-tier timeout SIGTERM-kill of the opus 4.8 writer (02:37:30 → 02:47:30 = a clean 600.2s wall; [headroom] 0% — ok rules out rate-limit). DEEP_TIMEOUT_SEC=600 (engine.sh:37), and 124=GNU timeout is classified transient (engine.sh:43). A transient failure with no retry path → permanent stall. Full trace in the findings comment.

Phase 1 — never silent, with cause (no caller-interface change)

• engine.sh — run_writer_with_fallback writes a one-line cause class to /tmp/dev-lead-failure-reason (rate-limited | missing-binary | engine-error), mirroring the existing rate-limit-reset sidecar. run_writer's redact + persist + $GITHUB_STEP_SUMMARY block now runs before the rate-limit early-return, so session output is captured for all failure classes.
• dev-lead-fix-issue.sh — replaces the silent exit 1 and unifies the rate-limit branch with handle_engine_failure: reads cause + reset, computes attempt from prior markers, posts a comment with the cause + run deep-link + redacted 40-line snippet, and applies a decision matrix (missing-binary → dev-lead:needs-human; retryable & attempt<MAX → retryable marker; attempt≥MAX → dev-lead:needs-human). Exit codes unchanged.

Phase 2 — bounded auto-retry

• dev-lead-retry.sh — new scan_repo_issues / scan_issue_for_retry / dispatch_issue_retry: enumerates open dev-lead issues (excludes PRs + dev-lead:needs-human), reads the newest <!-- dev-lead-issue N … --> marker, and re-dispatches dev-lead-issue-retry honouring the reset= window, attempt < MAX_ATTEMPTS (3), and "skip if an open PR exists". Runs even when a repo has no open PRs.
• dev-lead-intent.sh — restructured so issue-only dispatch payloads aren't dropped; dev-lead-issue-retry routes back to the existing issue intent (full-fidelity re-run).
• dev-lead.yml (thin caller) — adds dev-lead-issue-retry to repository_dispatch.types (established pattern, allowed per AGENTS.md). dev-lead-reusable.yml — adds a client_payload.issue_number → dev-lead-issue-<n> concurrency lane.

Tests & docs

• tests/dev-lead/unit/test_dev_lead_retry.bats (new, 10 cases — decision matrix + dispatch payload).
• Extended test_fix_issue.bats (+4) and test_engine_fallback.bats (+5); test_intent_ci.bats (+3); new fixture repository_dispatch_issue_retry.json; e2e scenario 07 Part I.
• All 448 dev-lead unit tests pass; shellcheck clean (--severity=warning -x); workflow YAML validates.
• docs/dev-lead/spec.md: dispatch table, §7.5 failure matrix, §10.1a issue-retry path.

Follow-up (separate PR, other repo)

The org-level standards/ci-standards.md / AGENTS.md in petry-projects/.github should document the new dev-lead-issue-retry dispatch type and dev-lead:needs-human semantics for consumer repos — out of scope for this .github-private PR.

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features
  • Added issue-specific retry dispatch with attempt tracking, machine-readable retry markers, and escalation to human review after retries
  • Expanded repository dispatch handling to support new issue-retry events and improved routing for retry scenarios
• Documentation
  • Updated the Dev-Lead architecture spec with the new event→intent mapping and failure/retry behavior
• Tests
  • Added unit and end-to-end coverage for issue retry scanning, dispatch gating, marker precedence, and engine-failure classification/summary behavior
• Chores
  • Updated security scanning configuration to suppress a known token placeholder false positive

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[coderabbitai]

Warning

Review limit reached

@don-petry, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 52 minutes and 47 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: bd9b6ecd-794d-4a91-ae2a-b942b3bbb4c1

📥 Commits

Reviewing files that changed from the base of the PR and between be9edcc and e8d754a.

📒 Files selected for processing (4)

• .gitleaks.toml
• .gitleaksignore
• docs/dev-lead/spec.md
• scripts/dev-lead-retry.sh

📝 Walkthrough

Walkthrough

Implements a bounded retry path for failed initial dev-lead issue implementations. engine.sh now writes a failure classification sidecar (/tmp/dev-lead-failure-reason) after persisting session output. dev-lead-fix-issue.sh reads that sidecar and posts machine-readable retry markers or escalates to dev-lead:needs-human. dev-lead-retry.sh gains an issue scanner that re-dispatches dev-lead-issue-retry. dev-lead-intent.sh and the workflows wire the new event type into the existing intent pipeline.

Changes

Issue Retry End-to-End Flow

Layer / File(s)	Summary
Engine failure sidecar and session persistence ordering scripts/engine.sh	run_writer now persists session output before the rate-limit check. run_writer_with_fallback clears /tmp/dev-lead-failure-reason at entry and writes a one-line classification (engine-error, missing-binary, or rate-limited) on every terminal failure path; clears on success.
Issue fix: retry markers and escalation via handle_engine_failure scripts/dev-lead-fix-issue.sh	Adds ISSUE_MARKER_PREFIX, MAX_ATTEMPTS, NEEDS_HUMAN_LABEL constants and count_prior_attempts, html_escape, session_snippet, ensure_needs_human_label, handle_engine_failure functions. handle_engine_failure reads the sidecar, selects missing-binary immediate escalation, retry-exhaustion escalation, or retryable marker posting (with run URL, redacted session tail, optional rate-limit reset). main now calls handle_engine_failure instead of inline branching.
Retry scanner: issue scan, dispatch, and BASH_SOURCE guard scripts/dev-lead-retry.sh	Adds issue-retry constants (ISSUE_MARKER_PREFIX, MAX_ATTEMPTS, DEV_LEAD_LABEL, NEEDS_HUMAN_LABEL), dispatch_issue_retry, open_issue_pr_exists, scan_issue_for_retry, and scan_repo_issues. scan_repo always runs issue scanning. scan_issue_for_retry selects the newest dev-lead-issue marker, parses status/attempt/reason/reset, enforces MAX_ATTEMPTS, honors rate-limit reset windows, and skips when a dev-lead PR is already open. Adds BASH_SOURCE guard.
Intent routing and workflow wiring for dev-lead-issue-retry scripts/dev-lead-intent.sh, .github/workflows/dev-lead.yml, .github/workflows/dev-lead-reusable.yml	dev-lead-intent.sh introduces subject_number fallback for label-fetch gating, adds pr_number guards for existing dispatch types, and handles dev-lead-issue-retry by emitting the issue intent with issue-retry-dispatch reason. dev-lead.yml adds dev-lead-issue-retry to trigger types. dev-lead-reusable.yml extends the concurrency group expression to route issue-retry dispatches into the dev-lead-issue-N lane.
Unit, e2e tests and event fixture tests/dev-lead/unit/test_engine_fallback.bats, tests/dev-lead/unit/test_fix_issue.bats, tests/dev-lead/unit/test_dev_lead_retry.bats, tests/dev-lead/unit/test_intent_ci.bats, tests/dev-lead/e2e/scenarios/07-rate-limit-retry.sh, tests/dev-lead/fixtures/events/repository_dispatch_issue_retry.json	Engine fallback tests cover sidecar classification values and rate-limit session persistence. Fix-issue tests cover engine-error retry marker, session snippet inclusion, missing-binary escalation, and attempt-exhaustion escalation. Retry unit tests cover scan_issue_for_retry decision matrix and dispatch_issue_retry payload/DRY_RUN. Intent tests cover dev-lead-issue-retry routing, missing-payload skip, and hands-off label gating. E2e scenario adds Part I dry-run issue-retry scan assertion.
Spec documentation and gitleaks allowlist docs/dev-lead/spec.md, .gitleaks.toml	Spec updated with new intent table entries, engine-failure handling procedure, and retry/escalation error-handling specification. Gitleaks allowlist adds suppressions for ghp_stub in test fixture files.

Sequence Diagram(s)

sequenceDiagram
  participant Engine as engine.sh (run_writer_with_fallback)
  participant Sidecar as /tmp/dev-lead-failure-reason
  participant FixIssue as dev-lead-fix-issue.sh (handle_engine_failure)
  participant GitHub as GitHub Issue API

  rect rgba(220, 80, 80, 0.5)
    note over Engine,Sidecar: First attempt — engine fails
    Engine->>Sidecar: write "engine-error"
    Engine-->>FixIssue: return rc=1
  end

  FixIssue->>Sidecar: read reason
  FixIssue->>GitHub: fetch prior attempt markers (count_prior_attempts)
  alt attempt < MAX_ATTEMPTS and reason != missing-binary
    FixIssue->>GitHub: post retry marker comment (status=failed, attempt=N, reason=engine-error)
    FixIssue-->>FixIssue: exit 1
  else missing-binary OR attempt >= MAX_ATTEMPTS
    FixIssue->>GitHub: apply dev-lead:needs-human label
    FixIssue->>GitHub: post needs-human marker comment
    FixIssue-->>FixIssue: exit 1
  end

Loading

sequenceDiagram
  participant Cron as dev-lead-retry.sh (cron)
  participant GitHub as GitHub API
  participant Intent as dev-lead-intent.sh
  participant FixIssue as dev-lead-fix-issue.sh

  Cron->>GitHub: list open issues with dev-lead label
  loop each issue
    Cron->>GitHub: fetch issue comments (newest dev-lead-issue marker)
    Cron->>GitHub: check open dev-lead/issue-N PR
    alt retryable marker and no open PR and attempt < MAX_ATTEMPTS
      Cron->>GitHub: POST repository_dispatch dev-lead-issue-retry (issue_number, attempt)
    end
  end

  GitHub->>Intent: dev-lead-issue-retry event received
  Intent->>Intent: validate issue_number, check hands-off label
  Intent-->>FixIssue: emit intent=issue, reason=issue-retry-dispatch
  FixIssue->>FixIssue: run engine, handle_engine_failure on failure

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• petry-projects/.github-private#366: Both PRs modify scripts/engine.sh's run_writer flow to ensure writer/session output is persisted to GITHUB_STEP_SUMMARY (and via /tmp/dev-lead-session-output.txt), including around rate-limit early returns.
• petry-projects/.github-private#371: Both PRs modify scripts/engine.sh in the run_writer path, changing how captured session output is persisted/appended to /tmp/dev-lead-session-output.txt / GITHUB_STEP_SUMMARY.
• petry-projects/.github-private#451: This PR adds repository_dispatch routing for the new issue-retry event to a dev-lead-issue-<n> concurrency lane in dev-lead-reusable.yml, aligning with test assertions for dev-lead-* concurrency lanes.

Suggested labels

needs-human-review

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 60.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The PR title 'feat(dev-lead): retry failed initial issue implementations, never silent' accurately captures the main change: adding retry support and failure visibility for failed issue implementations, directly addressing the core gap identified in #781.
Linked Issues check	✅ Passed	All acceptance criteria from #781 are met: failed initial issue implementations now trigger bounded auto-retry with max 3 attempts [engine.sh, dev-lead-fix-issue.sh, dev-lead-retry.sh]; retry counts are observable via machine-readable markers in comments [dev-lead-fix-issue.sh]; and the implementation path is covered by 448+ passing unit tests [test_dev_lead_retry.bats, test_fix_issue.bats, test_engine_fallback.bats].
Out of Scope Changes check	✅ Passed	All changes are directly within scope of #781: workflow trigger updates for dev-lead-issue-retry dispatch, failure-reason sidecar for cause visibility, retry marker logic, issue scanning in retry cron, test coverage, and documentation of the new architecture. No extraneous changes detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch claude/issue-781-analysis-ecvnhc

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[donpetry-bot]

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-19T20:42:43Z.

[don-petry]

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning

- **Analyze (actions)** — in_progress
- **Analyze (python)** — in_progress
- **CodeQL** — completed, neutral ✓
- **CodeRabbit** — completed, success ✓
- **add-to-project checks** — in_progress or skipped
All completed checks have passing or neutral conclusions. No checks are failing or blocked.
### Review Threads
No review comments with requested changes exist on the PR.
### Conclusion
**No actionable code issues to address.** The bot comments are service-level notifications about API/billing limits, not code quality findings. There are no code defects, security issues, or style problems identified that require fixing.

[gemini-code-assist]

Code Review

This pull request implements a bounded auto-retry mechanism for failed initial issue implementations, introducing unified engine-failure handling, sidecar-based failure classification, and automated re-dispatching via cron scans. The feedback highlights a critical, recurring issue in both dev-lead-fix-issue.sh and dev-lead-retry.sh where gh api --paginate is used; because paginated API responses return multiple sequential JSON arrays, processing them with jq without the -s (slurp) flag leads to incorrect maximum attempt calculations, broken 'newest marker' logic, and bash syntax errors due to multi-line outputs. Applying the suggested jq -s workarounds will ensure robust handling of paginated API data.

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.

[don-petry]

Dev-Lead — review-changes (applied)

Changes committed and pushed.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/dev-lead/spec.md`:
- Around line 632-635: The document inconsistently uses both British English
("labelled") and American English ("labeled") variants for the same word.
Standardize the spelling throughout the document by replacing all instances of
"labelled" with "labeled" to maintain consistency, particularly in the section
describing the dev-lead-retry cron's scan_repo_issues function where
"dev-lead-labelled issues" should be changed to "dev-lead-labeled issues" to
match the American English variant used elsewhere in the specification.

In `@scripts/dev-lead-retry.sh`:
- Around line 316-326: The open_issue_pr_exists function only fetches the first
100 open PRs due to the per_page=100 parameter in the gh api call, which means
matching dev-lead/issue-<N>-* branches could be missed in repositories with more
than 100 open PRs. Add the --paginate flag to the gh api command to iterate
through all pages of results, ensuring all open PRs are checked regardless of
the total count.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 97dd1878-82e2-49eb-8d01-ae8f7a04cf30

📥 Commits

Reviewing files that changed from the base of the PR and between e69a2ac and 7cbe053.

📒 Files selected for processing (14)

• .github/workflows/dev-lead-reusable.yml
• .github/workflows/dev-lead.yml
• .gitleaks.toml
• docs/dev-lead/spec.md
• scripts/dev-lead-fix-issue.sh
• scripts/dev-lead-intent.sh
• scripts/dev-lead-retry.sh
• scripts/engine.sh
• tests/dev-lead/e2e/scenarios/07-rate-limit-retry.sh
• tests/dev-lead/fixtures/events/repository_dispatch_issue_retry.json
• tests/dev-lead/unit/test_dev_lead_retry.bats
• tests/dev-lead/unit/test_engine_fallback.bats
• tests/dev-lead/unit/test_fix_issue.bats
• tests/dev-lead/unit/test_intent_ci.bats

[don-petry]

Dev-Lead — waiting on PR blockers (intent: review-changes)

PR: #810
No changes were committed, but the PR still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews). The retry cron will re-attempt automatically. Next attempt after: 2026-06-20T07:19:59Z

[don-petry]

Note

@don-petry I reviewed this PR and no code changes were needed, but it still has blocking checks or reviews (failing or cancelled checks, or changes-requested reviews), so I cannot mark it done yet. I'll re-check automatically.
Next attempt after: 2026-06-20T07:19:59Z

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[don-petry]

Dev-Lead — fix-bot-comment (no-changes)

Agent reasoning

- `scripts/dev-lead-retry.sh` (open_issue_pr_exists function verified)
- `docs/dev-lead/spec.md` (spelling verified)
**Summary:**
All bot findings have been previously addressed:
- **CodeRabbit findings** (2): Both spelling consistency and `--paginate` flag are correct in the current code
- **Gemini finding** (1): The `jq -s` slurp flag is applied correctly at lines 38-45 of dev-lead-fix-issue.sh; review thread now resolved
- **SonarCloud quality gate**: PASSED (0 new issues, 0 security hotspots)
- **CI status**: All checks passing (success or skipped)
- **Blocker status**: No Tier 1 blockers (no failures, no CHANGES_REQUESTED reviews)
No additional changes needed. The PR is ready.