Repository navigation
feat(lastcode): detect and recover stale Codex and Claude runs - #282
Conversation
|
@codex review |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
@codex review |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 905adc98bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
@codex review |
|
@codex review |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 77d0867367
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
|
Astra High round 5 is not clean at One P2 remains: a source restart or new run can supersede the incident while repair launch is awaiting target creation/message acceptance. Preparation is scheduled before the link is recorded; the link then rejects, leaving a running but unlinked repair and an RPC error. The diagnostic prompt protects newer source work but does not prevent wasted provider resources. The recommended next change is a durable acceptance step before scheduling: create an inert repair target, commit the exact-incident acceptance and target identity under the source command lock, release the lock, then schedule the ordinary target launch idempotently. The accepted identity must remain resumable if the source advances. Holding the source command lock across the current launch is unsafe because message admission acquires its mutex before thread locking. The two GitHub findings reported on |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 265cb8dd97
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
265cb8d to
f99ee59
Compare
|
The source-supersession blocker from the last Astra review is fixed in f99ee59. Repair creation is inert until the exact source incident is accepted; acceptance and the source link commit atomically under the source command lock. Provider scheduling runs afterward, outside that lock. Accepted retries recover the same target and message identity even after the source advances, including interrupted preparation. Added regressions for supersession before and after acceptance and accepted launch retries. The branch is rebased onto current lastcode/main. Further Astra High review and fresh GitHub CI are underway. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f99ee59fd1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
f99ee59 to
99772e9
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 99772e9419
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bb46a6e531
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
bb46a6e to
7e87ab0
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7e87ab0bc3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
f6e953e to
eb88ba2
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9b7ac9a2ff
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Carry-Group: incubator
Carry-Group: incubator
Carry-Group: incubator
Carry-Group: incubator
Carry-Group: incubator
Carry-Group: incubator
Carry-Group: incubator
Carry-Group: incubator
Reuse live repair conversations, reopen archived ones, and replace deleted repairs with stable retry identities. Hold the recovery incident lock through repair acceptance and linking. Carry-Group: incubator
Carry-Group: tooling
Accept repair conversations before scheduling their provider, retain accepted retries after source supersession, and reconcile released runtimes and their abandoned native background work with exact ownership guards. These recovery-specific service, adapter, persistence, and regression changes are inseparable from the incubating thread recovery feature. Carry-Group: incubator
Pair background-work observations with internal stream barriers so native completion output commits before missing outcomes are settled. Retain unknown probes and post-terminal readers, and settle exact released background ownership without adding a timer. These service, adapter, and regression changes remain inseparable from the incubating recovery feature. Carry-Group: incubator
Close subscriptions created after exact runtime release, hydrate native child roots through their durable ownership, and preserve orderly restart cancellation and its next-turn note. Correct the reported type errors and test registration and make recovery test projections reflect committed rows. These manager, service, persistence, and regression changes remain inseparable from the incubating thread recovery feature. Carry-Group: incubator
Reattach retained failed, interrupted, and cancelled turns as well as completed turns. Restore saved native child routing, preserve real queued outputs, and settle missing outcomes only after an ordered drain confirms them. Add regressions for lost completions and unknown probes while preserving exact ownership, completed history, release, and shutdown behavior. These changes are inseparable from the incubating recovery feature. Carry-Group: incubator
Keep one latest terminal per provider thread for its captured runtime lifetime instead of evicting recovery evidence after 32 unrelated threads complete. Add Codex and Claude adapter regressions for unrelated completion volume and same-thread supersession. These adapter changes remain inseparable from the incubating thread recovery feature. Carry-Group: incubator
Acknowledge exact provider turns after their outcomes are saved, retaining recovery evidence across unrelated completions and failed finalization. Carry-Group: incubator Carry-Fix: #282
Use full-access caller limits for repair and migrate the JSON recovery fixture to the current invocation shape. Update cross-project promotion assertions to distinguish visible ordinary threads from missing threads. Carry-Group: incubator
Carry-Group: incubator
Carry-Group: incubator
ca91f3c to
0eb8b8b
Compare
|
@codex review |
|
Codex Review: Didn't find any major issues. Already looking forward to the next diff. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
A merge that landed while a checkpoint was validating discarded the run: repaired checkpoints published tag and main in one atomic push, so the validated tag was rejected and the repair had to be reselected and revalidated by hand; ordinary checkpoints published their tag but failed the run at promotion. Publish the validated tag and source ref without touching main, and have promotion defer, instead of failing, when main advanced or a guarded merge holds the main write lock. The merge's own service request then publishes a revision that replays it onto the new tag and promotes that. Merges after recovery selection no longer invalidate it; only a main that no longer descends from the selected source does. Carry-Group: tooling Carry-Observation: On 2026-10-05 the merges of #280 and #282 each forced a manual fold-in and full revalidation of a repaired checkpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A merge that landed while a checkpoint was validating discarded the run: repaired checkpoints published tag and main in one atomic push, so the validated tag was rejected and the repair had to be reselected and revalidated by hand; ordinary checkpoints published their tag but failed the run at promotion. Publish the validated tag and source ref without touching main, and have promotion defer, instead of failing, when main advanced or a guarded merge holds the main write lock. The merge's own service request then publishes a revision that replays it onto the new tag and promotes that. Merges after recovery selection no longer invalidate it; only a main that no longer descends from the selected source does. Carry-Group: tooling Carry-Observation: On 2026-10-05 the merges of #280 and #282 each forced a manual fold-in and full revalidation of a repaired checkpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A merge that landed while a checkpoint was validating discarded the run: repaired checkpoints published tag and main in one atomic push, so the validated tag was rejected and the repair had to be reselected and revalidated by hand; ordinary checkpoints published their tag but failed the run at promotion. Publish the validated tag and source ref without touching main, and have promotion defer, instead of failing, when main advanced or a guarded merge holds the main write lock. The merge's own service request then publishes a revision that replays it onto the new tag and promotes that. Merges after recovery selection no longer invalidate it; only a main that no longer descends from the selected source does. Carry-Group: tooling Carry-Observation: On 2026-10-05 the merges of #280 and #282 each forced a manual fold-in and full revalidation of a repaired checkpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A merge that landed while a checkpoint was validating discarded the run: repaired checkpoints published tag and main in one atomic push, so the validated tag was rejected and the repair had to be reselected and revalidated by hand; ordinary checkpoints published their tag but failed the run at promotion. Publish the validated tag and source ref without touching main, and have promotion defer, instead of failing, when main advanced or a guarded merge holds the main write lock. The merge's own service request then publishes a revision that replays it onto the new tag and promotes that. Merges after recovery selection no longer invalidate it; only a main that no longer descends from the selected source does. Carry-Group: tooling Carry-Observation: On 2026-10-05 the merges of #280 and #282 each forced a manual fold-in and full revalidation of a repaired checkpoint. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Merging a PR while a checkpoint was validating threw the run away. Repaired checkpoints published the tag and `lastcode/main` in one `--atomic` push, so a mid-run merge rejected the validated tag too. The repair then had to be reselected, with the merge folded in by hand, and fully revalidated. Ordinary checkpoints published their tag but failed the run at promotion, which alerted the maintenance thread. On 2026-10-05, #280 and #282 each cost a manual fold-in and a 10–15 minute revalidation. The validated tag no longer depends on main: - **Repaired checkpoints** publish the tag and immutable source ref atomically, then promote separately, the same as ordinary checkpoints. - **Promotion defers instead of failing** when main has advanced to a descendant of the candidate's source or git's lease rejects the push because a descendant merge landed mid-push. A rewrite that drops the pinned source still fails promotion after tag publication, and the rejected-push path fetches the competing head before checking ancestry. The run succeeds and logs that promotion is left to the next run. Failure to acquire the promotion lock still fails the run after the tag publishes; the lock ref cannot prove its writer is active, and abandoned locks or authentication/transport errors must remain visible to maintenance. - The guarded merge already requests a service run, and the supervisor runs it straight after the current run. That run publishes a revision replaying only the merged work onto the new tag, then promotes it. If main still equals the source, it promotes the published tag directly. - **Merges after recovery selection no longer invalidate it.** Only a main that no longer descends from the selected source (for example, rewritten by another promotion) requires reselection. - **Recovery cleanup follows promotion validation.** A main rewrite or promotion failure keeps the repaired worktree and selection for inspection. Retry promotes the existing immutable tag without republishing; successful promotion or validated descendant deferral releases the repair. You can merge at any time. The build can start from the checkpoint tag immediately, and merged work follows minutes later as a revision. Runbook text in `fork-conventions.md`, `release.md`, `nightly-workflow.md` and the `lastcode-pr` skill is rewritten to match. Validation: focused checkpoint and recovery tests, selected-recovery lifecycle tests, and carry lifecycle tests pass. Lifecycle regressions cover descendant merges after selection, during validation, and during the promotion push; unrelated remote-only rewrites retain the selected repair and fail visibly while preserving the published tag and rewritten main. They also verify blocked retries preserve that repair, restored-source retries promote the existing tag, and promotion-lock failures retain recovery until a successful retry. The scripts typecheck and scoped format/lint checks pass. GitHub CI and Codex review are required on the final head and base. Quick CI was skipped at the maintainer's request; GitHub CI is the gate. Includes merged #293 as its current base. Implemented with Claude Opus 5.5 in the Claude Code harness. Refreshed and repaired with GPT-6.1-Sol in the Codex harness. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Carry-Source-Ref: refs/lastcode/carry-sources/pr-294/e70ba03bd32f5e16161d968d3da613b6bd5b0d9b Carry-Source-Base: 00eaf51 Carry-Source-Head: e70ba03
1 participant
A provider can finish or disappear after LastCode stops recording its events, leaving a thread marked Working and follow-up messages aimed at a turn that no longer responds.
This adds recovery for Codex and Claude through the existing liveness cadence. Exact provider-turn evidence lets LastCode reconcile confirmed finished turns without replaying work. When the provider state cannot be confirmed or deterministic recovery fails, the thread offers an ordinary repair conversation using the project’s default provider and model. Queued messages and existing work are preserved, and recovery receipts disclose that missed output may remain missing.
Recovery distinguishes an unknown live turn from an exact runtime the app has released. Released attempts can be safely cancelled without claiming their work completed. Recovery reconnects event recording for surviving background work and settles abandoned native work with transaction guards that preserve completed records, newer runs, and independent agents. Repair acceptance and the source link commit before provider scheduling; accepted retries retain their identity after the source advances. Archived repairs reopen and deleted repairs get stable replacement identities. The composer and sidebar present recovery progress and failure across web, desktop, and mobile.
Focused recovery, adapter, routing, repair, and client tests passed during initial implementation. GitHub CI validates the later regression additions and full suite; local Quick CI is skipped at the maintainer’s request. Isolated browser QA verified repair creation, direct navigation, reuse, source-thread MCP reads, and recovered-notice dismissal. Native mobile QA has not been run. Independent implementation review uses GPT-6 Astra High.
The requested Local Reviews rule is included verbatim in AGENTS.md: review budgets limit review rounds, and findings from the final round still get fixed before continuing PR delivery.
The feature is being tried in LastCode first. #283 tracks an upstream proposal only if a real recurrence demonstrates practical value. The UI fixtures below are synthetic and do not establish recovery of a real incident.
Before opening repair:
After creating repair, the source offers the existing conversation:
Repair-launch navigation recording.
Implemented with GPT-6 Astra and GPT-6.1 Sol through the Codex harness, with delegated implementation and Claude Opus 5.5 UI guidance. Independently reviewed with GPT-6 Astra High.