Skip to content

fix(server): a message sent during a rollback no longer undoes it - #17079

Merged
t3dotgg merged 1 commit into
mainfrom
t3/rollback-thread-lock
Oct 8, 2026
Merged

t3dotgg merged 1 commit into
mainfrom
t3/rollback-thread-lock

Conversation

@t3dotgg

@t3dotgg t3dotgg commented Oct 8, 2026

Copy link
Copy Markdown
Member

Problem

If you send a message while a rollback is still running, the next turn can run on the conversation the rollback was supposed to remove.

The rollback service writes its result (for Pi, the new forked session file) outside the per-thread lock that thread commands use. A message.dispatch that read the provider thread before that write, and committed after it, wrote the old session back. The following turn then failed with "Pi turn requested for a different native session". With a real provider it would continue the old, un-rolled-back history.

In the event log of a failing run, seq 71 (rollback) writes session-2.jsonl, and seq 75 (message-3) writes session-1.jsonl back.

Fix

CheckpointRollbackService now builds and commits its events under ThreadCommandExecutor.withLock(threadId). ProviderEventIngestor already uses this pattern for the same reason. The provider call stays outside the lock, so a slow rollback does not block the thread's commands.

Proof

  • The rewinds safely cases now assert that the thread lock is held when the rollback commits. 4 of them fail on main's service and pass with this change.
  • While testing a worker-thread SQLite connection (not part of this PR), the timing window got wider and thread_rollback/pi, thread_rollback_after_stop/pi and thread_rollback/claudeAgent failed on every run. All three pass with this fix. The full replay fixture suite, the rollback, restore-safety and effect-worker tests, and the runtime layer tests pass on main's SQLite (240 tests).

Reviewed with sol-loop: 1 round with GPT-6.1 Sol on high.

Made with Claude Opus 5.5 in Claude Code.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added vouch:trusted PR author is trusted by repo permissions or the VOUCHED list. size:S 10-29 changed lines (additions + deletions). labels Oct 8, 2026
@juliusmarminge juliusmarminge added the macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews label Oct 8, 2026
@macroscopeapp

macroscopeapp Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Approvability

Verdict: Approved at d75dd63

Macroscope's review found this PR approvable — This is a focused concurrency fix that serializes rollback commits with existing per-thread commands while keeping the slow provider operation outside the lock. The remaining changes only wire and test the existing lock, with no schema, default, security, or deployment impact.

You can add or adjust custom eligibility rules. Learn more.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Thread transfer impact

✅ Thread transfer remains within every enforced ceiling.

Provider Metric Main baseline This PR Impact PR ceiling
Codex Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Codex Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Codex Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Codex Live turn WebSocket decoded 20.9 KiB 20.9 KiB 0 B (0.0%) 29.3 KiB ✅
Codex Live turn messages 2 2 0 (0.0%) 8 ✅
Claude Total thread wire 5.0 KiB 5.0 KiB 0 B (0.0%) 6.8 KiB ✅
Claude Thread snapshot wire 3.8 KiB 3.8 KiB 0 B (0.0%) 4.9 KiB ✅
Claude Live turn WebSocket wire 1.2 KiB 1.2 KiB 0 B (0.0%) 2.0 KiB ✅
Claude Live turn WebSocket decoded 21.2 KiB 21.2 KiB 0 B (0.0%) 29.3 KiB ✅
Claude Live turn messages 2 2 0 (0.0%) 8 ✅

Baseline: 0bfd9dd · PR result: d75dd63 · Source CI: success

Scenario and decoded snapshot size

10 historical turns, 5 command tools per turn, 878.9 KiB retained MCP result per historical turn, and a 1.05 MiB retained result in the measured turn.

  • Codex decoded thread snapshot: 108.5 KiB
  • Claude decoded thread snapshot: 108.8 KiB

Updated in place by a trusted workflow. PR artifacts are strictly validated and never executed.

@coderabbitai

coderabbitai Bot commented Oct 8, 2026

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Path: .coderabbit.config.ts
  • Review profile: CHILL
  • Plan: Team
  • Run ID: 8ea69486-738e-42fe-b04f-c5a5fb26e0c3
📥 Commits

Reviewing files that changed from the base of the PR and between 0bfd9dd and d75dd63.

📒 Files selected for processing (5)
  • apps/server/src/orchestration-v2/CheckpointRestoreSafety.test.ts
  • apps/server/src/orchestration-v2/CheckpointRollbackService.test.ts
  • apps/server/src/orchestration-v2/CheckpointRollbackService.ts
  • apps/server/src/orchestration-v2/runtimeLayer.ts
  • apps/server/src/orchestration-v2/testkit/ProviderReplayHarness.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.


📝 Walkthrough

Walkthrough

Checkpoint rollback now writes its rollback events under a per-thread command lock. The rollback service requires ThreadCommandExecutor, which is provided by the runtime, replay harness, and tests.

Changes

Checkpoint rollback locking

Layer / File(s) Summary
Lock rollback event writes
apps/server/src/orchestration-v2/CheckpointRollbackService.ts, apps/server/src/orchestration-v2/CheckpointRollbackService.test.ts
The rollback service moves rollback event construction and writing into a lock for the input thread. The safe-rewind test uses a lock-backed executor and checks that the thread is locked during the event write.
Provide the thread command executor
apps/server/src/orchestration-v2/runtimeLayer.ts, apps/server/src/orchestration-v2/testkit/ProviderReplayHarness.ts, apps/server/src/orchestration-v2/CheckpointRollbackService.test.ts, apps/server/src/orchestration-v2/CheckpointRestoreSafety.test.ts
The runtime layer, replay harness, and rollback test setups provide the executor required by the service.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~12 minutes

Change: Bug fix

Suggested reviewers: juliusmarminge

Merge Risk: ⚪ Minimal · up to d75dd

The rollback write and message dispatch share the intended thread lock in the inspected production setup. No actionable issue remains before merge.

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main fix: preventing a message sent during rollback from undoing the rollback.
Description check ✅ Passed The description explains the problem, expected failure, fix, locking behavior, and focused verification results. It uses equivalent headings for Change and Verification; the Scope and approval section…
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@t3dotgg
t3dotgg merged commit c28cd90 into main Oct 8, 2026
33 checks passed
@t3dotgg
t3dotgg deleted the t3/rollback-thread-lock branch October 8, 2026 05:20
github-actions Bot added a commit to omarcresp/t3code-flake that referenced this pull request Oct 8, 2026
## What's Changed
* chore(review): remove the custom Approvability check by @esthor in pingdotgg/t3code#17018
* perf(mobile): show the cached thread list sooner and stop the freeze after it by @juliusmarminge in pingdotgg/t3code#16713
* fix(server): replaying a command no longer freezes the server by @SunkenInTime in pingdotgg/t3code#17041
* fix(server): opening a long thread no longer blocks the server for a second by @SunkenInTime in pingdotgg/t3code#17029
* fix(server,web,mobile): thread links reference the thread id, not a baked title by @juliusmarminge in pingdotgg/t3code#17017
* fix(web): messages with quotes or links no longer collapse when short by @flamboh in pingdotgg/t3code#16624
* refactor(web): PR loading skeleton shares the detail panel's layout by @flamboh in pingdotgg/t3code#15583
* fix(web): PR file headers keep the file name in narrow panels by @flamboh in pingdotgg/t3code#15562
* fix(web): PR timeline no longer shifts when its scrollbar appears by @flamboh in pingdotgg/t3code#15588
* fix(server): held queued wakes no longer keep delegated tasks running by @juliusmarminge in pingdotgg/t3code#17028
* fix(server): a message sent during a rollback no longer undoes it by @t3dotgg in pingdotgg/t3code#17079
* fix(web): PR file stats ignore the hide-whitespace toggle by @flamboh in pingdotgg/t3code#16162
* perf(mobile): omit duplicated turn items from bounded thread snapshots by @juliusmarminge in pingdotgg/t3code#15385
* perf(mobile): pause elapsed-time timers on hidden thread screens by @juliusmarminge in pingdotgg/t3code#15397
* fix(mobile): pause hidden home thread list updates by @juliusmarminge in pingdotgg/t3code#15705
* fix(mobile): restore file viewer insets and glass header by @juliusmarminge in pingdotgg/t3code#17073
* fix(web): PR code toolbar no longer overlaps in narrow panels by @flamboh in pingdotgg/t3code#15561
* fix(web): PR commit menu no longer stretches across the window by @flamboh in pingdotgg/t3code#15560
* fix(web): command palette scrollbar no longer clipped at the top by @flamboh in pingdotgg/t3code#17035
* fix(web): Usage breadcrumb stays centered on small viewports by @flamboh in pingdotgg/t3code#15552
* fix(mobile): stop refreshing Git status on streamed thread updates by @juliusmarminge in pingdotgg/t3code#15893
* fix(mobile): skip move indexes for empty and single-thread sections by @juliusmarminge in pingdotgg/t3code#16115
* perf(mobile): reuse encoded rows in shell cache saves by @juliusmarminge in pingdotgg/t3code#16129
* perf(mobile): skip showcase subscriptions in normal builds by @juliusmarminge in pingdotgg/t3code#16131
* perf(mobile): remove unused Home project sorting by @juliusmarminge in pingdotgg/t3code#16177
* perf(mobile): reduce move-menu index allocations by @juliusmarminge in pingdotgg/t3code#16256
* perf(mobile): skip impossible thread-key lookups by @juliusmarminge in pingdotgg/t3code#16263
* fix(mobile): collect UI runtime garbage on iOS memory warnings by @juliusmarminge in pingdotgg/t3code#16296
* fix(mobile): stop Git sheet refresh loop by @juliusmarminge in pingdotgg/t3code#16305
* fix(mobile): refresh Git status after reconnect by @juliusmarminge in pingdotgg/t3code#16329
* fix(mobile): update the iOS Git header menu when status changes by @juliusmarminge in pingdotgg/t3code#16330
* perf(mobile): reuse the settled sort when settled rows are unchanged by @juliusmarminge in pingdotgg/t3code#16369
* perf(mobile): highlight source files in small batches that keep grammar state by @juliusmarminge in pingdotgg/t3code#16729
* feat(web): quote chips show what you said about the quote by @flamboh in pingdotgg/t3code#15703
* feat(web): projectless threads show their machine in the sidebar by @flamboh in pingdotgg/t3code#17022
* fix(lineage): keep agent effort and speed after completion by @Bil0000 in pingdotgg/t3code#16925
* fix(mobile): align diff scrolling with glass headers by @juliusmarminge in pingdotgg/t3code#17085
* fix(web): workspace page headers can no longer grow past the top-bar height by @maria-rcks in pingdotgg/t3code#17086
* feat(mobile): redesign the Add environment sheet by @juliusmarminge in pingdotgg/t3code#17092


**Full Changelog**: pingdotgg/t3code@v0.0.46-nightly.20261008.2801...v0.0.46-nightly.20261008.2813

Upstream release: https://github.com/pingdotgg/t3code/releases/tag/v0.0.46-nightly.20261008.2813
github-actions Bot added a commit to davidvanderklay/t3code-flake that referenced this pull request Oct 8, 2026
## What's Changed
* chore(review): remove the custom Approvability check by @esthor in pingdotgg/t3code#17018
* perf(mobile): show the cached thread list sooner and stop the freeze after it by @juliusmarminge in pingdotgg/t3code#16713
* fix(server): replaying a command no longer freezes the server by @SunkenInTime in pingdotgg/t3code#17041
* fix(server): opening a long thread no longer blocks the server for a second by @SunkenInTime in pingdotgg/t3code#17029
* fix(server,web,mobile): thread links reference the thread id, not a baked title by @juliusmarminge in pingdotgg/t3code#17017
* fix(web): messages with quotes or links no longer collapse when short by @flamboh in pingdotgg/t3code#16624
* refactor(web): PR loading skeleton shares the detail panel's layout by @flamboh in pingdotgg/t3code#15583
* fix(web): PR file headers keep the file name in narrow panels by @flamboh in pingdotgg/t3code#15562
* fix(web): PR timeline no longer shifts when its scrollbar appears by @flamboh in pingdotgg/t3code#15588
* fix(server): held queued wakes no longer keep delegated tasks running by @juliusmarminge in pingdotgg/t3code#17028
* fix(server): a message sent during a rollback no longer undoes it by @t3dotgg in pingdotgg/t3code#17079
* fix(web): PR file stats ignore the hide-whitespace toggle by @flamboh in pingdotgg/t3code#16162
* perf(mobile): omit duplicated turn items from bounded thread snapshots by @juliusmarminge in pingdotgg/t3code#15385
* perf(mobile): pause elapsed-time timers on hidden thread screens by @juliusmarminge in pingdotgg/t3code#15397
* fix(mobile): pause hidden home thread list updates by @juliusmarminge in pingdotgg/t3code#15705
* fix(mobile): restore file viewer insets and glass header by @juliusmarminge in pingdotgg/t3code#17073
* fix(web): PR code toolbar no longer overlaps in narrow panels by @flamboh in pingdotgg/t3code#15561
* fix(web): PR commit menu no longer stretches across the window by @flamboh in pingdotgg/t3code#15560
* fix(web): command palette scrollbar no longer clipped at the top by @flamboh in pingdotgg/t3code#17035
* fix(web): Usage breadcrumb stays centered on small viewports by @flamboh in pingdotgg/t3code#15552
* fix(mobile): stop refreshing Git status on streamed thread updates by @juliusmarminge in pingdotgg/t3code#15893
* fix(mobile): skip move indexes for empty and single-thread sections by @juliusmarminge in pingdotgg/t3code#16115
* perf(mobile): reuse encoded rows in shell cache saves by @juliusmarminge in pingdotgg/t3code#16129
* perf(mobile): skip showcase subscriptions in normal builds by @juliusmarminge in pingdotgg/t3code#16131
* perf(mobile): remove unused Home project sorting by @juliusmarminge in pingdotgg/t3code#16177
* perf(mobile): reduce move-menu index allocations by @juliusmarminge in pingdotgg/t3code#16256
* perf(mobile): skip impossible thread-key lookups by @juliusmarminge in pingdotgg/t3code#16263
* fix(mobile): collect UI runtime garbage on iOS memory warnings by @juliusmarminge in pingdotgg/t3code#16296
* fix(mobile): stop Git sheet refresh loop by @juliusmarminge in pingdotgg/t3code#16305
* fix(mobile): refresh Git status after reconnect by @juliusmarminge in pingdotgg/t3code#16329
* fix(mobile): update the iOS Git header menu when status changes by @juliusmarminge in pingdotgg/t3code#16330
* perf(mobile): reuse the settled sort when settled rows are unchanged by @juliusmarminge in pingdotgg/t3code#16369
* perf(mobile): highlight source files in small batches that keep grammar state by @juliusmarminge in pingdotgg/t3code#16729
* feat(web): quote chips show what you said about the quote by @flamboh in pingdotgg/t3code#15703
* feat(web): projectless threads show their machine in the sidebar by @flamboh in pingdotgg/t3code#17022
* fix(lineage): keep agent effort and speed after completion by @Bil0000 in pingdotgg/t3code#16925
* fix(mobile): align diff scrolling with glass headers by @juliusmarminge in pingdotgg/t3code#17085
* fix(web): workspace page headers can no longer grow past the top-bar height by @maria-rcks in pingdotgg/t3code#17086
* feat(mobile): redesign the Add environment sheet by @juliusmarminge in pingdotgg/t3code#17092


**Full Changelog**: pingdotgg/t3code@v0.0.46-nightly.20261008.2801...v0.0.46-nightly.20261008.2813

Upstream release: https://github.com/pingdotgg/t3code/releases/tag/v0.0.46-nightly.20261008.2813
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

macroscope-review Opt PRs made by unvouched contributors in for Macroscope review. Vouched contributors auto-reviews size:S 10-29 changed lines (additions + deletions). vouch:trusted PR author is trusted by repo permissions or the VOUCHED list.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants