Skip to content

Terminate workers blocked in Atomics.wait() when recording - #175

Merged
Andarist merged 2 commits into
masterfrom
andarist/worker-terminate-hang
Oct 7, 2026
Merged

Andarist merged 2 commits into
masterfrom
andarist/worker-terminate-hang

Conversation

@Andarist

@Andarist Andarist commented Oct 7, 2026 •

Copy link
Copy Markdown
Member

A worker blocked in Atomics.wait() and terminated while recording never stopped, so the process never exited. The stop watchdog (#173) invalidated the recording after 5s and terminated the worker's execution, which woke the futex wait. But FutexEmulation::WaitSync handles interrupts with events disallowed (#170), and StackGuard::HandleInterrupts ignores interrupts while events are disallowed (#173, ported from replayio/chromium-v8#30). The termination stayed pending and the waiter went back to waiting on the futex emulation's condition variable, with the parent idle in its event loop, waiting for the worker's exit.

WaitSync now handles a termination that HandleInterrupts left pending. Only another thread can terminate a thread blocked there, which already invalidates the recording (StackGuard::RequestInterrupt), so this doesn't change what a usable recording or its replay does. Other interrupts are still left for the next stack check, and without recording HandleInterrupts has already handled the termination, so the check is a no-op.

Once the recording is finished, Worker::Exit now stops workers right away instead of posting the stop to their event loop and starting the stop watchdog. When the parent exits, through process.exit() or the end of its event loop, the recording is already finished before stop_sub_worker_contexts runs, so a deterministic stop gains nothing: a worker that never returned to its event loop, e.g. one blocked in Atomics.wait(), held up the exit for the watchdog's 5s. With a stuck worker, exiting now takes about as long as on stock node 16.

Tested with https://github.com/replayio/backend/pull/13450 and its test/node-recording/tests/worker-threads/terminate-stuck/

A worker blocked in Atomics.wait() and terminated while recording never
stopped, so the process never exited. The stop watchdog (#173) invalidated
the recording after 5s and terminated the worker's execution, which woke the
futex wait, but FutexEmulation::WaitSync handles interrupts with events
disallowed (#170), and StackGuard::HandleInterrupts ignores interrupts while
events are disallowed (#173, ported from replayio/chromium-v8#30). The
termination stayed pending and the waiter went back to waiting on the futex
emulation's condition variable, with the parent idle in its event loop,
waiting for the worker's exit.

WaitSync now handles a termination that HandleInterrupts left pending.
Only another thread can terminate a thread blocked there, which already
invalidates the recording (StackGuard::RequestInterrupt), so this doesn't
change what a usable recording or its replay does. Other interrupts are still
left for the next stack check, and without recording HandleInterrupts has
already handled the termination, so the check is a no-op.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Andarist
Andarist requested a review from Domiii October 7, 2026 11:58
When recording or replaying, Worker::Exit called from another thread posts
the stop to the worker's event loop and starts the stop watchdog (#173), so
that the worker stops at a point the replay can reproduce. When the parent
exits, through process.exit() or the end of its event loop, the recording is
already finished (RecordReplayFinishRecording runs before
stop_sub_worker_contexts), so there is nothing left to reproduce: a worker
that never returned to its event loop, e.g. one blocked in Atomics.wait(),
held up the exit for the watchdog's 5s for no benefit. Worker::Exit now
stops the worker right away once the recording is finished, as without
recording.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Andarist
Andarist force-pushed the andarist/worker-terminate-hang branch from ba44be5 to b9189dc Compare October 7, 2026 12:27
@Andarist
Andarist merged commit 87ef8b5 into master Oct 7, 2026
2 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant