Skip to content

fix(runtime): #1114 — spin-throttle bypassed by NOTIFIED fast-path reset - #1181

Merged
proggeramlug merged 1 commit into
mainfrom
fix/1114-spin-throttle
May 20, 2026
Merged

fix(runtime): #1114 — spin-throttle bypassed by NOTIFIED fast-path reset#1181
proggeramlug merged 1 commit into
mainfrom
fix/1114-spin-throttle

Conversation

@proggeramlug

Copy link
Copy Markdown
Contributor

Summary

The #1114 spin throttle in js_wait_for_event is structurally bypassed when NOTIFIED flips on every iteration. The fast-path return called spin_streak_reset(), and js_notify_main_thread fires from inside every promise resolve / async-step chain / net-event push — so a tight async tick that does any internal promise work prevents the streak counter from ever accumulating.

This PR makes the streak survive across NOTIFIED fast-path returns; only an actual cvar.wait_timeout sleep resets it. The designed sub-µs hot path is preserved (calls that never hit budget_ms == 0 never grow the streak), but a true "timer pinned in the past + hot internal notifies" wedge now accumulates streak and the throttle fires.

Repro / regression test

Added notified_interleave_does_not_mask_wedge in crates/perry-runtime/src/event_pump.rs. The test:

  1. Pins a js_set_timeout(0.0) so every call sees budget_ms == 0.
  2. Alternates js_notify_main_thread() + js_wait_for_event() past SPIN_THROTTLE_AFTER.
  3. Measures one further budget-0 call and asserts it's ≥700 µs.

I verified the test catches the bug: re-applying just the offending spin_streak_reset() line on the fast path made the test fail with best post-threshold call 125ns — the throttle is bypassed by the notify fast-path, exactly the #1114 wedge.

All 5 existing event_pump tests still pass.

Honest caveat

This fixes one of the structural defects discussed on #1114, but shop-admin's specific JobLoop reproduction is not fully resolved by this change. After installing the patched perry, the first request to /healthz returns 200 in ~100 ms; subsequent requests time out, with the server alive at ~95 % CPU. The earlier madvise-dominant profile is gone — sample now shows __psynch_cvwait dominant, suggesting the main event loop is no longer the source. Best guess for what's left: the @perryts/mysql tx() connection-release path is leaving state behind that starves the hyper accept-loop, or a tokio scheduler interaction with perry's event pump. That's a deeper investigation I haven't completed.

This is still worth landing on its own merits — the throttle is supposed to be a wedge backstop and it was completely bypassed for the most common wedge shape (interval timer + async work). With this fix it actually backs off as designed.

I'll keep #1114 open with both this PR linked and a note about the remaining residual.

Test plan

  • cargo test -p perry-runtime --release event_pump:: — all 5 pass
  • Toggled the buggy line back on, ran just the new test — failed as expected with a sub-µs post-threshold call
  • Toggled fix back in — new test passes (≥700 µs measured)
  • Built v0.5.1017 and reinstalled, re-ran shop-admin's e2e — confirms (a) the boot-time [5/7] WS broker registration still works (FastifyInstance.server returns a number — blocks ws/http2 upgrade piggybacking #1113 stays fixed), (b) the throttle fix alone doesn't close the shop-admin wedge (next investigation), (c) the sample profile signature changed from madvise-dominant to cvwait-dominant, consistent with the runtime no longer thrashing on allocations.

Tested against the JobLoop in ~/projects/skelpo-shop-admin/server/jobs/loop.ts, which is a 1Hz setInterval that awaits claimJobs(workerId, BATCH_SIZE) via @perryts/mysql.

…path reset

The #1114 spin throttle (added to catch wedges where a timer deadline
reads as due-now on every iteration) reset its streak counter on every
`NOTIFIED`-fast-path return. `js_notify_main_thread` is called from
inside every `js_promise_resolve`, every `js_async_step_chain`, and
every net/ws/http event push — so a tight async tick that does any
internal promise work flips `NOTIFIED` on essentially every iteration
of the event loop. The streak counter therefore never accumulated
enough consecutive `budget_ms == 0` returns to cross the threshold,
and the throttle was structurally bypassed exactly when it was needed.

Fix: only the `cvar.wait_timeout` sleep path counts as "real progress"
for streak-reset purposes. The NOTIFIED fast-path return (both the
top-of-function atomic-load variant and the under-the-lock recheck
variant) leaves the streak untouched — neither incrementing nor
resetting it. Increments only come from the `budget_ms == 0` branch,
as before. The designed sub-µs async hot path is preserved (calls
that never hit `budget_ms == 0` never grow the streak), but a true
"timer pinned in the past + hot internal notifies" wedge now
accumulates streak across iterations and the throttle fires.

Regression test `notified_interleave_does_not_mask_wedge` reproduces
the bug shape: alternate `js_notify_main_thread() + js_wait_for_event()`
with a perpetually-due 0 ms timer past `SPIN_THROTTLE_AFTER`, then
measure one further budget-0 call. With the bug, the measured call is
~125 ns (throttle never fires). With the fix, it's ≥1 ms (throttle
sleeps as designed).

NOTE: this fix resolves the structural defect the spin-throttle was
intended to address, but shop-admin's specific JobLoop wedge has at
least one additional trigger downstream of `js_wait_for_event` —
after the throttle is correctly fitted, the first `claimJobs()` tick
through `@perryts/mysql` still leaves the server unable to respond
to subsequent HTTP requests (first request OK at 100 ms, second
times out at 30 s, server alive). The sample profile no longer shows
the previous `madvise`-thrash pattern; the main thread now shows
`__psynch_cvwait` dominant, which suggests the wedge is now about
something else holding the runtime back (perhaps tx-release in the
@perryts/mysql pool, or a tokio scheduler interaction). Leaving #1114
open with this fix as one of two components.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant