fix(runtime): #1114 — spin-throttle bypassed by NOTIFIED fast-path reset - #1181
Merged
Conversation
…path reset The #1114 spin throttle (added to catch wedges where a timer deadline reads as due-now on every iteration) reset its streak counter on every `NOTIFIED`-fast-path return. `js_notify_main_thread` is called from inside every `js_promise_resolve`, every `js_async_step_chain`, and every net/ws/http event push — so a tight async tick that does any internal promise work flips `NOTIFIED` on essentially every iteration of the event loop. The streak counter therefore never accumulated enough consecutive `budget_ms == 0` returns to cross the threshold, and the throttle was structurally bypassed exactly when it was needed. Fix: only the `cvar.wait_timeout` sleep path counts as "real progress" for streak-reset purposes. The NOTIFIED fast-path return (both the top-of-function atomic-load variant and the under-the-lock recheck variant) leaves the streak untouched — neither incrementing nor resetting it. Increments only come from the `budget_ms == 0` branch, as before. The designed sub-µs async hot path is preserved (calls that never hit `budget_ms == 0` never grow the streak), but a true "timer pinned in the past + hot internal notifies" wedge now accumulates streak across iterations and the throttle fires. Regression test `notified_interleave_does_not_mask_wedge` reproduces the bug shape: alternate `js_notify_main_thread() + js_wait_for_event()` with a perpetually-due 0 ms timer past `SPIN_THROTTLE_AFTER`, then measure one further budget-0 call. With the bug, the measured call is ~125 ns (throttle never fires). With the fix, it's ≥1 ms (throttle sleeps as designed). NOTE: this fix resolves the structural defect the spin-throttle was intended to address, but shop-admin's specific JobLoop wedge has at least one additional trigger downstream of `js_wait_for_event` — after the throttle is correctly fitted, the first `claimJobs()` tick through `@perryts/mysql` still leaves the server unable to respond to subsequent HTTP requests (first request OK at 100 ms, second times out at 30 s, server alive). The sample profile no longer shows the previous `madvise`-thrash pattern; the main thread now shows `__psynch_cvwait` dominant, which suggests the wedge is now about something else holding the runtime back (perhaps tx-release in the @perryts/mysql pool, or a tokio scheduler interaction). Leaving #1114 open with this fix as one of two components.
This was referenced May 20, 2026
Async early-return + unreached await-in-for-loop allocates ~200 MB/call (was #1114 root cause)
#1190
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The #1114 spin throttle in
js_wait_for_eventis structurally bypassed whenNOTIFIEDflips on every iteration. The fast-path return calledspin_streak_reset(), andjs_notify_main_threadfires from inside every promise resolve / async-step chain / net-event push — so a tight async tick that does any internal promise work prevents the streak counter from ever accumulating.This PR makes the streak survive across NOTIFIED fast-path returns; only an actual
cvar.wait_timeoutsleep resets it. The designed sub-µs hot path is preserved (calls that never hitbudget_ms == 0never grow the streak), but a true "timer pinned in the past + hot internal notifies" wedge now accumulates streak and the throttle fires.Repro / regression test
Added
notified_interleave_does_not_mask_wedgeincrates/perry-runtime/src/event_pump.rs. The test:js_set_timeout(0.0)so every call seesbudget_ms == 0.js_notify_main_thread()+js_wait_for_event()pastSPIN_THROTTLE_AFTER.I verified the test catches the bug: re-applying just the offending
spin_streak_reset()line on the fast path made the test fail withbest post-threshold call 125ns — the throttle is bypassed by the notify fast-path, exactly the #1114 wedge.All 5 existing event_pump tests still pass.
Honest caveat
This fixes one of the structural defects discussed on #1114, but shop-admin's specific JobLoop reproduction is not fully resolved by this change. After installing the patched perry, the first request to
/healthzreturns 200 in ~100 ms; subsequent requests time out, with the server alive at ~95 % CPU. The earliermadvise-dominant profile is gone —samplenow shows__psynch_cvwaitdominant, suggesting the main event loop is no longer the source. Best guess for what's left: the@perryts/mysqltx()connection-release path is leaving state behind that starves the hyper accept-loop, or a tokio scheduler interaction with perry's event pump. That's a deeper investigation I haven't completed.This is still worth landing on its own merits — the throttle is supposed to be a wedge backstop and it was completely bypassed for the most common wedge shape (interval timer + async work). With this fix it actually backs off as designed.
I'll keep #1114 open with both this PR linked and a note about the remaining residual.
Test plan
cargo test -p perry-runtime --release event_pump::— all 5 pass[5/7] WS brokerregistration still works (FastifyInstance.server returns a number — blocks ws/http2 upgrade piggybacking #1113 stays fixed), (b) the throttle fix alone doesn't close the shop-admin wedge (next investigation), (c) the sample profile signature changed frommadvise-dominant tocvwait-dominant, consistent with the runtime no longer thrashing on allocations.Tested against the JobLoop in
~/projects/skelpo-shop-admin/server/jobs/loop.ts, which is a 1HzsetIntervalthat awaitsclaimJobs(workerId, BATCH_SIZE)via @perryts/mysql.