Avoid NOTE_ABSOLUTE timers for UPTIME/MONOTONIC on FreeBSD/OpenBSD.#943
Avoid NOTE_ABSOLUTE timers for UPTIME/MONOTONIC on FreeBSD/OpenBSD.#943max-potapov wants to merge 1 commit into
Conversation
### Motivation: libdispatch programs every EVFILT_TIMER with NOTE_ABSOLUTE on every platform, computing an absolute deadline value via `_dispatch_time_now_cached` and passing it to kqueue with `NOTE_NSECONDS | NOTE_ABSOLUTE` (= `NOTE_ABSTIME` on BSD). On FreeBSD/OpenBSD only the WALL clock works that way — kqueue interprets the `NOTE_NSECONDS|NOTE_ABSTIME` value as CLOCK_REALTIME nanoseconds, which matches what libdispatch computes for `DISPATCH_CLOCK_WALL`. For UPTIME / MONOTONIC, libdispatch passes a CLOCK_MONOTONIC nanosecond value that BSD kqueue still interprets as realtime (decades in the past after boot). kqueue reports the timer as already-expired on every register, so the dispatch event loop fires the timer block immediately, re-arms the timer, fires it again, and so on. Observed on a long-running HummingbirdCore HTTP server using `Task.sleep` as ~48 000 kevent()/s with one CPU core pinned to ~100 % even with no real work pending. PRs swiftlang#879 and swiftlang#931 corrected the NOTE_NSECONDS scaling for the relative-time path on FreeBSD but did not touch the absolute-time deadline computation this PR addresses. ### Modifications: For `os(FreeBSD)` and `os(OpenBSD)`, split the per-clock flag selection so NOTE_ABSOLUTE is kept for DISPATCH_CLOCK_WALL (which works correctly on BSD) but dropped for DISPATCH_CLOCK_UPTIME and DISPATCH_CLOCK_MONOTONIC. Introduce `DISPATCH_NOTE_ABSOLUTE_<kind>` macros next to the existing `DISPATCH_NOTE_CLOCK_<kind>` ones, and have `_dispatch_timer_index_to_fflags` use them. In `_dispatch_event_loop_timer_arm`, when running on BSD with a non-WALL clock, convert the absolute deadline back to a relative delay by subtracting the cached `now`. This preserves any leeway already folded into `target` by `_dispatch_timers_force_max_leeway` above (a simpler `target = range.delay` would silently drop that addition because `range.leeway` has been zeroed by that point). The change is guarded behind `#if defined(__FreeBSD__) || defined(__OpenBSD__)` and is a no-op on every other platform. ### Result: Reproduced on FreeBSD 15.0-RELEASE-p9 with the official Apple Swift FreeBSD preview toolchain (Swift 6.3-dev) by running the same Swift binary (HummingbirdCore HTTP server + a single periodic `Task.sleep(for: .seconds(3600))` loop, no real probes/work) against both upstream and patched libdispatch.so via LD_LIBRARY_PATH: ``` === upstream libdispatch === %CPU RSS COMMAND 37.9 39736 /tmp/netwatch-exporter syscall seconds calls errors _umtx_op 3.999564140 16 0 kevent 0.716818577 117827 0 === patched libdispatch === %CPU RSS COMMAND 1.0 39820 /tmp/netwatch-exporter syscall seconds calls errors _umtx_op 3.999207012 16 0 kevent 1.999276981 10 0 ``` Same binary, same machine, same Swift toolchain — only `/opt/swift/lib/swift/freebsd/libdispatch.so` swapped. kevent rate drops by ~11 000×; CPU drops from one core pinned to ~idle. In production (a Hummingbird-based Prometheus exporter on FreeBSD 15) the same swap took the service from 70-98 % CPU down to 0.0 %. Existing `dispatch_timer*` tests (`dispatch_timer`, `dispatch_timer_short`, `dispatch_timer_timeout`, `dispatch_timer_set_time`, `dispatch_timer_bit31`, `dispatch_timer_bit63`) pass on FreeBSD with both upstream and patched libdispatch — they exercise the user-facing dispatch_after / DispatchSource APIs which complete before the busy loop can settle, so they do not detect the underlying bug. Two smoke tests documenting the per-clock split also pass cleanly on the patched build: ``` $ ./dispatch_timer_walltime dispatch_after(walltime + 2.0s) returned after 2.001s PASS $ ./dispatch_timer_uptime uptime test: dispatch_after(NOW + 2s) returned after 2.001s PASS ``` These two C-level reproducers are tiny and could be folded into the existing tests/ directory if maintainers want them landed — happy to add as a follow-up commit.
caa5570 to
9c26068
Compare
| /* On BSD, kqueue with NOTE_NSECONDS|NOTE_ABSTIME interprets data as a | ||
| * CLOCK_REALTIME nanosecond absolute deadline. That matches the value | ||
| * libdispatch computes for DISPATCH_CLOCK_WALL, so wall timers stay on | ||
| * the absolute-time path and continue to track wall-clock adjustments. | ||
| * | ||
| * For UPTIME / MONOTONIC libdispatch hands kqueue a CLOCK_MONOTONIC | ||
| * nanosecond value which BSD kqueue would interpret as realtime | ||
| * (decades in the past after boot), reporting the timer as | ||
| * already-expired on every register and producing a kevent() tight | ||
| * loop. Drop NOTE_ABSOLUTE for these clocks and let | ||
| * _dispatch_event_loop_timer_arm fall through to the relative-time | ||
| * path below. */ |
There was a problem hiding this comment.
IMO this comment reads a little more complicated than it probably needs to, and maybe the one below as well. The root of the problem seems to be that _dispatch_time_now_cached is assumed below to always returrn an absolute time, but our UPTIME/MONOTONIC (which are synonyms, at least on FreeBSD) are relative to boot @ time=0.
Writing that out loud, I'm wondering if this breaks other assumptions of users that consume these clocks through _dispatch_time_now* -- it might be better to have _dispatch_uptime and _dispatch_monotonic_time fetch kern.boottime and add that in to avoid bugs like this in the future.
There was a problem hiding this comment.
(That is: I see the note in the PR description, but I'm not sure that's actually a much larger change and it avoids other inconsistencies at the same time)
There was a problem hiding this comment.
Thanks @kevans91 – agreed, I can trim the comments.
I also agree that normalizing the BSD uptime/monotonic path around boottime is probably the cleaner architectural fix if those values are meant to feed the absolute-deadline timer path.
My hesitation is mostly validation, not preference for the narrow patch. The existing dispatch_timer tests pass on FreeBSD with both upstream and this patch, and the simple dispatch_after wall/uptime smoke tests also pass. So they don't currently exercise the long-running re-arm/tight-loop behavior that exposed this in the integration workload.
Before I rework this toward the boottime-based fix, what validation would you consider sufficient here? I can either:
- keep this PR as the narrow fix and a record of the bug,
- rework it into the boottime-based approach and validate with the current suite plus the original repro,
- or first try to add a BSD-specific regression test that actually fails upstream.
I'm fine with whichever path is more useful for the project.
There was a problem hiding this comment.
Thanks @kevans91 – agreed, I can trim the comments.
I also agree that normalizing the BSD uptime/monotonic path around boottime is probably the cleaner architectural fix if those values are meant to feed the absolute-deadline timer path.
My hesitation is mostly validation, not preference for the narrow patch. The existing dispatch_timer tests pass on FreeBSD with both upstream and this patch, and the simple dispatch_after wall/uptime smoke tests also pass. So they don't currently exercise the long-running re-arm/tight-loop behavior that exposed this in the integration workload.
Before I rework this toward the boottime-based fix, what validation would you consider sufficient here? I can either:
- keep this PR as the narrow fix and a record of the bug,
- rework it into the boottime-based approach and validate with the current suite plus the original repro,
- or first try to add a BSD-specific regression test that actually fails upstream.
IMO option 3 is the better option here- presumably you wouldn't really need to call it a BSD-specific test: it may not provide the exact same value on Linux, but it exercises a path through libdispatch that is seemingly not well-exercised today (whether that be kqueue or epoll/etc).
I'm fine with whichever path is more useful for the project.
With a disclaimer that I'm only tangentially involved in libdispatch things, though heavily involved in FreeBSD itself- I'd probably want someone else to stamp the path before you expand too much energy on it since their standards and values will certainly differ.
Motivation
libdispatch programs every
EVFILT_TIMERwithNOTE_ABSOLUTEon everyplatform, computing an absolute deadline value via
_dispatch_time_now_cachedand passing it to kqueue withNOTE_NSECONDS | NOTE_ABSOLUTE(=NOTE_ABSTIMEon BSD).On FreeBSD / OpenBSD only the WALL clock works that way: kqueue with
NOTE_NSECONDS | NOTE_ABSTIMEinterprets the data as aCLOCK_REALTIMEnanosecond absolute deadline (FreeBSD subtracts theboot offset internally; OpenBSD documents the realtime semantics
directly). That matches the value libdispatch already computes for
DISPATCH_CLOCK_WALL, so wall timers behave correctly and continueto track wall-clock adjustments.
For
DISPATCH_CLOCK_UPTIME/DISPATCH_CLOCK_MONOTONIC, libdispatchhands kqueue a
CLOCK_MONOTONICnanosecond value that BSD kqueueinterprets as realtime — decades in the past after boot. kqueue
reports the timer as already-expired on every register, so the
dispatch event loop fires the timer block immediately, re-arms the
timer, fires it again, and so on. Observed on a long-running
HummingbirdCore HTTP server using
Task.sleepas ~48 000 kevent()/swith one CPU core pinned to ~100 % even with no real work pending.
PRs #879 and #931 corrected the
NOTE_NSECONDSscaling for therelative-time path on FreeBSD but did not touch the absolute-time
deadline computation this PR addresses.
Modifications
For
os(FreeBSD)andos(OpenBSD), split the per-clock flagselection so that NOTE_ABSOLUTE is kept for
DISPATCH_CLOCK_WALL(which works correctly on BSD) but dropped for
DISPATCH_CLOCK_UPTIMEandDISPATCH_CLOCK_MONOTONIC. IntroduceDISPATCH_NOTE_ABSOLUTE_<kind>macros next to the existingDISPATCH_NOTE_CLOCK_<kind>ones, and have_dispatch_timer_index_to_fflagsuse them.In
_dispatch_event_loop_timer_arm, when running on BSD with anon-WALL clock, convert the absolute deadline back to a relative
delay by subtracting the cached
now. Usingtarget -= nowratherthan the simpler
target = range.delaypreserves any leeway alreadyfolded into
targetby_dispatch_timers_force_max_leewayabove(
range.leewayhas been zeroed by that point, so an overwrite wouldsilently drop the addition and break
LIBDISPATCH_TIMERS_FORCE_MAX_LEEWAY).The change is guarded behind
#if defined(__FreeBSD__) || defined(__OpenBSD__)and is a no-op on every other platform.Result
Reproduced on FreeBSD 15.0-RELEASE-p9 with the official Apple Swift
FreeBSD preview toolchain (Swift 6.3-dev) by running the same Swift
binary (HummingbirdCore HTTP server + a single periodic
Task.sleep(for: .seconds(3600))loop, no real probes/work) againstboth upstream and patched
libdispatch.soviaLD_LIBRARY_PATH:Same binary, same machine, same Swift toolchain — only
/opt/swift/lib/swift/freebsd/libdispatch.soswapped.keventratedrops by ~11 000×; CPU drops from one core pinned to ~idle.
In production (a Hummingbird-based Prometheus exporter on FreeBSD 15)
the same swap took the service from 70-98 % CPU down to 0.0 %.
Checks
dispatch_timer*tests (dispatch_timer,dispatch_timer_short,dispatch_timer_timeout,dispatch_timer_set_time,dispatch_timer_bit31,dispatch_timer_bit63) pass on FreeBSD with both upstreamand patched libdispatch. (They exercise the user-facing
dispatch_after / DispatchSource APIs which complete before the
busy loop can settle, so they don't independently detect the
underlying tight-loop bug.)
dispatch_after(dispatch_walltime(NULL, 2s))returns at 2.001 son the patched build, identical to upstream.
dispatch_after(dispatch_time(DISPATCH_TIME_NOW, 2s))returnsat 2.001 s on the patched build.
#if defined(__FreeBSD__) || defined(__OpenBSD__); no behaviour change on Apple/Linux.LIBDISPATCH_TIMERS_FORCE_MAX_LEEWAYsemantics preserved —the relative-time conversion (
target -= now) keeps any leewayalready folded into
targetabove.Notes
requires the Swift Concurrency cooperative executor + a concurrent
NIO event loop to manifest, which is awkward to express against
libdispatch's existing test suite without depending on the Swift
runtime. The two smoke tests above (walltime / uptime
dispatch_after) document the per-clock split and could be foldedinto
tests/if maintainers want them landed — happy to add as afollow-up commit.
NOTE_NSECONDS|NOTE_ABSTIMEclock semantics in_dispatch_time_now_cacheditself (so uptime/monotonic values getconverted to realtime before being handed to kqueue). This PR
intentionally takes the narrower route — sidestepping the absolute
path for the two affected clocks — because it keeps the change
small, self-contained, and lets users with broken
Task.sleep-on-FreeBSD ship code today.