Skip to content

Latest commit

 

History

History
294 lines (227 loc) · 16 KB

File metadata and controls

294 lines (227 loc) · 16 KB

Benchmark suite

A performance harness for @dunky.dev/state-machine. It measures the engine's hot paths in isolation and compares the runnable parts against XState and Zag.

Numbers below are from one clean run (Node 24, Apple Silicon). Absolute figures vary by machine, Node version, and thermal state — run it yourself — but the ranking and the scaling shape are what hold.

Want to see it instead of read tables? pnpm benchmark:demo opens a live side-by-side where each engine drives a grid of real per-cell machines under a ramping load — watch which panels' backlogs grow as the load climbs. See demo/.

Running

pnpm benchmark          # from the repo root (delegates here)
# or
cd benchmark && pnpm benchmark

Either runs the whole suite in one Node process with --expose-gc (the memory bench needs accurate GC; the flag is harmless for the rest). Output is a series of console.tables grouped by section.

Each section is also exported as a run*() function, so you can run one in isolation:

node --expose-gc --import tsx -e "import('./tests/memory').then(m => m.runMemory())"

Who's compared, and why not always all three

The engines don't all fit every test, because they don't all run the same way:

Engine How send works Where it's measured
Dunky synchronous everywhere
XState synchronous the ops/sec loops, construction, memory, rendering
Zag async (microtask-batched) construction, memory, and React rendering only

A tinybench ops/sec loop counts synchronous iterations, so it can only compare synchronous engines fairly. Zag's headless send is microtask-batched, so it would either deadlock or report meaningless numbers in a tight sync loop — it appears only where it runs synchronously (construction, memory via the headless VanillaMachine) and in the React arena, where it runs natively via @zag-js/react.

Two XState variants appear in the fine-grain tables:

  • xstateactor.subscribe with a hand-written value diff in the listener (the same dedup Dunky does for free).
  • xstate-raw — stock actor.subscribe, which fires on every snapshot change with no diff.

Showing both separates "what XState costs out of the box" from "what it costs once you add a differ."

A key fairness rule across construction + memory: all engines share one module-level config across instances — the shape a real app has (a component's machine config is a const; every instance reuses it). So those loops time machine construction, not config-literal allocation.

How to read the ops/sec tables. ops/sec (higher is better) is the headline; ±rme % is the run-to-run noise floor — a gap between two rows is only real if it clears both rows' margins. The (anti-DCE SINK: …) lines in the raw output are just proof the JIT didn't dead-code-eliminate the work — ignore them.

Every comparison table lists all three engines, so you always see the full field. Where one can't run a given test, the cell is marked — same marker everywhere:

  • n/a ᵃasync. Zag's send is microtask-batched, so it can't run in a synchronous ops/sec or flushSync loop.
  • n/a ᶠno equivalent feature. The engine has no first-class primitive for this scenario (e.g. XState has no lazy/memoized computed), so there's nothing comparable to time.

1. Fan-out / fine-grain / throughput (tests/fan-out.ts)

The selection layer at scale — the thing that decides whether thousands of machines stay cheap. Zag can't run these (async send).

A. Propagation — change 1 of N. ONE machine, N fields, N observers (one per field). Bump one field. Dunky's select is a coarse bus: every selection re-evaluates its selector on each notify and value-compares, so only the touched field's listener fires (downstream is O(changed)) — but the re-eval pass itself is O(N observers) per write. The table shows how that degrades with N versus XState's coarse actor.subscribe.

Change 1 of N Dunky (ops/sec) XState (ops/sec) Zag
100 325 K 253 K n/a ᵃ
1000 10.7 K 10.7 K n/a ᵃ
5000 7.9 K 741 n/a ᵃ

→ Roughly par at small N, but Dunky ~10× faster at 5000 observers — XState's coarse subscribe degrades much faster as the observer set grows.

B. Fine-grain — change an UNOBSERVED field. Change a field nobody selects. The dedup layer re-evaluates and value-compares, so no listener fires — the subscriber-side cost is ~zero. This is the "irrelevant write" that a cell-per-field model gets for free and a coarse bus has to work to ignore.

Irrelevant write, N cells Dunky (ops/sec) XState (ops/sec) Zag
1000 4.5 M 536 K n/a ᵃ
5000 1.9 M 453 K n/a ᵃ

→ Dunky is ~8× faster at shrugging off a write nobody is watching (1000 cells); the value-deduping bus skips waking observers entirely.

C. Throughput — single machine, one event. Per-transition cost with no selection scaling — the raw send price.

Single machine, one event ops/sec
Dunky 7.2 M
XState (raw) 898 K
XState (diffed) 897 K
Zag n/a ᵃ

→ Dunky pushes ~8× the events/sec of XState. Context is mutated in place, so a transition allocates nothing; XState builds a fresh snapshot per event.

2. Compose / synced machines (tests/compose.ts)

The cross-region machinery compose adds, scaled by member count. No competitor column: neither XState nor Zag has a first-class orthogonal-region primitive that maps to combine/sync, so there's nothing equivalent to time (n/a ᶠ for both).

A. combine — one value-deduped Selection derived across M members but reading only m0. It re-evaluates on any member change but fires its listener only when m0 changes. B. sync — a coarse cross-region rule that wakes on any member change (the O(members) path by design).

Members Dunky combine (ops/sec) Dunky sync (ops/sec) XState Zag
2 6.7 M 7.1 M n/a ᶠ n/a ᶠ
10 6.6 M 6.5 M n/a ᶠ n/a ᶠ
50 5.8 M 6.2 M n/a ᶠ n/a ᶠ

→ Cross-region coordination stays in the ~5.8–7.1 M ops/sec band even at 50 synced members — the O(M) re-eval pass costs ~13% going from 2 to 50.

A third "chain" sub-test (a sync rule that send()s downstream every change) was removed: under a tight loop it shows superlinear slowdown. That's a real compose.sync + cross-machine-send interaction worth investigating on its own, not a benchmark-tuning artifact — see the note in tests/compose.ts.

3. Computed (tests/computed.ts)

The most machinery-heavy subsystem — read-key tracking via proxies, memoization against a dep snapshot, glitch-free computed→computed chains. This is a subsystem profile: XState has no first-class lazy/memoized computed (n/a ᶠ), and Zag's send is async (n/a ᵃ), so neither has a comparable primitive to time.

Scenario Dunky (ops/sec) XState Zag
Cached read (no change) 16.6 M n/a ᶠ n/a ᵃ
Fine-grain (change unread, re-read) 6.2 M n/a ᶠ n/a ᵃ
Recompute (change read field) 2.1 M n/a ᶠ n/a ᵃ
4-deep chain (change root, read tip) 567 K n/a ᶠ n/a ᵃ

→ A cached read is ~16.6 M/sec (near-free memo hit), and changing a field the computed doesn't read stays a memo hit at ~6.2 M/sec — read-key tracking means you only pay the recompute when an input you actually read changes.

4. Engine hot paths (tests/engine.ts)

The parts of send that do real statechart work (everything else in the suite stays in one state and only mutates context). These probe Dunky internals in isolation — there's no comparable isolated path to time in XState (n/a ᶠ), and Zag's send is async (n/a ᵃ).

Scenario Dunky (ops/sec) XState Zag
Guard fallthrough — 2 candidates 3.4 M n/a ᶠ n/a ᵃ
Guard fallthrough — 8 candidates 2.9 M n/a ᶠ n/a ᵃ
Guard fallthrough — 32 candidates 2.0 M n/a ᶠ n/a ᵃ
State churn — exit+entry every event 5.6 M n/a ᶠ n/a ᵃ
Effect churn — boot+cleanup each trans 5.5 M n/a ᶠ n/a ᵃ
Sub churn — stable set 7.0 M n/a ᶠ n/a ᵃ
Sub churn — churning set (rebuild) 4.8 M n/a ᶠ n/a ᵃ

→ Even the heavy paths hold ~2–7 M ops/sec: a 32-candidate guard walk, full state transitions with entry/exit actions, and effect boot/cleanup every transition all stay in the same order of magnitude as a bare send.

5. Construction cost (tests/construct.ts)

Wall-clock to build + start() N machines, no events sent (matches a real mount). Synchronous for all three, so it's a fair three-way table. Median of 5 passes, JIT warmed first.

Build + start Dunky (µs/machine) XState Zag
10 000 2.42 1.95 8.16

→ Construction is the one axis where Dunky doesn't win — XState spins up ~1.2× faster. Dunky's bet is flat memory + hot-path throughput, not spin-up; it's still ~3.4× faster than Zag's per-field reactive cells.

6. Memory per machine (tests/memory.ts)

Build 5000 machines, hold them live, report retained heap per machine (heapMB() double-GCs before sampling). Two context widths — thin (2 fields) and fat (64 fields) — because the whole point of the plain-object model is that memory stays ~flat in field count. Rows below are the written mode (one hit each — the footprint a churny app actually pays).

Context Dunky (KB/machine) XState Zag
2-field 3.60 3.62 9.06
64-field 4.10 4.10 134

→ Going 2 → 64 fields costs Dunky only ~0.5 KB/machine — memory grows with the data you store, not with a per-field cell. Zag is the contrast: one reactive cell per field balloons the 64-field context to ~134 KB/machine — ~33× more than Dunky.

Idle vs written. Dunky owns its context copy from construction and mutates it in place forever, so its idle and written footprints match by design — while a lazy-copy scheme steps up once writes start:

64-field, 5000 machines Dunky XState Zag
Idle (never written) 4.10 3.55 130
Written (1 event each) 4.10 4.10 134

→ Dunky idle ≡ written; XState's first assign allocates a per-actor context, so its written row grows.

7. React rendering (tests/rendering/)

The thing that actually hurts in an app — how many React components render — under jsdom. A list of N rows, 50 highlight moves. Two numbers: rows woken / move (the fine-grained payoff) and wall-clock per phase. Strategies: selector (shared machine + useSelector per row), naive (whole-snapshot — the anti-pattern), core/instance (one machine per row), plus xstate/selector and zag/instance.

List of 1000 rows:

Strategy Rows woken / move Mount (ms) Re-render wall (ms)
Dunky/instance 2 5.6 3.9
Dunky/selector 2 8.4 5.9
xstate/selector 2 5.7 6.8
zag/instance 2 6.2 n/a ᵃ
naive (anti-pattern) 980 7.1 56.2

→ Every properly-set-up engine wakes only the 2 rows that changed (vs. the naive whole-snapshot subscription, which re-renders all 980 — a ~490× gap and ~14× the wall time). Among the surgical strategies Dunky re-renders ~1.7× faster than XState.

Zag mounts and wakes the same 2 rows, but its re-render wall is n/a ᵃ — the microtask-batched send can't be timed under a synchronous flushSync loop, so only its row-count is comparable.

Where this actually matters

The loads here (and in the demo) are deliberately extreme to force the engines to diverge. That only reflects real software when a single view holds thousands of independently-stateful, live-updating cells in one frame budget. That's not a hypothetical — these are whole product categories:

Real workload Example products Where the count comes from Live cells
Full L2 order book / DOM ladder Bookmap, Sierra Chart, ATAS 500–2,000 price levels per book × several books, each level a live cell 3k–15k
Options chain / vol surface thinkorswim, Tastytrade, OptionStrat hundreds of strikes × calls+puts × 8–16 live fields (bid/ask/IV/Δ/Γ/Θ/V) 5k–20k
Screeners / heatmaps TradingView screener, CoinMarketCap, Finviz 1,000–5,000 symbols × a few live fields (price, %chg, flash, sparkline) 3k–15k
Live-data spreadsheets Excel/Sheets + market add-ins, Bloomberg BQNT a visible sheet is 5k–20k cells, many bound to streaming feeds + recompute 5k–20k
Observability walls Grafana, Datadog, Netdata hundreds of panels × series, or a per-second host grid 5k–50k
Live log / trace tails Datadog Live Tail, Kibana, Sentry streaming rows, each a tiny stateful unit, into a long virtualized buffer 5k–50k
Dense collaborative canvases Figma, tldraw 5k–50k nodes; a pointermove / cursor stream fans out to all visible shapes 5k–50k
NOC / k8s / network walls k9s, Lens, traffic grids thousands of pods/nodes/flows, each a live cell 5k–20k

The sharpest target is a dense live financial surface (full order book or options chain): genuinely 5k–20k live cells, updating tens of thousands of times a second, often needed on web and native from one codebase, where flat per-cell memory matters — every property the benchmark measures pays at once.

What does not need this: a handful-of-symbols chart view, a typical SaaS dashboard, a chat app — dozens to a few hundred live elements. The engine works there too, but so does anything; at that scale you'd pick it for the agnostic-render and bundle/memory reasons, not the throughput these tables show.