Repository navigation
Input wanted before I measure: is the retained allocator slack fragmentation, or one allocation pattern in aft? #205
Description
Follow-on from #201, where the shipped malloc_trim(0) relief turned out not to reclaim anything on a long-lived Linux daemon. The free space is pinned beneath live allocations, so trim structurally can't return it.
The open question is which of two things that is:
| mechanism | remedy | whose code |
|---|---|---|
| ordinary heap fragmentation | allocator with decay-based purging | aft's #[global_allocator] |
| one pathological allocation pattern | fix the site | aft, specific |
mallinfo2 gives aggregates only. The per-arena and per-bin free-space distribution separates the two, and that's malloc_info(3) — which the crate doesn't bind today.
I've designed an experiment and I'd rather ask before building it, because you may already know the answer or want it shaped differently. Specifically:
-
Do you have prior context on the allocation pattern? Long-lived per-root artifacts (search index, semantic index, callgraph store, symbol cache) are each individually capped but have no aggregate ceiling across roots — the daemon here held 7+. Many large, long-lived, differently-sized buffers arriving and departing on independent schedules is a textbook way to strand free space. If that's a known shape to you, the experiment may be unnecessary.
-
Has a non-default allocator been considered and rejected? jemalloc/mimalloc with decay purging is the obvious remedy for fragmentation-pinned space, and it's a
#[global_allocator]change with real consequences — binary size, thepanic = "unwind"interaction with ONNX Runtime, platform matrix. If there's a standing reason not to, I'd like to know before I spend a run producing evidence for it. -
Would you shape the measurement differently? My rig is
subc_storm_test.rs— it already drives binds, tool traffic, watcher churn, and semantic collection across N roots viaAFT_STORM_SCALE, and it spawns its own ephemeral daemon so nothing production-facing is touched. One known weakness: it runs the module in-process, somalloc_infomeasures the whole test binary. I'm handling that with start/end deltas plus a zero-workload control arm to measure the scaffolding's own contribution, with a dedicated bin target as fallback if that control shows the scaffolding isn't constant.
Design in brief — four arms (scaffolding-only, stock, MALLOC_ARENA_MAX=2 as a negative control on the rig, decay-purging allocator), malloc_info + mallinfo2 + smaps census at start and end, predictions registered before running so the outcome can't be interpreted after the fact. Full design doc available if useful; happy to paste it here or open it as a PR against a docs path if you'd prefer it in-repo.
What I'm not doing: proposing an allocator change. That would need this evidence first, its own gate cycle, and your agreement that it's a direction worth taking at all.
Context on severity, so this is priced correctly: after #202 landed (inspect threads 88 → 8 on the same daemon) the live instance sits at ~2.9 GB swap after 12 h, against 15.3 GB at 18 h before. It's churn-bound rather than time-linear — regions and VmData go flat across idle hours. So this isn't urgent, and I'd rather spend the effort in the direction you'd actually use.
Activity
ualtinok commentedon Aug 10, 2026
Answers to all three, in order — and the short version is: build it, with one arm-sharpening addition.
1. Prior context on the allocation pattern: yes, and it narrows your experiment rather than obviating it. The long-lived per-root artifacts you named are the known shape: each root independently owns a search-index snapshot (postings blob + delta), semantic embeddings, callgraph store buffers, and symbol caches, and they arrive/depart on independent schedules (bind, watcher churn, idle eviction, reconfigure) with sizes spanning KBs to hundreds of MB. Interleaved lifetimes of differently-sized long-lived buffers is exactly the textbook stranding pattern — so "one pathological site" is unlikely to be the answer; "a structural pattern spread across per-root artifact lifecycles" is the prior. Your per-arena/per-bin distribution measurement still decides it, because the prior doesn't tell us whether the stranding concentrates (fixable at a site) or smears (allocator-level remedy).
2. Non-default allocator: not rejected on principle — never seriously proposed, because the standing costs are real and known. Three we'd weigh against any evidence you produce: (a) ONNX Runtime interaction — we load onnxruntime dylibs in-process and the
panic = "unwind"+ foreign-allocator combination across that FFI boundary is exactly where allocator swaps produce the subtle class of bug; (b) platform matrix — the swap lands on macOS/Windows too (or gets cfg-gated, which forks allocator behavior across the platforms we debug on); (c) binary size and supply chain on a shipped-binary product. None of these is a veto; all of them mean the evidence bar is "distribution data showing decay-purging fixes what an arena cap cannot", not "fragmentation exists". Your reordering (arena-cap outcome checked before allocator-swap) is exactly the right sequencing from our side.3. Rig shape: sound, one addition. Storm rig + zero-workload scaffolding control + start/end deltas is right. Add to arms A and D a
malloc_info(3)arena-count and per-arena free-distribution capture (not just aggregates) — A-vs-D is your cleanest causal read and the per-arena shape is what separates "many low-occupancy arenas" (arena cap wins) from "few arenas, deep internal fragmentation" (cap does nothing, decay-purging or site-fix). Also pin thread counts in the report (your 128-arenas-vs-150-threads ceiling observation is the kind of number that should ride along).On the pool-size override (from #201): yes, and we'd take it as your PR — env var, not config field. Exact shape so it lands first-pass:
AFT_INSPECT_POOL_THREADSread once at pool init indispatch.rs::default_pool_size(), parsed asusize, clamped tomax(1, n), capped at 512 as an absurdity guard; absent/unparseable → current derivation unchanged.AFT_STORM_SCALEandAFT_SEMANTIC_QUIET_WINDOW_MS.LazyLock, so the test needs the env set before first pool touch — same pattern the storm-scale tests use).Looking forward to the distribution data.
inspect: allow overriding the shared pool thread count
iceteaSA commentedon Aug 10, 2026
Ran it. The per-arena data points at a bounded arena count — but it contradicts a production measurement I reported to you two days ago, and that contradiction needs to come first because you'd reach it in one step anyway.
The contradiction
On #201 I told you
MALLOC_ARENA_MAX=2was measured ineffective in production: 0.95 GB/h swap accumulation against a 0.85 GB/h uncapped baseline, wrong direction. We removed the cap on that basis.The rig says the same cap is decisive: 65% less retained free space, and the low-occupancy arena population goes to zero.
I cannot adjudicate that from the data I have. Three candidate reconciliations, none confirmed:
AppContexts, real binds, real watcher churn — but 6 roots × 2 sessions for 90 s is not a 41-hour daemon serving real work.8bd35ff4binary at 211 threads; the rig runs the post-fix regime.Explanation 3 was my working assumption until the A-vs-D result, which weakens it — see below.
Numbers
Storm workload, 6 roots × 2 sessions, 90 s, release, each arm in its own
systemd-run --scope -p MemoryMax=14G -p MemorySwapMax=0. Four runs per arm.MALLOC_ARENA_MAX=2AFT_INSPECT_POOL_THREADS=88Arm A's end-state distribution, one representative run:
Fifteen of sixteen arenas sit at ≥0.86 free. One is doing the work.
The finding that complicates your mechanism
Arena count saturates at 16 regardless of thread pressure. Arm D ran 2.4× the peak threads of arm A (143 vs 60) and produced an identical arena count and no material difference in retained free (27.8 vs 28.6 MB).
That cuts against the arena-per-contending-thread story as the whole explanation. If the count saturates well below thread count, then
8bd35ff4's 88→8 inspect-thread cut did not help primarily by reducing arenas. It also weakens reconciliation 3 above: production-B had 2 arenas too, and still accumulated swap faster than baseline.16 equals this host's core count. I have not established that as causal and am not going to guess at it — flagging it as unexplained rather than smoothing it.
The ask, and the part I'd rather you decide
mallopt(M_ARENA_MAX, N)at daemon startup, if you still want that conversation.The value is genuinely open and I don't think 2 is right. 2 was the maximum-signal probe number, not a production recommendation. Arm A's data says 16-with-10-idle, which argues for well below 16 — but 4, 8, or
cores/2are all defensible and I have no data separating them.On contention: 2 arenas under 60+ concurrent threads is exactly the configuration that trades lock contention for footprint, and latency was a non-goal for the experiment but cannot be a non-goal for the remedy. One weak data point in the cap's favour: arm B passes the storm harness's own bind-latency assertion (
max bind ack ≤ 2 sacross all binds) at 62 peak threads. That is a bound, not a measurement — it says the cap didn't blow the existing budget, not that p50 was unaffected.If you want a latency arm before deciding, say so and I'll add one rather than ship a footprint win that costs tail latency.
What this does not license
Anything about decay-purging allocators — arm C wasn't run. Its registered trigger required the arena cap being ruled out first, and the cap works, so building a second binary would have been doing the expensive work to answer a question this run closed. Also nothing about which of the three changes was operative in production, and no extrapolation of these absolute numbers (tens of MB) to the 29.5 GB the production daemon reached.
One registered stop clause misfired and I'm retiring it in the open rather than quietly: the design said "rig does not reproduce if arm B comes out materially better than production measured it." It did. But that clause depended on arm B being a clean control, and your arena-per-thread correction had already downgraded B to a rough sanity check — the same correction invalidated the clause, and I failed to update it at the time. The governing half doesn't fire: arm A grew retained free 0.24 → 28.6 MB and arenas 2 → 16, so the rig reproduces.
Full writeup with method and threats to validity available if useful.
iceteaSA commentedon Aug 10, 2026
Chased the unexplained ceiling from my last comment rather than leaving it flagged. It's explained, and it changes how the A-vs-D null should be read.
16 is glibc's default arena limit on this host — reproducible, and independent of how much contention you apply. Standalone C probe, N concurrent allocating threads then
malloc_info, glibc 2.44, 16 cores:Low counts give exactly
main + N, which validates the counting. From 16 up it pins and never moves — not at 256 concurrent allocators.It's a real limit, not a workload artifact. Raising it explicitly moves the ceiling in lockstep:
Two things worth having on the record:
It is not
8 × ncores. The documented default would be 128 here. Observed default is 16, and an explicit 128 reaches 65 — so the mechanism can exceed 16, the default just doesn't. I don't know whether that's a distro build-time choice or something in the 2.44 defaults, and I'm not going to guess at it.CPU affinity doesn't move it.
taskset -c 0-1through0-15all give 16 at 64 concurrent allocators, so it isn't computed from the affinity mask at runtime.Still open: 16 equals this host's core count, and one machine can't separate "16 = ncores" from "16 = a constant". A different-core-count host settles it in one run. It doesn't matter for the remedy — the cap gets set explicitly either way — so I've recorded it rather than chased it further.
What this does to the A-vs-D result I reported: both arms were already pinned at the ceiling, so thread count couldn't have moved arena count in either. The null is what the mechanism predicts once you know the ceiling exists — not evidence against arena multiplication. I framed it as complicating your mechanism story; that was too strong, and this is the correction.
The narrower true statement: the 88→8 inspect-thread cut could only have reduced arena count if it dropped concurrent allocators below 16. Production data doesn't tell us whether it did.
Doesn't change the ask.
mallopt(M_ARENA_MAX, N)still follows from the free-space distribution — 15 of 16 arenas at ≥0.86 free with one arena doing the work — and the value question is still open, with 4/8/cores÷2all defensible and this run separating none of them.ualtinok commentedon Aug 10, 2026
The ceiling find is a genuinely useful datum beyond this issue — 16-not-128 with explicit raises moving in lockstep is the kind of thing that saves the next person a day of confounded arms, and the A-vs-D reframe (both arms ceiling-pinned → null predicted, not contradicting) restores the arena-multiplication mechanism cleanly. Appreciated as always that the correction came from you chasing your own flagged anomaly. No action needed our side; the #209/#210 threads carry the follow-ups that came out of this line of work.
ualtinok commentedon Aug 14, 2026
Closing with the measured answer: the retained slack was fragmentation-pinned free space under glibc arena allocation, not one allocation pattern —
malloc_trim(0)released ~0 bytes against it, and the dominant factor was arena count scaling with thread count, which the process-wide inspect pool (#202) and the thread-count reduction addressed.AFT_INSPECT_POOL_THREADSremains available for regime testing (#207). Thanks for the methodical thread — the PRs it produced (#207, #208) both shipped.