Repository navigation
feat(taint): allocator-backed heap taint bits - #52
Draft
RomainMuller wants to merge 23 commits into
Draft
RomainMuller wants to merge 23 commits into
RomainMuller wants to merge 23 commits into
Conversation
Plan and proof of concept for a taint bitmap with one bit for each heap byte, stored next to the Go allocator metadata. Orchestrion aspects add the bitmap to `runtime.heapArena`, add a flag to `runtime.mspan`, and clear the bits of dead objects in `(*sweepLocked).sweep` and `freegc`. With this, the check "does this value have taint?" needs no lock and works for sub-strings. The proof of concept is in `_docs/plans/allocator-taint-bits-poc/`. Its tests pass on go1.26.6, go1.26.8 and go1.27.1 (default flags, -race, -N -l, checklinkname, and 3 GOEXPERIMENT variants), and also on linux/amd64 and linux/arm64. Its benchmarks show no measurable overhead when no value is tainted. The phase 1 plan went through 4 critic rounds and the user approved it. It also records the decisions for the next phases: keep tainted data on the heap (stack values), runtime specials (origins and marks), and `main` as the base branch.
Add `internal/taint/heapbits`: one taint bit for each heap byte, stored next to the Go allocator metadata. Orchestrion aspects on package `runtime` add a lazily allocated bitmap to `heapArena`, a flag to `mspan`, and hooks in `(*sweepLocked).sweep` and `freegc` that clear the bits of dead objects, so that new objects at the same address do not get old taint. The root `orchestrion.tool.go` imports the package. This is step 1 of the plan in `_docs/plans/allocator-taint-bits.md`: the proof of concept moves into the repository. The review of this step also fixed these problems: - `Set` reported success when the feature was off or when no bitmap could be allocated. It now allocates all bitmaps first and returns false with no bit changed. - `Set` and `Clear` now refuse a range that crosses into a neighbour allocation slot. Inside a 16-byte tiny slot, the runtime cannot see the bounds of each value, so the caller must give the range of one value (documented; the string and slice helpers always do). - The `freegc` hook clears the whole slot, and runs only when `freegc` can free the object. - The test knobs (`nosweep`, `freegc`) do nothing unless the program links `heapbitstest`, so production binaries cannot use them. - The feature is on only on linux and darwin on amd64 and arm64, without ASan or MSan. Other targets compile the woven runtime with the feature off. The tests now prove that addresses are really used again, and a negative control (sweep hook off) must show stale taint for each size. Tests pass on go1.26.6 and go1.27.1, with -race, -N -l, checklinkname, GOEXPERIMENT=runtimefreegc and nogreenteagc, in linux/arm64 and linux/amd64 containers, and on js/wasm (feature off). The woven runtime compiles for 20 targets.
The Go wrappers pass `uintptr` values to the runtime, so that checked values do not escape to the heap. A `uintptr` is not updated when the goroutine stack moves: a stack address could become stale between the conversion and the check, and then point to unrelated memory. Each runtime entry point (`set`, `clear`, `any`) is now `nosplit` up to the "is this heap memory?" decision, with a nosplit arena lookup. A stack address is refused before the stack can move. Then a `noinline` worker does the long work. Workers count work units (bitmap words and arenas) and run a checkpoint every 512 units that calls `Gosched` when the scheduler asks for it, because runtime code is not an async safe point. `Any` has a nosplit fast path for up to 16 bitmap words; its slow path stops at the end of the span. Test knobs (active only when the program links `heapbitstest`): a stack probe at classification, a worker probe, a forced stack move before the classifier (negative control), forced yields, and a forced slow path. New tests: - the stack probe test and its negative control, meaningful with `-gcflags=all=-d=maymorestack=runtime.mayMoreStackMove`; - `TestNoWorkerForStackAddress`: no worker runs for a long stack range; - `TestGeneratedCode`: rebuilds the woven runtime with `-S` (default and `-N -l`) and checks the nosplit and call rules; - checkpoint, fast-path boundary and fast-path/worker equivalence tests, and micro benchmarks. Plan: step 2 of `_docs/plans/allocator-taint-bits.md`.
The proof of concept used one flat 8 MiB bitmap for each heap arena, with no limit: clearing could touch every page, the memory was never charged to the runtime memory limit, and racing first writers could each allocate a bitmap. The bits now live in 16 KiB chunks (128 KiB of heap each). Each arena gets a directory (one chunk) on first use. Chunks come from 1 MiB slabs that the runtime maps with the non-fatal direct mmap. A global budget of mapped bytes (default 64 MiB, maximum 1 GiB, `heapbits.SetBudget` before the first Set) is reserved before each mmap. One goroutine at a time maps a slab; others drop and never wait. Slots are claimed before the allocation and given back on every failure. Set is all or nothing. Clear and the sweep hook skip missing chunks and write only words that have bits to clear. A successful mapping is charged to `other_sys` and `mappedReady` (the memory limit sees it); a failure charges nothing. Tests (in child processes with their own budget): exact accounting, mmap failure with chunk and directory slot rollback, a parked refill owner, a claimed slot, scan-then-refill (no second mapping), concurrent refills with a high-water mark, the budget limit with all-or-nothing in one arena and across arenas, and the default and zero budgets. Plan: step 3 of `_docs/plans/allocator-taint-bits.md`.
Chunks were never given back: a large tainted object that died kept its chunks, and the sweep hook cleared all their words, which bounded neither the memory in use nor the sweep work. The sweep hook now gives back each chunk that is fully inside a dead object: it removes the chunk from the directory, then sets its dirty bit and its free bit on the slab descriptor (two Or64, no loop). The goroutine that takes a dirty chunk zeroes it (non-zero words only, with checkpoints) before it publishes it. A chunk at the edge of a dead object is only cleared, because a live neighbour can use it. Set now refuses a range in a span larger than 64 MiB (a Span drop). This bounds the sweep work for one span to 513 chunk slots and two partial chunks. The feature is now off on arm64 CPUs without LSE atomics (the plan required it): there, Or64 and And64 are LL/SC loops with no progress bound, and the sweep hook cannot yield. Tests: recycling and zeroing, a live neighbour in an edge chunk, the span limit (and the recycling of a dead 64 MiB object), and a stress test where large tainted objects die and their addresses are used again while the GC runs. Plan: step 4 of `_docs/plans/allocator-taint-bits.md`.
…work The flag protocol of the sweep hook (flag = 0, clear the dead objects, check the live ones, flag = 1 again if one has taint) needs ordered accesses. The bitmap words used plain loads and stores, and the sweep prologue read the span flag with a plain load. - All bitmap word accesses are atomic (Load64, Store64, Or64, And64), also the zeroing of recycled chunks; the sweep prologue uses Load8. On arm64 (LDAR), Any on 4 KiB is 20-30% slower; 16 B and 256 B do not change. - The live check of the sweep hook has a work limit of 1024 units (chunks and words). At the limit, the flag stays 1, which is always safe. - The finalizer check walks the sorted specials list once for each pass (a cursor), not from the head for each object. - TestGeneratedCode also checks that no injected runtime function calls a panic or throw function when built with optimizations. Tests: a dead neighbour in the same bitmap word while the live one gets Set calls during GC; forced interleavings of Set with the sweep hook of one span (after the flag reset, and after the live check; a mutation that writes the flag last fails); Set during continuous GC; the bounded flag reset (many live objects: the flag stays 1; few: it goes to 0). Plan: step 5 of `_docs/plans/allocator-taint-bits.md`.
Record the user decision on the cost of atomic loads (step 5): all bitmap accesses stay atomic. The cost is only for long values on arm64, and the step 10 benchmarks check it again.
The propagation hooks of phase 4 need to read the tainted ranges of a value and to copy bits from inputs to results. - Next and NextClean find the first tainted or clean byte from an offset. Together they give the tainted ranges of a value without allocation. Memory that is not heap memory is never tainted, and the bytes after the end of the span of the value count as clean. - Copy makes the bits of a destination equal to the bits of a source, with memmove semantics (overlap is correct). It classifies both pointers before any stack move. A source that is not in one heap span counts as clean. A clean source makes Copy a clear, which needs no storage. Else all destination chunks are obtained first (all or nothing); the destination words are written in the memmove order, full words with Store64 and partial words with a Cas64 loop, then the span flag is set. Copy has the same span limit as Set. Tests: model-based Next/NextClean and Copy (all 64x64 alignments, both overlap directions, across chunks for source and destination), edge cases, a Copy-only destination that dies and is reused, a concurrent neighbour writer, all or nothing with a failing mmap, and a progress test with asynchronous preemption off (longest wait to stop the world: 0.2 ms, limit 10 ms). The stack probe, worker probe, checkpoint and generated-code tests cover the new entry points. Plan: step 6 of `_docs/plans/allocator-taint-bits.md`.
…aint bits - TestRandomModel runs 10000 random operations (allocate objects of 1 B to 1 MiB, Set, Clear, Copy with and without overlap, drop, GC, and checks with Any and Next/NextClean) against a model of the expected bits. Every new object must be clean. The seed and the number of operations can be set with HEAPBITS_MODEL_SEED and HEAPBITS_MODEL_OPS. It finds these mutations: no sweep hook, a forward-only Copy, and a clear that writes a whole partial word. - New negative controls, with test knobs that only work in programs that link heapbitstest: the freegc hook off (1000 of 1000 reuses keep the taint with GOEXPERIMENT=runtimefreegc), and the finalizer check off (100 of 100 revived objects lose their taint). In a runtimefreegc build, the freegc tests fail (not skip) if freegc frees nothing. - The Go functions that call the runtime are go:noinline: their stack check is a synchronous preemption point, because the runtime entry points are nosplit. TestGeneratedCode checks that the stack check is present. - TestCopyProgress failed on a loaded machine, where the OS also delayed plain Go code by tens of milliseconds. It now alternates rounds of a plain Go baseline and of the Copy work, keeps the absolute 10 ms limit for the rounds with a fast baseline, and skips when fewer than 3 of 5 rounds are conclusive. Plan: step 7 of `_docs/plans/allocator-taint-bits.md`.
The runtime aspects depend on unexported runtime internals. Compile-time assertions catch a signature or type change, but not a change of behavior, such as a new path that frees heap memory without one of the hooks. TestRuntimeDrift (plain go test, no weaving) reads the runtime source of the toolchain in use and writes an inventory: the call sites of the functions that free memory or use it again, the writes of freeindex and allocBits, the special kinds, the mmap definitions, and the full code of the functions whose behavior the hooks depend on (mark bits, span lookup, sysAlloc, the freegc reuse path, the whole sweep, the order of the specials list, mmap). It compares the inventory with a golden file for the Go minor version, and shows an ordered diff. A new minor version, or a change, fails the test until a person reviews the runtime code and runs it with -update-drift. The golden files for go1.26 and go1.27 were reviewed: no path frees heap memory without one of the hooks. go1.26.6 and go1.26.8 give the same inventory. Plan: step 8 of `_docs/plans/allocator-taint-bits.md`.
The runtime aspects of internal/taint/heapbits use unexported runtime internals, so they must be tested on each supported Go minor version and with the build modes that change the runtime or the generated code. - heapbits: oldstable and stable, on linux/amd64 and linux/arm64, with default flags (and -checklinkname=1), -race, -N -l, maymorestack, GOEXPERIMENT=nogreenteagc, GOEXPERIMENT=runtimefreegc and CGO_ENABLED=0, plus darwin/arm64 (29 cells). GOTOOLCHAIN=local, so that a cell tests its own Go version. - heapbits-feature-off: woven cross builds for 10 targets where the feature is off, js/wasm tests under Node.js, and -asan tests. - heapbits-prerelease (gotip, not required): the drift test without weaving, then the woven tests. A failure gives a warning and a line in the job summary. The first two jobs are required in green-ci. The drift test now reads the Go version with go env GOVERSION when there is no VERSION file (a gotip build), and names development versions go1.N-devel. Plan: step 9 of `_docs/plans/allocator-taint-bits.md`.
Add the benchmarks and the latency gate of Phase 1 step 10 (plan _docs/plans/allocator-taint-bits.md, sections 5.5, 7.1 and 7.2): - microbench (no test knobs linked): Any, Set, Next, Copy, fresh chunk, first-chunk contention, and the map baseline for comparison; - budgetbench (own binary, 4 MiB budget): Set with a full budget and quota saturation with recovery; - worst-case sweep benchmarks and allocation latency in package heapbits, with Set cost and GC cost measured separately; - TestSweepHookLatency: 8 worst-case scenarios, each in a child process, p99 of the sweep hook at most 20 µs; - overhead runner: a third 'active' variant, HeapBits workloads, a three-column comparison and gate.txt (enforced with -gate only; the CI job shows it as information, shared runners are too noisy). The latency gate found that 2 sweepers that recycle the chunks of large dead objects contended on the slab words and the counter (22-27 µs per hook). The recycle now batches the updates for each run of chunks of the same slab, and adds to the counter once per call (2-6 µs).
Fix the findings of the Phase 1 full review (six reviewers, two follow-up rounds): - a Set or Copy that failed after it got a whole chunk inside the object did not set the span flag, so the chunk was never recycled; a live large object with a whole chunk inside now keeps the flag without taint, for the same reason; - the live check of the sweep hook reads each run of live objects in one walk (one chunk lookup for each chunk, not for each object): a span of small live clean objects took 12-38 us in a Linux VM, now about 1 us; - the sweep hook stops at once for a span larger than 64 MiB, which can have a stale flag on a reused mspan; - the overhead runner refuses -gate with fewer than 4 samples, and its variant check uses the arm64 LSE condition of the runtime; - more tests (failed Set and full budget recycle, Copy partial failure, large slots with one live slot) and more robust ones (Copy neighbour scheduling, freegc address reuse, negative control reuse); - CI cells for cgo off with runtimefreegc, and macOS race and -N -l.
This comment has been minimized.
This comment has been minimized.
The new recycle case of the full review failed in CI (go1.26, -N -l): the Set of the last free chunk needed a directory too (a new arena), and a GC during the fill gave chunks of dead objects back. fillBudget now uses regions whose arena has a directory when a Set of a fresh region fails, and fills again after a GC until the budget stays full.
Many heap taint bit tests run in child processes of the test binary (storage, recycle, latency, negative controls). Their coverage was lost, so the code that only the children run counted as not covered. The children now get the coverage directory of the parent (-test.gocoverdir), and go test adds their counters to the profile.
The proof of concept has its own go.mod (module example.com/allocbits, and github.com/puzpuzpuz/xsync/v3 for its baseline). The license check reads all go.mod files, so LICENSE-3rdparty.csv was out of date. Phase 1 replaces the proof of concept; it stays in the history of the plan commit.
The woven test run cannot reach the code for a runtime without the aspects (the 'feature off' paths of internal/taint/heapbits and of its test knobs). The unit test job now also runs the tests without Orchestrion and adds the counts of both profiles. New tests cover the knobs without the woven runtime and a budget above MaxBudget.
On GitHub runners, the sweep hook of the 64 MiB scenarios took 24 to 29 us (99th percentile), against 3 to 9 us on a developer machine: the runners are slower and shared. The limit of the plan (20 us) stays for developer machines; with CI set, the limit is 50 us, so that the gate still finds large regressions.
fillBudgetButOne failed in CI (linux/amd64, nogreenteagc): with 2 free chunks, a Set in a new arena took both (the directory and the bits), so the state with exactly one free chunk never came. With 2 free chunks, it now takes only regions whose arena has a directory.
The previous commit also held changes to go.mod, go.sum and orchestrion.tool.go: an old Orchestrion binary, used for a local cross build, added dd-trace-go/orchestrion/all/v2 and its dependencies. This restores the three files.
The pre-release CI job failed because TestRuntimeDrift had no golden for the Go development version. I read the changes against go1.27: riscv64 is added to the build tags of the cgo mmap (not a supported platform of the feature), ensureSwept has a fast path for spans that are already swept, and specialFindSplicePoint is rewritten with the same order of the specials (offset, then kind). None of them changes what the aspects depend on.
The previous fix skipped fresh regions in arenas without a directory when 2 chunks were free. In CI (cgo off), the fresh regions went to new arenas every time, and the loop ended before it reached one free chunk. fillBudgetButOne now makes 2 spare regions in an arena with a directory before the budget is used, and uses one of them in this state.
Add a README section on the internal heapbits storage: how the woven runtime keeps one taint bit per heap byte, and how the bits relate to arenas, spans, chunks and slabs. Diagrams show the address-to-bit mapping, the memory budget and the life cycle of a bit through GC sweep.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds allocator-backed heap taint bits: one taint bit for each heap byte, stored next to the allocator metadata. With these bits, the question "does this value have taint?" is a few loads, without a map lookup.
Why
The taint tracking needs a fast check on hot paths (
Anyis called on many values that never have taint). Maps keyed by address are slow under contention (about 85 ns for each parallel lookup) and must be cleaned when objects die. When the bits are in the runtime, the GC clears them for dead objects in the sweep, and the check stays at 3–5 ns (0.3 ns in parallel).What
internal/taint/heapbits/orchestrion.yml: Orchestrion aspects that add code to the Goruntimepackage:heapbits.SetBudget). When the budget is used, aSetdrops the taint. It never waits and never fails the host program;mspan: the sweep hook runs only for spans that got taint. It clears the bits of dead objects, recycles the whole chunks of dead large objects, and keeps the work of each span bounded (about 20 µs or less);freegc(GOEXPERIMENT=runtimefreegc), and correct handling of objects that finalizers revive;nosplitentry points that refuse stack, global, user-arena and non-Go memory, andnoinlineworkers with preemption checkpoints.internal/taint/heapbits/heapbits.go: the Go API (Enabled,Set*,Clear*,Any*,Next,NextClean,Copy,SetBudget). The feature is on for linux and darwin on amd64 and arm64 (on arm64, LSE atomics are necessary), and not with asan/msan. On other targets, the calls do nothing.Copy/Next, a random model test, negative controls, generated-code checks (nosplit, no panics), a sweep-hook latency gate, andTestRuntimeDriftwith one golden inventory for each Go minor version. The inventory detects changes in the runtime internals that the aspects use.microbench,budgetbench, and worst cases inheapbits), and a thirdactivevariant inbenchmarks/overheadwithHeapBits*workloads andgate.txt. The gates fail the run only with-gate; in CI they are only shown in the summary, because shared runners give too much noise.heapbits(Gooldstable/stable× amd64/arm64 × build modes, and macOS),heapbits-feature-off(cross builds, js/wasm,-asan), and a non-requiredheapbits-prereleasejob (gotip).Notes for review
_docs/plans/allocator-taint-bits.md; it will be removed before merge. The proof of concept is in the history of the first commit only (it had its owngo.mod, which the license check reads).TestRuntimeDriftand the pre-release job are there to find breakage early.-exportfdcompiler flag; v1.13.1 knows it, but itsx/toolscannot read the export data version 5 of gotip). Its drift step passes (golden forgo1.28-devel).