[tamirms]

Description

On master, FastAnd intersects its inputs pairwise, so it allocates an intermediate container for each key and input even when the final result is empty. This PR looks up each key in the inputs before allocating anything for it, and builds a result only once a value is known to survive, however late in the inputs the result empties.

This depends on #566, so the first commit here is #566's. It uses clearBitmapGaps, containerFromWords and the allocation-free run intersects from #566, and GitHub cannot base a pull request on a branch in a fork. The diff shrinks to this PR's own commits once #566 merges.

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Performance improvement
• Code refactoring
• Documentation update
• Test improvements
• Build/CI changes

Changes Made

What was changed?

• FastAnd looks up each key of the input with the fewest keys in the other inputs before allocating anything for that key, with the cursors on the stack. If that input has one key, every input is looked up first, up to eight. Otherwise four are looked up, and the rest are intersected into the result one at a time.
• For an array, the key is also checked for a value it shares with each input, which usually stops at the first value. The check pauses while keys keep surviving every input, since there it only costs time.
• A surviving key is intersected without allocating until a value is known to survive. The smallest array is narrowed in a buffer on the stack. Bitmaps are counted in a scratch bitmap, or ANDed straight into the result when they share a bit in their first eight words. Runs allocate only where they overlap.
• An array against a single interval, the shape of a partial key range, is cut with two binary searches instead of a merge. And gets this too.
• Two inputs on one key take the same path. Two inputs on many keys use And.
• bitmapContainer.andArray is a quarter faster with its slice headers hoisted out of the loop, and ParAnd's worker calls the shared andKChain.
• Added BenchmarkFastAndShapes and BenchmarkRealDataFastAnd.

Why was it changed?

• On master, an empty intersection allocates up to 8 KiB per key and input, and dense inputs do all of the intersection work for an empty answer.
• Stellar RPC calls FastAnd once per filter and key range. On master, a page of 256 filters that never match allocates 318 MiB in 610 ms; with this PR it allocates 3.9 MiB in 244 ms.

How was it changed?

• A walk over many keys looks up four inputs per key and intersects the rest into the result, which keeps calls whose keys all survive as fast as pairwise And.
• The result's slices are allocated once, at its first key, for up to 64 keys.

Testing

• fastand_kway_test.go compares FastAnd with pairwise And on 400 random cases of 3 to 12 inputs that mix arrays, bitmaps and runs, including whole-key ranges. Results must pass Validate, use arrays and bitmaps by the usual size rule, and share no storage with the inputs.
• An empty intersection must allocate only the result. The cases cover arrays of 4,096 values, bitmaps and runs on a whole or partial range, inputs that intersect in pairs but not all together, results that only the fourth or fifth input empties, a pair whose term lies past a partial range, and 1,000 single-value inputs.

go test ./...
go test -tags appengine ./...
GOOS=linux GOARCH=386 go build ./...    # also arm, arm64, ppc64

All pass on linux/amd64 (Xeon 8375C, with AVX-512 VBMI2) and darwin/arm64.

Formatting

gofmt -s -l . reports nothing. go vet ./... reports only the existing ReadFrom signature warning from master.

Fuzzing

The FuzzSmat target in the template does not exist on master, so -fuzz=FuzzSmat matches nothing. The smat corpus runs and passes, but none of its sequences calls FastAnd, so the randomized comparison under Testing is what exercises this change.

Performance Impact

BenchmarkFastAndShapes (added) compares FastAnd with pairwise And, the algorithm on master, on the same inputs and in the same binary. One key unless noted; the first seven rows add a whole-key range, like a query window, as the last input. Xeon 8375C:

	pairwise And	FastAnd	speedup
array30-bitmap	243 ns, 200 B, 6 allocs	245 ns, 188 B, 6 allocs	0.99x
array4000-bitmap	6.7 µs, 8328 B, 6 allocs	6.7 µs, 8316 B, 6 allocs	1.00x
array30-runs	458 ns, 168 B, 3 allocs	272 ns, 80 B, 1 alloc	1.68x
array4000-runs	3.5 µs, 8328 B, 6 allocs	3.5 µs, 8316 B, 6 allocs	0.98x
bitmap-bitmap	2.4 µs, 8336 B, 6 allocs	2.2 µs, 8324 B, 6 allocs	1.11x
runs-bitmap	3.5 µs, 4232 B, 6 allocs	3.1 µs, 4220 B, 6 allocs	1.12x
runs-runs	2.9 µs, 2176 B, 13 allocs	2.9 µs, 2168 B, 13 allocs	1.01x
jointly-empty	9.8 µs, 33200 B, 18 allocs	1.5 µs, 80 B, 1 alloc	6.6x
eight-keys	1.3 µs, 1056 B, 24 allocs	1.2 µs, 936 B, 20 allocs	1.06x
singletons-10000	25.7 µs, 106 B, 3 allocs	9.3 µs, 80 B, 1 alloc	2.8x
equal-x128	49.0 µs, 1056 B, 24 allocs	49.4 µs, 936 B, 20 allocs	0.99x
surviving-1024x256	13.4 ms, 33456 B, 535 allocs	13.6 ms, 32272 B, 522 allocs	0.98x

And shares the new run and array code, so the rows with runs understate the gain over master.

BenchmarkRealDataFastAnd (added), master against this PR. Each dataset's full intersection is empty, so this measures how quickly FastAnd finds that out. Xeon 8375C:

	master	this PR	speedup
census-income	1.3 µs, 336 B	295 ns, 80 B	4.3x
census-income_srt	2.4 µs, 1400 B	321 ns, 80 B	7.4x
census1881	556 ns, 80 B	269 ns, 80 B	2.1x
census1881_srt	560 ns, 80 B	282 ns, 80 B	2.0x
dimension_003	50.8 µs, 80 B	21.7 µs, 80 B	2.3x
dimension_008	14.8 µs, 520 B	6.4 µs, 424 B	2.3x
dimension_033	24.7 µs, 2237 B	278 ns, 80 B	89x
uscensus2000	550 ns, 80 B	279 ns, 80 B	2.0x
weather_sept_85	18.2 µs, 15632 B	424 ns, 80 B	43x
weather_sept_85_srt	2.8 µs, 1024 B	255 ns, 80 B	11x
wikileaks-noquotes	647 ns, 128 B	286 ns, 80 B	2.3x
wikileaks-noquotes_srt	616 ns, 112 B	316 ns, 80 B	1.9x

Performance Analysis

• Empty intersections are 1.7x to 6.6x faster than pairwise And, and every real dataset is 1.9x to 89x faster than master.
• Every dataset but dimension_008 allocates only the result. dimension_008 has keys that only its fifth input or a later one rules out, and it allocates less than on master.
• When every key survives, FastAnd does the same intersections plus the lookups. Dense walks of six and eight inputs over 256 keys stay within 7% of pairwise And on the Xeon and within 2% on an M1.
• On an M1 the jointly-empty gain is smaller, 11.1 µs to 5.3 µs, because andCardSlice has no NEON version.
• BenchmarkIntersection*, BenchmarkUnion* and BenchmarkAndAny/* are unchanged.

Breaking Changes

None. Results are the same as with pairwise And.

Related Issues

Related to #566.

Additional Notes

arrayContainer.iandBitmap would gain from the same hoisting. That is left for a separate change.

🤖 Generated with Claude Code

[tamirms]

@guymolinari @lemire thanks for the quick turnaround on #570. Could you take a look at #566 and this PR when you have time? This one builds on #566 (its first commit is #566's), so #566 is the place to start. We use roaring in Stellar RPC, the RPC server for the Stellar network, and these two PRs significantly reduce allocations in our event queries, so I'd love to get them upstream rather than maintain a fork.

[guymolinari]

Thanks @tamirms. These look useful. I checked both changes against the base commit on Linux/amd64, using Go 1.24.4 on an AMD EPYC 9V74. I have two requests before approval:

#572: Avoid the large cursor allocation on early-empty intersections

I reproduced the dimension_003 allocation increase. The dataset contains 15,482 input bitmaps, so make([]int, len(bitmaps)) requests 123,856 bytes, rounded by Go to a 131,072-byte allocation. Together with the empty result, this explains the reported increase from 80 to 131,152 B/op.

The first two inputs already have an empty intersection, but the cursor array is allocated before discovering that. In three isolated runs, median time was approximately 37.5 us/op on the base implementation versus 55.0 us/op with #572. Allocation measurements were 80 B/op versus approximately 131,154 B/op.

Could you avoid allocating the full cursor workspace on this early-empty path and add a regression benchmark covering a large input count with an empty first-pair intersection?

#566: Benchmark fresh inputs for the in-place cases

The current in-place benchmark clones dense once and repeatedly intersects the same result. After the first iteration, it measures an already-filtered bitmap.

I reran both cases with a fresh clone before every iteration, excluding cloning from the timed section and checking each result against the expected intersection. Medians of three runs:

• Four long ranges: 2.05 us/op on base versus 0.577 us/op with runcontainer: count or test before allocating in run-bitmap intersections #566.
• 1,024 short intervals: 6.80 us/op on base versus 4.53 us/op with runcontainer: count or test before allocating in run-bitmap intersections #566.
• Both cases: temporary allocations fell from 8,224 B/op and 2 allocs/op to zero.

The gains hold up with fresh inputs. However, the four-range case measured approximately 258 ns/op with the reused input versus 577 ns/op with fresh inputs.

Could you update the in-place benchmarks and reported results to cover fresh inputs? A separate clone-plus-intersection benchmark would also help show the end-to-end cost.

The targeted run-container intersection and FastAnd tests passed in my checks.

[tamirms]

Thanks @guymolinari, both are addressed, and I fixed a few more things while I was there. Both PRs are rebased on current master, and the tables are re-measured on a Xeon 8375C.

#572, the cursor allocation. FastAnd now walks only the first four inputs key by key, with their cursors in a fixed array on the stack, and intersects the rest into the result one at a time, so there is nothing left to allocate. dimension_003 is back to 80 B/op and 1 alloc, as are 9 of the other 11 datasets in BenchmarkRealDataFastAnd. BenchmarkFastAndShapes has a singletons-10000 row, 10,000 inputs with an empty first pair, and TestFastAndEmptyAllocatesNoContainer asserts a single allocation for 1,000 of them.

#566, the in-place benchmark. The in-place cases now intersect a fresh copy each iteration, cloned in batches outside the timer, and new clone+ cases show the end-to-end cost. The table is updated.

Also changed:

• Walking every input for every key made FastAnd up to 1.9x slower than the plain pairwise And it replaces when keys survive many inputs (1,024 inputs of 256 keys). Walking only the first four removes that.
• A key with several many-interval runs no longer intersects the runs into a new container; each run clears its gaps in the scratch bitmap.
• The random FastAnd test now uses inputs that share keys, up to 12 of them, so about a third of its cases reach the intersection; before, most stopped at a missing key. The shape benchmark puts a real input first and uses real run containers.
• ParAnd's worker ran the same loop as the new andKChain and now calls it.
• In runcontainer: count or test before allocating in run-bitmap intersections #566, the scratch path is used only where it is faster: runs covering more than 4096 values, or CPUs with the AVX-512 vector fill. For runs of many short intervals it was up to 2.9x slower. An in-place And with a full range no longer copies a bitmap or intersects a run, and Intersects between an array and a run is up to 1.9x faster.
• Both descriptions are rewritten, and a few claims that did not hold are corrected.

[guymolinari]

Thanks @tamirms. I reviewed #566 at 1c0052a and #572 at fccdc63 and independently reran the relevant checks with Go 1.24.4 on Linux/amd64, AMD EPYC 9V74.

Both original concerns are resolved:

• fastaggregation: count before allocating in FastAnd #572: Fixed-size cursor storage removes the large early-empty workspace allocation. dimension_003 now measures 80 B/op and one allocation, with a three-run median of 16.5 µs versus 32.6 µs on current master. The 10,000-singleton benchmark also measures 80 B/op and one allocation.
• runcontainer: count or test before allocating in run-bitmap intersections #566: The in-place benchmarks consume fresh clones outside timing, and the clone+ cases expose end-to-end costs. Fresh-input medians improved from 1.76 to 0.317 µs for four long ranges and 6.98 to 4.62 µs for 1,024 intervals, with zero timed allocations in both revised cases.

Both full test suites passed, along with the focused appengine-build checks on #572. Additional late-driver, late-empty, copy-on-write, and 1,024-input/256-surviving-key checks passed. Longer runs of the surviving-key benchmark showed no reproducible regression.

The scratch masking, container selection, and result ownership paths look sound. Please consider retaining the larger surviving-key workload as a benchmark.

My previous objections are resolved; I’m comfortable approving both, with #566 merged first.

[tamirms]

Thanks @guymolinari. Done: BenchmarkFastAndShapes now has a surviving-1024x256 row, 1,024 inputs of 256 keys that all survive, and it's in the table in the description. On the Xeon it takes 13.3 ms against 13.1 ms for pairwise And, with fewer allocations.

[tamirms]

@guymolinari thanks again for reviewing these. Testing #572 further, I found that FastAnd could still allocate for a key whose intersection comes out empty: when the smallest container was an array, when every container was a run, or when only an input after the first four emptied it. I pushed one more commit on top, d5dd08c, to cover those cases, and the commits you reviewed are unchanged.

When the input with the fewest keys has a single key, FastAnd now checks every input, up to eight, before allocating, so an empty intersection allocates only the result bitmap, whatever the container kinds. For example, a one-key range ANDed with two disjoint 4,096-value arrays went from 8,296 bytes per call to 80. The description has the updated numbers.

Would you mind taking a look at the new commit when you have a chance?