Add RepMaxCDC as a selectable remote cache chunking function

[ifutivic]

Description

Adds support for the RepMaxCDC function to the remote cache chunking feature, selectable via a new flag --experimental_remote_cache_chunking_function={auto,fast_cdc_2020,rep_max_cdc}. With auto (the default) the function is negotiated from the server capabilities: FastCDC 2020 is used if the server advertises it (including when it advertises both, for backward compatibility), otherwise RepMaxCDC. The explicit values require the selected function to be advertised by the server. Against servers that advertise FastCDC 2020 today, behavior is unchanged.

• RepMaxCdcChunker ports the algorithm from the Go reference implementation (buildbarn/go-cdc). Chunk boundaries are byte-identical to the reference implementation. The test vectors in RepMaxCdcChunkerTest were generated by running it on the same test image used by the FastCDC test vectors.
• ChunkingConfig becomes a sealed interface with per-function parameter records (FastCdcChunkingConfig, RepMaxCdcChunkingConfig) negotiated from the server capabilities (FastCdc2020Params / RepMaxCdcParams).
• Both chunkers implement ContentDefinedChunker and share the Gear table (GearTable).
• Capability validation, SplitBlob/SpliceBlob requests, and the remote worker use the selected function instead of hardcoding FAST_CDC_2020.
• Tests mirror the existing FastCDC coverage: chunker unit tests with reference test vectors, config negotiation tests, a JMH benchmark, and an integration test running the chunked cache flow end-to-end with RepMaxCDC against the remote worker.

Performance

JMH throughput of chunkToDigests (chunking plus BLAKE3 chunk digests, single thread, Apple M-series). Average chunk sizes are measured: FastCDC's 512 KiB parameter is its "expected" size and comes out at ~595 KiB in practice, so RepMaxCDC is shown both with its defaults and with min = 448 KiB, which matches FastCDC's true average chunk size:

Function	Parameters	True avg chunk size	64 MiB	256 MiB	1 GiB
FastCDC 2020	expected 512 KiB (defaults)	~595 KiB	1.17 ± 0.06 GiB/s	1.16 ± 0.05 GiB/s	1.15 ± 0.04 GiB/s
RepMaxCDC	min 256 KiB (defaults)	~340 KiB	0.98 ± 0.02 GiB/s	0.99 ± 0.03 GiB/s	0.96 ± 0.10 GiB/s
RepMaxCDC	min 448 KiB (matched to FastCDC)	~598 KiB	1.00 ± 0.02 GiB/s	1.00 ± 0.04 GiB/s	0.97 ± 0.14 GiB/s

RepMaxCDC reaches ~85% of FastCDC throughput.

Note: these numbers include one additional optimization compared to the original Go implementation, a four-way unrolling of the Gear hash loop. Chunk boundaries are unaffected and the test vectors are still the same. Without it, RepMaxCDC sits at ~70-75% of FastCDC.

RepMaxCdcChunker carries a small primitive IntList helper instead of using ArrayList<Integer>: a variant backed by a boxed list measured ~2.2x slower overall.

Disclaimer

This PR heavily leveraged Claude Fable to rewrite the Go implementation of RepMaxCDC into Java but the code was manually reviewed by me (which is not saying a lot since I'm not a Java dev :D).

Motivation

Tracking issue: #30132

--experimental_remote_cache_chunking currently always uses FastCDC 2020 to split large blobs. The Remote Execution API also defines the RepMaxCDC chunking function (REP_MAX_CDC / RepMaxCdcParams), as implemented by buildbarn/go-cdc, but Bazel does not implement it as a client.

RepMaxCDC provides a deduplication rate comparable to FastCDC while giving tight bounds on chunk sizes: all chunks fall within [min_chunk_size, 2 * min_chunk_size), so for a given blob it is trivial to check whether it is already chunked purely by looking at its size. Supporting it lets Bazel clients share chunk data with servers and other clients that use RepMaxCDC (e.g. future Buildbarn deployments).

Build API Changes

This PR adds a new experimental command-line flag, --experimental_remote_cache_chunking_function, and extends the existing --experimental_remote_cache_chunking feature.

1. RepMaxCDC is already part of the Remote Execution API (ChunkingFunction.REP_MAX_CDC / RepMaxCdcParams); this PR implements the client side in Bazel.
2. The change is fully backward compatible: the new flag defaults to auto, which negotiates the function from the server capabilities and prefers FastCDC 2020 whenever the server advertises it, preserving the current behavior of --experimental_remote_cache_chunking against existing servers.
3. Not a breaking change.

Checklist

• I have added tests for the new use cases (if any).
• I have updated the documentation (if applicable).

Release Notes

RELNOTES[NEW]: Added --experimental_remote_cache_chunking_function to select the content-defined chunking function used by --experimental_remote_cache_chunking. Supported values are auto (the default, which negotiates the function from the server capabilities, preferring FastCDC 2020), fast_cdc_2020 and rep_max_cdc.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[EdSchouten]

Hello Ivan,

Performance

JMH throughput of chunkToDigests (chunking plus BLAKE3 chunk digests, single thread, Apple M-series), each algorithm with its default parameters (FastCDC: 512 KiB average chunk size; RepMaxCDC: 256 KiB minimum chunk size, 2 MiB horizon):

Input FastCDC 2020 RepMaxCDC RepMaxCDC with ArrayList<Integer>
8 MiB 137.7 ± 13.8 ops/s 104.3 ± 3.1 ops/s 46.4 ± 2.7 ops/s
64 MiB 18.0 ± 1.1 ops/s 12.6 ± 0.4 ops/s 5.6 ± 0.2 ops/s
RepMaxCDC reaches roughly 70-75% of FastCDC throughput (~0.8 GiB/s vs ~1.1 GiB/s). The last column is why RepMaxCdcChunker carries a small primitive IntList helper instead of using ArrayList<Integer>.

Yeah, these numbers seem to be expected. FastCDC and RepMaxCDC should have virtually the same throughput when it comes to raw input processing, as both use Gear hashing. However, RepMaxCDC needs to do slightly more work:

• Whenever RepMaxCDC discovers a new 'best' cutting point during its hashing loop, it inserts it into the incompleteChunks list. Only a fraction of these points end up being selected as true cutting points, meaning some effort is wasted. I expect that for sufficiently large chunk sizes this isn't really noticeable though.
• FastCDC has a feature that the paper refers to as 'sub-minimum chunk cut-point skipping'. This causes certain parts of the input file to never get hashed. This makes the algorithm run faster, but results in worse deduplication rates.

The above may also not be an apples to apples comparison. With the parameters used above, FastCDC will likely have a true average chunks size closer to 600 KB, as the 512 KB you stated is what is in FastCDC referred to as the 'expected' or 'normal' size. In the case of RepMaxCDC you'll likely see that the average chunk size is about $256 \cdot 1.3376 \approx 342$ KB. It might be worth rerunning the same benchmark with comparable chunk sizes.

Disclaimer

This PR heavily leveraged Claude Fable to rewrite the Go implementation of RepMaxCDC into Java but the code was manually reviewed by me (which is not saying a lot since I'm not a Java dev :D).

Glad to see the Go implementation was of use!

[ifutivic]

Hi Ed, thanks for taking a look!

The above may also not be an apples to apples comparison. With the parameters used above, FastCDC will likely have a true average chunks size closer to 600 KB, as the 512 KB you stated is what is in FastCDC referred to as the 'expected' or 'normal' size. In the case of RepMaxCDC you'll likely see that the average chunk size is about
256 ⋅ 1.3376 ≈ 342 KB. It might be worth rerunning the same benchmark with comparable chunk sizes.

I reran the benchmark with comparable chunk sizes by setting RepMaxCDC's min to 448 KiB (448 * 1.3376 ≈ 599 KiB), which produced a measured ~598 KiB true average, matching FastCDC. Throughput of chunkToDigests (chunking plus BLAKE3 chunk digests, single thread, Apple M-series, JMH):

Function	Parameters	True avg chunk size	64 MiB	256 MiB	1 GiB
FastCDC 2020	expected 512 KiB (defaults)	~595 KiB	1.18 ± 0.04 GiB/s	1.18 ± 0.04 GiB/s	1.15 ± 0.11 GiB/s
RepMaxCDC	min 256 KiB (defaults)	~340 KiB	0.81 ± 0.02 GiB/s	0.79 ± 0.06 GiB/s	0.79 ± 0.03 GiB/s
RepMaxCDC	min 448 KiB (matched)	~598 KiB	0.80 ± 0.02 GiB/s	0.79 ± 0.11 GiB/s	0.76 ± 0.05 GiB/s

Seems like matching the chunk sizes changes almost nothing: RepMaxCDC at min 448 KiB is identical to min 256 KiB within error, even though it produces ~1.75x fewer chunks.

[ifutivic]

Hey @EdSchouten, while looking into the throughput gap I found an optimization for the gear hash loop.

In h = (h << 1) + gear[b] every step needs the previous hash, so the CPU is stuck doing one shift+add at a time. But since the recurrence is linear, the hashes of the next four positions can all be computed directly from the hash at a block start: h(i+k) = (hash << k) + s(k), where s(k) is the gear sum of the k block bytes. That leaves one shift+add per four bytes on the critical path. Every intermediate hash is computed exactly, so cutting point selection and bestHash updates are untouched and chunk boundaries stay byte-identical, the test vectors generated with go-cdc still pass unmodified.

It's worth ~1.4x on raw chunking throughput (1.32 to 1.89 GiB/s here), which brings RepMaxCDC to ~85% of FastCDC in the end-to-end benchmark. I updated the PR description with the new numbers. The unroll factor is an empirical optimum. 2/4/8-way give 1.38/1.89/1.48 GiB/s.

The change is in eca3101 on this PR. Since you're the original author of RepMaxCDC, would you mind taking a look to make sure I'm not breaking a case the tests don't catch? And if it holds up, the same trick should apply to the Go implementation, so it might make sense to add it to go-cdc as well. Happy to send a PR there if you're interested.

[EdSchouten]

Yeah, that seems about right. I've also seen implementations where they have multiple Gear tables. A stock one, and one that is pre-shifted by 1 or more bits. In theory we could also make a change like that where we compute recomputedHash, but it's likely less useful. That code path is hardly ever exercised and only runs for short inputs.

I was also wondering, what would the impact of something like this be?

          long original_best = best;
          if (Long.compareUnsigned(best, h1) < 0) {
            best = h1;
            incompleteChunks.add(currentChunk + idx + 1);
          }
          if (Long.compareUnsigned(original_best, h2) < 0 && Long.compareUnsigned(best, h2) < 0) {
            best = h2;
            incompleteChunks.add(currentChunk + idx + 2);
          }
          if (Long.compareUnsigned(original_best, h3) < 0 && Long.compareUnsigned(best, h3) < 0) {
            best = h3;
            incompleteChunks.add(currentChunk + idx + 3);
          }
          if (Long.compareUnsigned(original_best, h4) < 0 && Long.compareUnsigned(best, h4) < 0) {
            best = h4;
            incompleteChunks.add(currentChunk + idx + 4);
          }

As in, only compare against the latest best in case the new hash value is higher than the original value of best at the start of the loop. Maybe that leads to even fewer dependencies?

With regards to making a change like that to the Go implementation: go for it! The reason I didn't fiddle with any of that was because I was mainly interested in designing/optimizing the algorithm at a high level, not a low level.

[ifutivic]

I benchmarked your original_best variant: it measures within error of the committed version, maybe a touch slower.

I think that we're currently very close to the practical maximum without changing how the function fundamentally works. What remains of the gap to FastCDC looks like the sub-minimum cut-point skipping you mentioned earlier. With min = 128 KiB and a ~595 KiB true average chunk size, FastCDC never hashes ~22% of the input, while RepMaxCDC has to hash every byte to track the maxima. This lines up well with the performance difference we're seeing between the two functions when it comes to pure chunking throughput (~22% slower).

Pure chunking throughput (no digests, 64 MiB random input, default parameters, JMH single thread, Apple M-series):

Implementation	Throughput
FastCDC 2020	~2.47 GiB/s
RepMaxCDC (4-way unroll)	~1.92 GiB/s
RepMaxCDC (before the unroll)	~1.32 GiB/s

Note that the table in the PR description includes the BLAKE3 digest calculation, which runs over every byte for both functions, so the relative gap is smaller there.

I'll put together a PR for go-cdc with the unroll once this (hopefully) gets merged, thanks for the green light!

[sluongng]

While through put is an important side of this implementation, the other side would be chunk deduplication and size distribution.

it would be interesting to see how this algorithm impacts something like a local disk cache after building a range of commits in bazel.git, or some other bazel repo, in comparison to no chunking and fastcdc

it would also be nice to see how the chunk size distribution differs versus fastcdc once those benchmarks finished

[EdSchouten]

While through put is an important side of this implementation, the other side would be chunk deduplication and size distribution.

it would be interesting to see how this algorithm impacts something like a local disk cache after building a range of commits in bazel.git, or some other bazel repo, in comparison to no chunking and fastcdc

it would also be nice to see how the chunk size distribution differs versus fastcdc once those benchmarks finished

I think that all of these things are well understood at this point in time, as the findings are documented in the README.md of go-cdc.

• The maximum chunk size is $\mathit{MinSize} * 2 - 1$.
• The average chunk size is $\mathit{MinSize} / 0.74759\ldots$, where $0.74759\ldots$ is Rényi's parking constant.
• The exact size distribution of chunks also resembles that of the distance between cars in Rényi's car parking problem. There is enough academic literature on this subject.
• RepMaxCDC is slightly better at deduplicating data than FastCDC. In benchmarks that I performed where I aimed at the same average chunk size (prior to deduplication), RepMaxCDC used ~2-3% less space.

[ifutivic]

To add to what Ed said: the reason this thread has been so focused on throughput is that it's the only property where this PR could differ from the reference. Chunk boundaries produced by this implementation are byte-identical to go-cdc (the test vectors were generated with it), so the size distribution and deduplication behavior documented there carry over 1:1. What the Java port can get wrong is speed, so the benchmarking here was about making sure the port doesn't regress against Ed's Go implementation and holds up against the FastCDC implementation already in Bazel.

That said, a dedup comparison on actual Bazel build outputs (rather than kernel tarballs) at the chunk sizes this PR defaults to would still be interesting as follow-up data,

[ifutivic]

I ran the disk cache experiment @sluongng suggested.

Setup: built //src:bazel-dev at three bazel.git commits (~10 commits apart), oldest to newest, using a Bazel client built from this PR. Both chunking functions use their default parameters. The CAS is measured after each commit, so the deltas show how much each successive commit adds to the cache. Cache was fully wiped before switching to the next chunking function.

Commits: bf0de33 -> 99be0f5 -> 9ae0848

CAS size after each commit:

Config	bf0de33 (clean)	99be0f5	9ae0848	vs no chunking
No chunking	1.82 GiB	2.13 GiB (+319 MiB)	2.45 GiB (+326 MiB)	n/a
FastCDC 2020	1.61 GiB	1.66 GiB (+51 MiB)	1.72 GiB (+61 MiB)	-30%
RepMaxCDC	1.52 GiB	1.56 GiB (+46 MiB)	1.61 GiB (+53 MiB)	-34%

RepMaxCDC ends up ~6% smaller than FastCDC overall. A caveat: at default parameters part of that comes from RepMaxCDC chunking at finer granularity (~340 KiB vs ~600 KiB average chunk size) and having a lower chunking threshold (512 KiB vs 2 MiB). At matched average chunk sizes Ed measured 2-3% when testing using the Linux kernel.

Blob size distribution in the CAS after the final commit (all blobs, including small unchunked outputs):

Config	blobs	p50	p90	p99	max	mean
No chunking	34,990	849 B	32 KiB	452 KiB	169.6 MiB	73 KiB
FastCDC 2020	36,608	899 B	89 KiB	859 KiB	2 MiB	49 KiB
RepMaxCDC	38,372	1,029 B	257 KiB	457 KiB	512 KiB	44 KiB

[ifutivic]

Chunk size distribution (chunks only, across all three commits, after deduplication):

Function	chunks	min	p10	p50	p90	p99	max	mean
FastCDC 2020	1,722	769 B	325 KiB	599 KiB	1.0 MiB	2 MiB	2 MiB	677 KiB
RepMaxCDC	3,698	256 KiB	263 KiB	324 KiB	455 KiB	507 KiB	511.8 KiB	343 KiB

In the FastCDC distribution you can see some of the drawbacks of the function: the minimum is very low because the trailing bytes of a file end up as a small chunk, and there's a pile of chunks at exactly 2 MiB where no cut point was found before the max chunk size.

The distribution matches what Ed has in the go-cdc repo pretty closely, despite the different parameters:

[fmeum]

LGTM, but the way to select the algorithm may need more thought