zstd multithreading: borg never sets nb_workers, and job_size must be forced for it to help

[ThomasWaldmann]

Follow-up to #8217 (closed), which already established that libzstd's nb_workers can speed up compression, including the measurements in that thread from Jan 2026. See also #37 and #3500 for borg-level multithreading.

Two things that thread did not cover, and that change the picture for borg specifically:

1. borg never enables it. compress.pyx:306 calls zstd.compress(data, self.level) with no options, so nb_workers is always 0 and no borg user has ever benefited from any of this.
2. Simply setting nb_workers would still do nothing. zstd splits work into jobs sized from the window log, and everything borg feeds it is smaller than one job, so the workers never engage. The speedup only appears once job_size is set explicitly.

End-to-end borg create, 10 GiB

Single 10.01 GiB compressible file (text-like, ~3.57x, non-dedupable), -C zstd,3, -e aes256-ocb -i sha256, warm cache, median of 2 runs. Apple M3 Pro, 12 cores (6P+6E), 18 GiB RAM.

All three rows are the same borg binary - the variants are produced by swapping borg.compress.zstd for a shim that passes options through, so only the arguments to zstd.compress() differ.

variant	create	MB/s	cpu	repo size	vs 1 thread
single-threaded (= today)	55.0s	186	53.9s	3,005,680,422 B	-
nb_workers=12, default job size	56.7s	181	57.9s	3,005,433,829 B	0.97x
nb_workers=12, job_size=512K	33.8s	303	65.7s	3,012,993,516 B	1.63x

So: enabling workers alone is a small regression; forcing a small job size makes borg create 1.63x faster end to end on this workload.

The compressor in isolation shows where the cliff is (same machine, level 3, semi-compressible input):

input	1 worker	12 workers, default job	12 workers, job_size=512K
2 MiB (typical chunk)	342 MB/s	321 MB/s (0.94x)	802 MB/s (2.55x)
8 MiB (max chunk)	317 MB/s	312 MB/s (0.98x)	1290 MB/s (4.08x)
32 MiB	317 MB/s	986 MB/s (3.11x)	-
128 MiB	308 MB/s	1369 MB/s (4.45x)	-

This is, I think, the answer to the objection in #8217 that a 2 MiB chunk is too little data to split usefully: it is enough, but only if zstd is told to actually split it. With the default job size it silently declines to.

Costs

• Compression ratio: +0.243% (3,005,680,422 -> 3,012,993,516 bytes, ~7.3 MB on a 3 GB repo). Smaller jobs mean less match history per job.
• CPU time: +22% (65.7s vs 53.9s). It buys wallclock with otherwise-idle cores. That is a much better ratio than blake3-mt in use multi-threaded blake3 for big chunks, #9958 #9959 (+55% CPU for a 7% wallclock gain), but it is still a bad trade on a loaded server or a low-core NAS, so it should probably not be unconditional.
• Output is no longer deterministic. Where MT genuinely engages, single- and multi-threaded compression produce different bytes for the same input (verified at 32 MiB: 9,446,012 vs 9,450,303 bytes, both decompressing correctly). Chunk ids are unaffected because they are computed on the plaintext before compression, but pack contents - and therefore pack ids - would depend on worker count and job scheduling. Worth confirming nothing relies on byte-reproducible compression output.

Open questions

• Opt-in or default? Given the CPU cost and the ratio cost, an env var (like BORG_BLAKE3_MT_THRESHOLD from use multi-threaded blake3 for big chunks, #9958 #9959) or a --compression zstd,3,mt style spec seems safer than switching it on for everyone.
• job_size is almost certainly level-dependent - higher levels have larger windows, so a fixed 512K is unlikely to be right across the range. Needs a sweep per level.
• Only measured on 12 cores with one data shape. A 2- or 4-core machine may see much less, and the ratio penalty may differ on other content.
• Does this interact badly with borg-level multithreading later (Multithreading #37, multithreading: input file discovery / reading parallelism #3500)? If borg eventually compresses several chunks in parallel, per-chunk MT would oversubscribe. Whatever is done here should be easy to switch off.

I have a zstd-mt branch to prototype this if it looks worth pursuing.

Measurement scripts (not in-tree): the end-to-end harness swaps the module-level zstd in borg.compress at runtime, so both variants run the same compiled extension - happy to contribute it if useful.

🤖 Generated with Claude Code