Skip to content
This repository was archived by the owner on Aug 25, 2025. It is now read-only.

Commit 3d7273c

Browse files
committed
Use at most 8 threads for the xz stream
At preset 6, xz2 uses about 173MB of memory per thread. This adds up quickly -- e.g. over 8GB of memory on a 48-CPU machine. If you happen to try this in a 32-bit build, you'll get `LZMA_MEM_ERROR`. We can limit this to a heuristic maximum number of threads to avoid using so much memory, like xz's [`04_compress_easy_mt` example]. // The number 8 is arbitrarily chosen and may be too low or // high depending on the compression preset and the computer // being used. [`04_compress_easy_mt` example]: https://github.com/xz-mirror/xz/blob/de1f47b2b40e960b7bc3acba754f66dd19705921/doc/examples/04_compress_easy_mt.c#L71
1 parent 85958b0 commit 3d7273c

File tree

1 file changed

+4
-2
lines changed

1 file changed

+4
-2
lines changed

src/tarballer.rs

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,11 @@ impl Tarballer {
4747
// Prepare the `.tar.gz` file.
4848
let gz = GzEncoder::new(create_new_file(tar_gz)?, flate2::Compression::best());
4949

50-
// Prepare the `.tar.xz` file.
50+
// Prepare the `.tar.xz` file. Note that preset 6 takes about 173MB of memory
51+
// per thread, so we limit the number of threads to not blow out 32-bit hosts.
52+
// (We could be more precise with `MtStreamBuilder::memusage()` if desired.)
5153
let stream = xz2::stream::MtStreamBuilder::new()
52-
.threads(num_cpus::get() as u32)
54+
.threads(Ord::min(num_cpus::get(), 8) as u32)
5355
.preset(6)
5456
.encoder()?;
5557
let xz = XzEncoder::new_stream(create_new_file(tar_xz)?, stream);

0 commit comments

Comments
 (0)