zstd as the new default compression

[ThomasWaldmann]

The change (support negative levels) and benchmarking done related to #10084 plus zstd's multithreading capabilities (supported since #9962) seem to indicate that it is a better default than lz4.

For the default compression, we prioritize speed over compression ratio (that's why we used lz4 in the past).

The benchmarks seem to indicate that zstd,-4 might be a good default, but we should do real-world benchmarks with real data before committing to that.

The goal is to be usually as fast or faster than lz4 and using MT for better speed. We do not only look at the big chunk sizes (target size of borg's chunker is usually ~2MiB), but we also consider that backups usually also backup a lot of small files that will only result in 1 small chunk and for these, MT won't kick in - so it should have good speed for these also.

If it also compresses better sometimes than lz4, that is welcome, too.

[ThomasWaldmann]

We could even think about whether we want to remove lz4 - that would remove one library dependency (unlike removing lzma and zlib: as they come from Python stdlib, they are kind of "for free").

[PhrozenByte]

We could even think about whether we want to remove lz4

Wouldn't that require most users (assuming they used the default) to recompress all their archives with borg2 transfer --from-borg1? Changing the default sounds reasonable following the findings in #9950, but removing lz4 is a whole different story IMHO.

[ThomasWaldmann]

Yes, but:

• considering the compression performance, some users may want to do it anyway
• users may want to do the full dance (over plaintext stream) when transferring due to other reasons:
  • blake2b -> blake3 (or hmac-sha256) - no blake2b in borg2
  • switch to a better (faster or more secure) chunker