Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alternate Compression Implementations #9555

Open
cody-littley opened this issue Oct 27, 2023 · 1 comment
Open

Alternate Compression Implementations #9555

cody-littley opened this issue Oct 27, 2023 · 1 comment
Assignees
Labels
Platform Tickets pertaining to the platform

Comments

@cody-littley
Copy link
Contributor

cody-littley commented Oct 27, 2023

Gzip compression in the standard java libraries is too slow for reconnect. Consider alternate implementations that are faster.

@cody-littley cody-littley added this to the v0.45 milestone Oct 27, 2023
@cody-littley cody-littley self-assigned this Oct 27, 2023
@cody-littley cody-littley changed the title Protocol specific compression Alternate Compression Implementations Oct 31, 2023
@poulok poulok removed this from the v0.45 milestone Feb 9, 2024
@kfrz
Copy link

kfrz commented Feb 17, 2024

https://engineering.fb.com/2016/08/31/core-infra/smaller-and-faster-data-compression-with-zstandard/

https://datatracker.ietf.org/doc/html/rfc8878

https://swiftpackageindex.com/facebook/zstd

https://github.com/facebook/zstd/blob/dev/Package.swift

https://swiftpackageindex.com/facebook/zstd

https://github.com/aperedera/SwiftZSTD

https://facebook.github.io/zstd/#other-languages

https://github.com/Cyan4973/FiniteStateEntropy

Without personal insight into the finer details of the existing benchmarks as it compares apples-to-apples, this still looks compelling to at least consider. When I searched I was pleased to find concurrent thinking!

I can't into Java-ing, but I did read about zstd this weekend they're puttin' up Pied Piper numbers, Weissman score in the fives:


The following is sourced from the README

Benchmarks

For reference, several fast compression algorithms were tested and compared
on a desktop running Ubuntu 20.04 (Linux 5.11.0-41-generic),
with a Core i7-9700K CPU @ 4.9GHz,
using lzbench, an open-source in-memory benchmark by @inikep
compiled with gcc 9.3.0,
on the Silesia compression corpus.

Compressor name Ratio Compression Decompress.
zstd 1.5.1 -1 2.887 530 MB/s 1700 MB/s
zlib 1.2.11 -1 2.743 95 MB/s 400 MB/s
brotli 1.0.9 -0 2.702 395 MB/s 450 MB/s
zstd 1.5.1 --fast=1 2.437 600 MB/s 2150 MB/s
zstd 1.5.1 --fast=3 2.239 670 MB/s 2250 MB/s
quicklz 1.5.0 -1 2.238 540 MB/s 760 MB/s
zstd 1.5.1 --fast=4 2.148 710 MB/s 2300 MB/s
lzo1x 2.10 -1 2.106 660 MB/s 845 MB/s
lz4 1.9.3 2.101 740 MB/s 4500 MB/s
lzf 3.6 -1 2.077 410 MB/s 830 MB/s
snappy 1.1.9 2.073 550 MB/s 1750 MB/s

The negative compression levels, specified with --fast=#,
offer faster compression and decompression speed
at the cost of compression ratio (compared to level 1).

Zstd can also offer stronger compression ratios at the cost of compression speed.
Speed vs Compression trade-off is configurable by small increments.
Decompression speed is preserved and remains roughly the same at all settings,
a property shared by most LZ compression algorithms, such as zlib or lzma.

@poulok poulok added the Platform Tickets pertaining to the platform label Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Platform Tickets pertaining to the platform
Projects
None yet
Development

No branches or pull requests

3 participants