-
Notifications
You must be signed in to change notification settings - Fork 507
compression: add an adaptive compressor #4937
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
compression: add an adaptive compressor #4937
Conversation
02a5fe9
to
e434ab4
Compare
This will be used to estimate compression ratio of blocks based on the compression ratios of recently sampled blocks.
Add AdaptiveCompressor, which estimates the size reduction of using a slower algorithm vs a faster one and chooses automatically (on a per-block basis). We will separately update the compression analyzer to run experiments with the adaptive compressors.
1f75a45
to
bedb5de
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 10 of 10 files at r1, 2 of 2 files at r2, 4 of 4 files at r3, 8 of 8 files at r4, all commit messages.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on @jbowens)
TFTR! Some initial tests on TPCC data show it is a pretty decent trade-off between MinLZ and Zstd1, especially in decompression speed. This is for ~32KB data blocks, broken down by the "compressibility" of the block (as determined by MinLZ). The cutoff is set to 30%:
|
Switch experiments from using `compression.Setting` to `block.CompressionProfile` and add two adaptive profiles.
bedb5de
to
bbb055b
Compare
Informs #4925
compression: return Setting from Compress
Flip-flopping on my earlier change :)
ewma: add a per-bye EWMA estimator
This will be used to estimate compression ratio of blocks based on the
compression ratios of recently sampled blocks.
compression: add an adaptive compressor
Add AdaptiveCompressor, which estimates the size reduction of using a
slower algorithm vs a faster one and chooses automatically (on a
per-block basis).
We will separately update the compression analyzer to run experiments
with the adaptive compressors.
compressionanalyzer: support adaptive compressors
Switch experiments from using
compression.Setting
toblock.CompressionProfile
and add two adaptive profiles.