deflate: Improve level 5+6 compression #367
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Improve deflate level 5+6 compression by checking an additional hash when the best found match ends. This improves compression in most cases at an acceptable speed loss which brings it will in line with the surrounding parameters. Level 5 tries one hash at length < 30 and level 6 tries 2 at all lengths. Before/after pairs... ``` file out level insize outsize millis mb/s github-june-2days-2019.json gzkp 5 6273951764 963122453 31498 189.96 github-june-2days-2019.json gzkp 5 6273951764 947567306 32795 182.45 github-june-2days-2019.json gzkp 6 6273951764 949824639 34851 171.68 github-june-2days-2019.json gzkp 6 6273951764 930428507 37312 160.35 nyc-taxi-data-10M.csv gzkp 5 3325605752 785784479 24729 128.25 nyc-taxi-data-10M.csv gzkp 5 3325605752 779343831 27189 116.65 nyc-taxi-data-10M.csv gzkp 6 3325605752 775719630 26690 118.83 nyc-taxi-data-10M.csv gzkp 6 3325605752 768153050 29905 106.05 enwik9 gzkp 5 1000000000 338823570 10477 91.02 enwik9 gzkp 5 1000000000 337489137 11353 84.00 enwik9 gzkp 6 1000000000 336549505 10791 88.37 enwik9 gzkp 6 1000000000 334933748 11961 79.73 gob-stream gzkp 5 1911399616 309832207 8596 212.03 gob-stream gzkp 5 1911399616 307765377 9101 200.28 gob-stream gzkp 6 1911399616 308962175 9626 189.35 gob-stream gzkp 6 1911399616 301609641 10305 176.88 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Improve deflate level 5+6 compression by checking an additional hash when the best found match ends.
This improves compression in most cases at an acceptable speed loss which brings it will in line with the surrounding parameters.
Level 5 tries one hash at length < 30 and level 6 tries 2 at all lengths.
Before/after pairs...