Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

zstd: Increase speed on incompressible data for best #314

Merged
merged 1 commit into from
Feb 5, 2021

Conversation

klauspost
Copy link
Owner

@klauspost klauspost commented Feb 5, 2021

Increase speed of incompressible data for "best"/"fastest" mode.

Should not have any impact on compressible data:

before/after...

sharnd.out.2gb	zskp	4	2147483647	2147581961	52931	38.69
sharnd.out.2gb	zskp	4	2147483647	2147581961	15426	132.76

silesia.tar	zskp	4	211947520	61381950	8394	24.08
silesia.tar	zskp	4	211947520	61327981	8142	24.82

BenchmarkRandom10MBEncodeAllFastest-32    	     279	   3971326 ns/op	2640.37 MB/s	       0 B/op	       0 allocs/op
BenchmarkRandom10MBEncodeAllFastest-32    	     349	   3498572 ns/op	2997.15 MB/s	       0 B/op	       0 allocs/op

Increase speed of incompressible data for "best" mode.

Should not have any impact on compressible data:

before/after...
```
sharnd.out.2gb	zskp	4	2147483647	2147581961	52931	38.69
sharnd.out.2gb	zskp	4	2147483647	2147581961	15426	132.76

silesia.tar	zskp	4	211947520	61381950	8394	24.08
silesia.tar	zskp	4	211947520	61327981	8142	24.82

BenchmarkRandom10MBEncodeAllFastest-32    	     279	   3971326 ns/op	2640.37 MB/s	       0 B/op	       0 allocs/op
BenchmarkRandom10MBEncodeAllFastest-32    	     349	   3498572 ns/op	2997.15 MB/s	       0 B/op	       0 allocs/op
```
@klauspost klauspost merged commit b5cd744 into master Feb 5, 2021
@klauspost klauspost deleted the zstd-less-aggressive-best branch February 5, 2021 13:17
mostynb added a commit to mostynb/zstdpool-syncpool that referenced this pull request Mar 27, 2021
There have been a few zstd improvements since v1.11.6:

* zstd: Big speedup on small dictionary encodes
  klauspost/compress#345
* zstd: Add WithLowerEncoderMem
  klauspost/compress#336
* Faster "compression" of incompressible data
  klauspost/compress#314
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant