Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
23ee9d3
Updating to use zStd library version 1.0.
joshcarter Sep 6, 2016
80eec81
Minor refactor to use built-in error handling methods.
joshcarter Sep 7, 2016
af19ae6
Adding C sources for zStd 1.0 in favor of shared lib.
joshcarter Sep 7, 2016
add9e2c
Small fixes to the README. Fix #8
Viq111 Dec 21, 2016
45f7613
Update zstd to 1.1.3
Viq111 Feb 22, 2017
e95daff
Silence Zbuff deprecation warnings. See #10
Viq111 Feb 22, 2017
b952002
Merge pull request #12 from DataDog/viq111/1.1.3_zbuff
Viq111 Feb 24, 2017
8119789
Update Readme to reflect the correct C sources
Viq111 Mar 23, 2017
dcab8fb
[github] Add issue templating
Viq111 Apr 10, 2017
97f12e5
Update to zstd 1.3.0
elijahandrews Aug 10, 2017
755efe7
Fix CompressBound
elijahandrews Aug 10, 2017
0727e17
Merge pull request #18 from DataDog/elijah/1.3.0
elijahandrews Sep 7, 2017
7f86b1e
reduce memory usage in Read()
davies Sep 6, 2016
f48c28d
remove dstBuffer
davies Sep 9, 2016
066032d
[readme] Add comment about running benchmarks
Viq111 Dec 25, 2017
c3115f6
zstd 1.3.3
ghatdev Jan 25, 2018
737c417
[travis] Add .travis.yml
Viq111 Jan 25, 2018
1f4c6b2
[travis] Put it under a script
Viq111 Jan 25, 2018
2363f41
[travis] Support only 2 most recent go version
Viq111 Jan 25, 2018
0c8ca9c
[travis] Try building on OSX too
Viq111 Jan 25, 2018
37456d4
Merge pull request #24 from DataDog/viq111/travis
Viq111 Jan 25, 2018
5cb8fa2
Merge remote-tracking branch 'upstream/1.x' into 1.x
ghatdev Jan 27, 2018
8581291
Update zstd_stream.go
davies Feb 3, 2018
871f24a
Merge pull request #20 from juicedata/reader
Viq111 Mar 13, 2018
e785095
[travis] Add Go 1.10 to list of tested go versions
Viq111 Mar 13, 2018
cb9eedf
Merge pull request #27 from DataDog/viq111/1.10
Viq111 Mar 13, 2018
8860579
[zstd][errors] Get rid of static error code translation and use C call
Viq111 Mar 14, 2018
039e6f5
[errors] nit: remove intermediate variable
Viq111 Mar 14, 2018
00c3d53
[tests][errors] Make TestFindIsDstSizeTooSmallError stricter
Viq111 Mar 14, 2018
3b90611
[zstd] Linting
Viq111 Mar 14, 2018
de6590b
Merge pull request #29 from DataDog/viq111/fix-error-assignment
Viq111 Mar 14, 2018
a74c416
Fix CompressBound to latest implementation
ghatdev Mar 15, 2018
ca64c66
Merge pull request #23 from ghatdev/1.x
Viq111 Mar 15, 2018
19f0f4b
Prevent from unbound buffer growth in BenchmarkStreamCompression
valyala Mar 21, 2018
e068f47
Eliminate memory allocations from Compress/Decompress.
valyala Mar 21, 2018
a75d40d
Update README for zstd 1.3.4
elijahandrews Mar 30, 2018
80699a7
Update vendored zstd c lib to 1.3.4
elijahandrews Mar 30, 2018
f300818
Merge pull request #33 from DataDog/elijah/1.3.4
elijahandrews Mar 30, 2018
f9cd78f
Merge branch '1.x' into remove-mallocs-from-compress-decompress
Viq111 Mar 30, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/ISSUE_TEMPLATE
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
Please answer these questions before submitting your issue. Thanks!

### What version of Go are you using (`go version`)?


### What operating system and processor architecture are you using (`go env`)?


### What did you do?

If possible, provide a recipe for reproducing the error.
If you have issues building, please parse the output of `go build -x`


### What did you expect to see?


### What did you see instead?

18 changes: 18 additions & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
language: go

go:
- 1.8.x
- 1.9.x
- 1.10.x

os:
- linux
- osx

install:
- "wget http://sun.aei.polsl.pl/~sdeor/corpus/mr.bz2"
- "bzip2 -d mr.bz2"
script:
- "go build"
- "PAYLOAD=`pwd`/mr go test -v"
- "PAYLOAD=`pwd`/mr go test -bench ."
16 changes: 11 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,12 @@

[C Zstd Homepage](https://github.com/Cyan4973/zstd)

The current headers and C files are from *v0.5.0* (Commit [201433a](https://github.com/Cyan4973/zstd/commits/201433a7f713af056cc7ea32624eddefb55e10c8)).

This version has been tested and used in Datadog production environment and is safe for use.
The current headers and C files are from *v1.3.4* (Commit
[2555975](https://github.com/facebook/zstd/releases/tag/v1.3.4)).

## Usage

There are two main APIs:
There are two main APIs:

* simple Compress/Decompress
* streaming API (io.Reader/io.Writer)
Expand Down Expand Up @@ -70,13 +69,20 @@ NewReader(r io.Reader) io.ReadCloser
NewReaderDict(r io.Reader, dict []byte) io.ReadCloser
```

### Benchmarks
### Benchmarks (benchmarked with v0.5.0)

The author of Zstd also wrote lz4. Zstd is intended to occupy a speed/ratio
level similar to what zlib currently provides. In our tests, the can always
be made to be better than zlib by chosing an appropriate level while still
keeping compression and decompression time faster than zlib.

You can run the benchmarks against your own payloads by using the Go benchmarks tool.
Just export your payload filepath as the `PAYLOAD` environment variable and run the benchmarks:

```go
go test -bench .
```

Compression of a 7Mb pdf zstd (this wrapper) vs [czlib](https://github.com/DataDog/czlib):
```
BenchmarkCompression 5 221056624 ns/op 67.34 MB/s
Expand Down
22 changes: 13 additions & 9 deletions ZSTD_LICENSE
Original file line number Diff line number Diff line change
@@ -1,18 +1,22 @@
ZSTD Library
Copyright (c) 2014-2015, Yann Collet
All rights reserved.

BSD License

For Zstandard software

Copyright (c) 2016-present, Facebook, Inc. All rights reserved.

Redistribution and use in source and binary forms, with or without modification,
are permitted provided that the following conditions are met:

* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.
* Redistributions of source code must retain the above copyright notice, this
list of conditions and the following disclaimer.

* Redistributions in binary form must reproduce the above copyright notice,
this list of conditions and the following disclaimer in the documentation
and/or other materials provided with the distribution.

* Redistributions in binary form must reproduce the above copyright notice, this
list of conditions and the following disclaimer in the documentation and/or
other materials provided with the distribution.
* Neither the name Facebook nor the names of its contributors may be used to
endorse or promote products derived from this software without specific
prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
Expand Down
Loading