Zstandard 1.0 released

[jadbox]

Has anyone tried the official 1.0 with the Go Bindings?

https://code.facebook.com/posts/1658392934479273/smaller-and-faster-data-compression-with-zstandard/

[joshcarter]

FWIW I attempted to replace the .c/.h files in this package from the ones in zStd 1.0's lib folder. After fixing a header file name, I'm getting these errors:

could not determine kind of name for C.ZSTD_compressBegin
could not determine kind of name for C.ZSTD_compressBegin_usingDict
could not determine kind of name for C.ZSTD_compressContinue
could not determine kind of name for C.ZSTD_compressEnd

I did some googling around for this type of error, but didn't find any obvious fix.

[Viq111]

There are a few API changes in 1.0.
Bumping up zstd version 1.0 is under way but we also need to find a way to update the package seamlessly without breaking it for people already using it (i.e: still compress with 0.5 compatibility for a bit but being able to decompress v0.5 through 1.0).

@joshcarter For could not determine kind of name, there is a "hidden" ifdef in zstd.h that you need to set.

[dgryski]

I think creating a DataDog/zstd1 repo for v1+ is justifiable. It's ugly, but solves the problem of backwards compatibility.

[joshcarter]

@Viq111 Thanks for the tip, I set ZSTD_STATIC_LINKING_ONLY so those function definitions would be visible. I'm able to compile against the zstd shared library but I get some unit test failures:

    zstd_stream_test.go:22: Compressed 130000 -> 33 bytes
    zstd_stream_test.go:11: Failed to decompress with Decompress(): tableLog requires too much memory
--- FAIL: TestErrorCode (0.00s)
    zstd_test.go:48: C function returned error for -19 but ours did not
--- FAIL: TestCompressDecompress (0.00s)
    zstd_test.go:68: Compressed: [40 181 47 253 32 12 97 0 0 72 101 108 108 111 32 87 111 114 108 100 33]
    zstd_test.go:76: Error while decompressing: tableLog requires too much memory
--- FAIL: TestTooSmall (0.00s)
    zstd_test.go:108: Failed decompressing: tableLog requires too much memory

(I'll poke at this some more.)

@dgryski Why create a whole separate project? From what I can tell, zStardard 1.0 can decompress all prior versions.

[dgryski]

@joshcarter Ah, cool. I wasn't sure how much backwards compatibility they kept while developing the format. In that case it shouldn't be a problem to upgrade this package. (Although it will likely lead to a few multi-stage deploys as the users try to figure out the right order to upgrade pieces of their pipeline so everything can still be read.)

[joshcarter]

About prior test failures: the messages are wrong. I replaced the manually-coded Error() function's switch statement with something that calls into zStandard's error mapping function. Words of wisdom: don't bring magic numbers from the C library into your Go code!

I'm homing in on fixes...

[Viq111]

Yes zstd have backward decoding capabilities but as @dgryski pointed out, you can't start compressing using 1.0 if some of your pipeline still run the 0.5-only compatible version.
We are going to use the same method as the zstd repo:

• The current master will be tagged as v0.5
• Will update the bindings to use 1.0 decompression but still do 0.5 compression, tagged as v0.5.1 and named "0.5-transition" (as in zstd: https://github.com/facebook/zstd/releases/tag/v0.6.2)
• Will finally update master to 1.0 and tag v1.0

@joshcarter Yep also for error mapping, maybe waiting on facebook/zstd#341 too which can lead to incorrect error codes

[joshcarter]

I fixed the issues I was seeing--mostly variations on the error map being stale--and submitted a pull request. Runs successfully against the zStd 1.0 library on MacOS X. Thanks DataDog folks for all the hard work on bringing zStd to Go!

[jcftang]

GIven there is a 1.1.0 release, is it worth updating to 1.1.0?

[Viq111]

Closing this one as we have released the bindings for 1.0

@jcftang I'll usually wait for the minor to be stable before releasing the bindings, seems like they are at 1.1.2 for now, let's try to update when 1.1 release cycle is complete (and feel free to reopen an issue so we can keep track of the update)

[dgryski]

What's the timeline for master being updated to 1.1 (via #4 (comment) )