compress/gzip: unable to reproduce the same gzip/DEFLATE compressed output as Python/Rust

[evandrix]

For the attached file eicar.bin.gz, 69 bytes of data X5O!P%@AP[4\PZX54(P^)7CC)7}$EICAR-STANDARD-ANTIVIRUS-TEST-FILE!$H+H*\n, including the trailing newline

Sample code in Go(lang):

	var buf bytes.Buffer
	zw,_ := gzip.NewWriterLevel(&buf,6)
	x,_ := hex.DecodeString("58354f2150254041505b345c505a58353428505e2937434329377d2445494341522d5354414e444152442d414e544956495255532d544553542d46494c452124482b482a0a")
	zw.Write(x)
	zw.Close()
	fmt.Println(hex.EncodeToString(buf.Bytes()))

produces a different output in the DEFLATE part, ignoring the gzip header, and CRC32 checksum and uncompressed size in footer: 8a30f5570c5075700c88368909888a3035d10888d3347776d634af5571f574760cd20d0e71f473710c72d175f40bf10cf30c0a0dd60d710d0ed175f3f4715554f1d0f6d0e202040000ffff

Under compression level=6, both Python and Rust (as well as the standard unix gzip cli) manage to all agree on the output: 8b30f5570c5075700c88368909888a3035d10888d3347776d634af5571f574760cd20d0e71f473710c72d175f40bf10cf30c0a0dd60d710d0ed175f3f4715554f1d0f6d0e20200 (DEFLATE part only, excluding gzip header and footer)

The difference, for the unobservant reader is in the starting byte 8a vs 8b, and the trailing bytes ...e20200 vs ...e202040000ffff

Is there a way to re-write the Golang code, so that the output is identical?

[martisch]

Its unclear if this is a question how to change the Go code to make the output match or a proposal that the standard library produce the exact byte by byte output as some other tools for compression.

If it is a question please see https://github.com/golang/go/wiki/Questions for venues to ask questions. The Go project does not use the issue tracker for answering questions about Go usage.

If this is a proposal please clarify why this matters (assuming the current output is otherwise correct according to compression specifications) and why it should be aligned with Python and Rust. Are there other tools/programming languages that produce different output? Why not align with them?

[evandrix]

nvm, i decided to go with zstd alternatively to avoid this inconsistency. thanks