Upstream zlib library not well optimized

[jakobnissen]

I was studying why FASTX.jl on gzipped FASTQ files was so slow in the benchmark at https://github.com/lh3/biofast. It appears the unzipping is at least 2x slower than whatever he uses for his C code.

I confirmed this by timing this code:

function bar(path)
	stream = GzipDecompressorStream(open(path))
	v = read(stream)
	close(stream)
	return length(v)
end

On a gzipped FASTQ file and comparing it with $ gunzip -dc my_file.fq.gz > test. Whereas the Julia code took around 2.1 seconds, the Bash code took 0.6 seconds. Profiling confirms that nearly 100% of the time is spent in the ccall to libz. Hence, it appears to be a problem with the binary itself.

Jaakko Ruohio on the Julia Slack suggested this might have something to do with how the Libz artifact is compiled. Perhaps it's missing some compile flags which makes it poorly optimized?

Update: JuliaPackaging/Yggdrasil#1051 sped up the Julia code by 2x, but it remains around 2x slower than both gunzip and C code that calls into zlib.h.

[jakobnissen]

The issue seems to be that the upstream implementation of zlib shipped as an artifact with CodecZlib is not as optimized as the zlib shipped with MacOS.

[bicycle1885]

I found a related topic on StackOverflow:
https://stackoverflow.com/questions/51789013/zlib-performance-difference-between-mac-os-x-system-version-and-locally-re-insta

The point is that macOS seems to be using a custom implementation for their zlib, which is significantly faster than a locally-compiled version of zlib from the latest release. I have no idea, but Mark Adler says so.

[jakobnissen]

Right, so the question is: Are there well tested, well known, free versions of libz available that we can just swap with, or would that create a hassle?

Edit: See this page:
http://www.htslib.org/benchmarks/zlib.html

Libdeflate is fastest, but not suitable as a generic replacement since it can't stream (right?). Cloudflare has a high-performance fork of libz that could be used.

Edit2: @bicycle1885 BGZFstreams could use libdeflate for significant speed increase, it seems samtools uses that internally, since that format never needs streaming and all chunks can be deflated in memory. Just if it's any interest

[bicycle1885]

And yet another alternative: https://github.com/zlib-ng/zlib-ng

All these forks and independent implementations are not widely used AFAIK and thus I'm reluctant to incorporate one of them as the default of CodecZlib.jl. Yes, 2x slowdown is bad, but not so bad to take a risk of doing harm to innocent users. Perhaps, a better approach is to fork this package and provide the drop-in-replacement API for performance enthusiasts who understand what they do.

[bicycle1885]

BGZFstreams could use libdeflate for significant speed increase, it seems samtools uses that internally, since that format never needs streaming and all chunks can be deflated in memory. Just if it's any interest

Yes, I'm interested in the idea. Also, I'd like to make BGZFStreams.jl or its fork a member of the TranscodingStreams.jl family so that we can use consistent APIs in all packages. In addition, I'm interested in parallelizing BGZF decompression because we know decompression of BGZF blocks is trivially parallelizable and the performance benefit is significant thanks to the parallel task runtime since Julia 1.3.

[jakobnissen]

Okay, it seems like the issue for this package is closed. CodecZlib should keep using zlib, but BGZFstreams could potentially use libdeflate. I've made an issue in that repo.