FR: Support for LZ4?

[gcflymoto]

First of all thank you for such an excellent library which provides our engineering community with simple, transparent, performant, and pythonic library to handle all kinds of compressed and uncompressed files. It reduces the complexity by 10x of handling all these formats.

We've recently been deploying LZ4, especially since the latest release 1.10.0 supports multi-threaded compression and decompression

Unfortunately https://github.com/python-lz4/python-lz4 development seems to have ceased.

Is there any possibility for adding python-zstandard style library? or support for invoking "lz4" CLI for now?

[marcelm]

Thanks for the kind words!

My first thought is that adding LZ4 sounds reasonable. Are you interested in submitting a PR? A couple of thoughts in random order:

• Spawning lz4 as a process would be easiest for now.
• Using python-lz4 as a library should also be needed, but the question is whether it should be a required or optional dependency. I don’t like that the lz4 wheel has 1.3 MB. This seems to be due to debugging info being included and could be reduced to around 100 kB. The Conda packages for lz4 itself plus the Python bindings (lz4-c and lz4) total <200 kB.
• python-lz4 apparently not being maintained is a bummer.
• Maybe the tests need some refactoring to avoid running too many of them?

@rhpvorderman What do you think?

[gnzsnz]

this is another vote for lz4. by the way python-lz4 latest release is from jan-2025

[marcelm]

Ah, great to hear it’s maintained. I’d be happy to have lz4 support, but don’t have time myself to work on this at the moment. So the answer at the moment is: PRs welcome!

[rhpvorderman]

Yes, I am also in favour of adding LZ4. The python package not being ideal is not that bad as long as the conda package are good in terms of size. Those affect biocontainer sizes. 1.3MB is doable on most internet connections.