patches?

[alexmyczko]

do you have patches for linux kernel btrfs (filsystem live compression?)
did you do benchmark to kernel forked lz4 code?

and CONFIG_KERNEL_LZ4=y

and zram?

https://tracker.debian.org/pkg/zxc

[hellobertrand]

Hi! Thanks for your interest and your work on the Debian packaging.

To be honest, I haven't thought about kernel-space integration yet! That's a very interesting perspective.

Currently, zxc is purely a user-space library and CLI tool. I don't have patches ready for the Linux kernel yet (neither for btrfs live compression, nor for zram block devices, nor for the crypto/ kernel API).

A kernel-space port of the zxc compression engine would be a great addition, especially considering the potential gains for zram and btrfs live compression. While this would require adapting the code to Linux kernel constraints, the core C implementation of zxc is already designed to be lightweight and allocation-free by default. I haven't started working on this yet, but it’s a highly valuable prospect and I would be thrilled to see community-driven efforts or experiments in that direction.

[alexmyczko]

i'd guess zramctl (zram) shows lz4 by default for me, most profitable here, together with btrfs live compression (both if used), and last the kernel itself. i'll try to find out where/what and maybe it'll be fun. (btrfs-compsize will be helpful to find out ratio, and other things for benchmark, i hope once compressed with stock kernel lz4 and switching to zxc lz4 will not cause any problem?)

[hellobertrand]

Here are some recent changes that may be relevant to the Debian packaging:

• Man page template added: zxc.1.md is now available in docs/man/.
• Obsolete images removed from docs/
• LICENSE updated: clarified that the embedded rapidhash code is under the MIT (Expat) license
• GNU/Hurd build fix: the multiarch CI build for Hurd should now work correctly

Regarding your earlier question about kernel integration: zxc is a userspace library with its own wire format: it is not a drop-in replacement for lz4. The two formats are incompatible, so you cannot swap one for the other in btrfs or zram. zxc's performance profile (compression 3.5x slower than lz4, decompression 40% faster, 2.5% better ratios) makes it better suited for read-heavy/store-once-read-many scenarios rather than kernel hot paths where compression speed is critical.

Let me know if you need anything else for the packaging.

[hellobertrand]

FYI: v0.8.0 is a breaking changes file format. This v0.8.0 will likely be the last major format shift before v1.0.0, where the format will be officially frozen. Since the package is currently in unstable/proposed, I think it’s nice to push these changes before it reaches testing/stable.

[hellobertrand]

Oops! I completely forgot to bump the SOVERSION in my CMakeLists.txt during the v0.8.0 release, even though there was a significant ABI/API break in the C library...

To prevent silently breaking existing dynamically linked packages, I just pushed a hotfix release: v0.8.1.

[alexmyczko]

https://buildd.debian.org/status/package.php?p=zxc&suite=sid

[hellobertrand]

nice!

[alexmyczko]

so for write once, read many, this would be right: CONFIG_KERNEL_LZ4=y

[hellobertrand]

Thx for the suggestion and the ideas around kernel integration. For worm workloads, I currently see 3 scenarios (before considering kernel integration) where zxc could make sense:

• squashfs images
• initramfs
• OCI containers where layers are decompressed frequently

What do you think about?

Over the next few days I’ll try to run more realistic benchmarks, especially around squashfs, comparing gzip, lz4, zstd, and zxc.

[alexmyczko]

are you aware of dwarfs?

initramfs as in https://github.com/alexmyczko/autoexec.bat/blob/master/config.sys/debian-system-initramfs

i think yes your three suggestions, oci (not that i need or use this a lot)

maybe if i find time, i try to patch and test the linux kernel thing and report how it went?

[hellobertrand]

I have implemented a prototype in the add-zxc branch of my fork: https://github.com/hellobertrand/squashfs-tools/tree/add-zxc), and I have conducted a series of benchmarks across three major architectures (Apple M3, AMD Turin, and Google Axion) using different datasets to evaluate zxc against lz4 within squashfs. I used squashfs-tools/squashfs-tools/bench.sh on 3 differents corpus: some logs (374MB), random data (not compressible 257MB) and linux kernel (1.7GB).

Architecture	Dataset	Algorithm	Comp. Time	Decomp. Time (Best)	Archive Size	Ratio (%)
Apple M3 (arm4)	linux	lz4	5.371s	6.949s	415M	23.9%
		zxc	4.868s	6.968s	374M	21.5%
Apple M3 (arm64)	logs	lz4	0.199s	0.223s	309M	82.4%
		zxc	0.941s	0.093s	296M	78.9%
Apple M3 (arm64)	random	lz4	0.114s	0.110s	257M	100.0%
		zxc	0.089s	0.069s	257M	100.0%
AMD Turin (x86_64)	linux	lz4	2.561s	1.003s	415M	23.9%
		zxc	5.248s	0.993s	374M	21.5%
AMD Turin (x86_64)	logs	lz4	0.463s	0.165s	309M	82.4%
		zxc	2.366s	0.157s	296M	78.9%
AMD Turin (x86_64)	random	lz4	0.141s	0.086s	257M	100.0%
		zxc	0.173s	0.087s	257M	100.0%
Google Axion (arm64)	linux	lz4	12.404s	2.216s	415M	23.9%
		zxc	10.994s	2.036s	374M	21.5%
Google Axion (arm64)	logs	lz4	1.856s	0.249s	309M	82.4%
		zxc	3.797s	0.235s	296M	78.9%
Google Axion (arm64)	random	lz4	1.527s	0.136s	257M	100.0%
		zxc	0.273s	0.129s	257M	100.0%