Skip to content

Bundle compression/decompression takes too much time #1340

Closed

Description

The crcbundle is currently compressed using xz. This format is not natively supported by Golang and we have to use github.com/xi2/xz library.

On my machine, it takes 3m47s to decompress a bundle using xi2/xz. With xz command line tool, it takes 2m20s.
xi2/xz documentation says:

// On the author's Intel Ivybridge i5, decompression speed is about
// half that of the standard XZ Utils (tested with a recent linux
// kernel tarball).

I also tried with https://github.com/ulikunitz/xz and it doesn't finish in less than 5min.

Given that the compression step in snc takes also a huge amount of time (about 30min by bundle if I'm correct), maybe we should investigate an other format.

crc_libvirt_4.4.6.crcbundle
xz: 2.0GB
zip (default level): 2.8GB
gzip (default level): 2.8GB
zstd: 2.6GB
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    kind/enhancementNew feature or requeststatus/pinnedPrevents the stale bot from closing the issue

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions