compression: add zstd:chunked compression type#6528
Open
JakeCooper wants to merge 1 commit intomoby:masterfrom
Open
compression: add zstd:chunked compression type#6528JakeCooper wants to merge 1 commit intomoby:masterfrom
JakeCooper wants to merge 1 commit intomoby:masterfrom
Conversation
Add support for zstd:chunked as a compression option for image layer export. When specified via `compression=zstd:chunked`, each file in the tar stream is compressed as an independent zstd frame with a table of contents (TOC) appended as a zstd skippable frame. This enables partial/lazy pulling by consumers that understand the format (e.g. podman, containers/storage) while remaining fully backwards- compatible with standard zstd decompression. The implementation wraps the well-tested compressor from containers/storage/pkg/chunked/compressor, which is explicitly designed for standalone use by external callers. Key design decisions: - Uses the same OCI media type as zstd (backwards-compatible) - Decompression reuses existing zstd decompressor - Skips conversion if layer is already zstd-compressed Related: moby#2345 Signed-off-by: Jake Cooper <jake@railway.com>
crazy-max
reviewed
Feb 20, 2026
|
|
||
| "github.com/containerd/containerd/v2/core/content" | ||
| "github.com/containerd/containerd/v2/core/images" | ||
| chunkedcompressor "github.com/containers/storage/pkg/chunked/compressor" |
Member
There was a problem hiding this comment.
I don't really like we consume an unrelated package just for zstd compression for a module not directly aimed at compression. If anything I think it should be either vendored here with tests or use a proper module. Also github.com/containers/storage is deprecated and has moved to https://github.com/containers/container-libs/tree/main/storage.
Member
There was a problem hiding this comment.
Also vendor is not updated in your PR (go mod vendor).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add support for
zstd:chunkedas a compression option for image layer export. This enables partial/lazy pulling by consumers that understand the format (e.g. podman, containers/storage).When specified via
--output "compression=zstd:chunked", each file in the tar stream is compressed as an independent zstd frame with a table of contents (TOC) appended as a zstd skippable frame.Format
The output blob structure follows the containers/storage zstd:chunked spec:
Where each
[FILE_N]is an independently compressed zstd frame containing the tar header + payload for that file, and the TOC contains file metadata (name, size, offset, digest) enabling consumers to fetch individual files without downloading the entire layer.Key design decisions
application/vnd.oci.image.layer.v1.tar+zstd) — fully backwards-compatible. Registries and runtimes that don't understand zstd:chunked simply decompress it as a regular zstd blob.containers/storage/pkg/chunked/compressoris explicitly designed for standalone use by external callers and is well-tested in production by podman and buildah.Usage
Impact
Changes
util/compression/zstdchunked.go(new) — implements theTypeinterfaceutil/compression/compression.go— registers the new type and adds it to the parserNew dependency
github.com/containers/storage/pkg/chunked/compressor— lightweight, standalone package. Most of its transitive deps (klauspost/compress/zstd,opencontainers/go-digest) are already vendored in BuildKit.Related: #2345