compression: register docker zstd stream processor

[sipsma]

Before this, buildkit was able to create and push layers of type vnd.docker.image.rootfs.diff.tar.zstd but if you tried to then pull and unpack those layers in buildkit, you'd get an error from containerd: failed to get stream processor for application/vnd.docker.image.rootfs.diff.tar.zstd: no processor for media-type

While that media type is not official, support for exporting it was added to buildkit in the past anyways since it works in practice and is accepted by many registries. It thus seems logical that buildkit should also support pulling and unpacking those layers too.

The fix works by just registering a custom stream processor w/ containerd.

There is support for registering stream processors in containerd, but
those only work when using the OCI worker since it relies on the apply
implementation being in the same process as buildkitd.

As a fallback, we instead just implement a hack that swaps out the
docker zstd media type for the oci one before sending it to containerd.
This works in practice because the two types are compatible.

---

Ran into problems with this in the real world with dagger here

[sipsma]

I just saw that the test only passes when using the OCI worker, fails on containerd worker. I'm guessing that's because when we use the containerd worker the stream processor is not in the buildkitd process but instead in containerd...

I'll poke around for ways of fixing, but open to suggestions.

---

EDIT:

I pushed what appears to be a fix by just changing the media type of docker zstd layers to be the oci equivalent before passing to the applier. This feels like a pretty dumb hack but as far as I can tell it should work consistently since the two types are fully compatible to my knowledge.

It also feels hacky since the code is in the winlayers package, but it seems like that's where we are handling all of our Applier-wrapping since there's also some nydus stuff in there too?

cc @tonistiigi @AkihiroSuda @ktock in case there's any other ideas or if you think this approach is okay.

[tonistiigi]

We have existing zstd tests like testPullZstdImage, testZstdRegistryCacheImportExport. What's the difference?

[tonistiigi]

This doesn't look specific to winApplier?

[tonistiigi]

Squashed this as all the stream processor code has been reverted in a later commit.

[sipsma]

Squashed this as all the stream processor code has been reverted in a later commit.

Thanks @tonistiigi! To answer your previous questions:

We have existing zstd tests like testPullZstdImage, testZstdRegistryCacheImportExport. What's the difference?

The previous testPullZstdImage test specifically forced oci types. I just updated it to cover both oci and docker types and also have more assertions on the correct media type being produced.

This doesn't look specific to winApplier?

I agree but I saw that we were also doing some nydus-specific handling here that didn't seem related to Windows, so I figured it was just our generic wrapper for any custom Applier logic.

Maybe it should just be renamed? Or I can do a follow up to split this into several different wrappers for windows, nydus and this zstd handling. Let me know if you think it matters.

[jonjohnsonjr]

Shouldn't this all be +zstd not .zstd?

[tonistiigi]

@jonjohnsonjr No, this is based on Docker mediatypes distribution/distribution@2ff77c0#diff-dcad444e0ff28a727a091ec0e27cf747e488d4433277c7c86d792fe8296bd84eR22 containerd/containerd@5a3151e#diff-054c8a3671a5a631a918aa1b64f25fe3ad640f0c55d554945df8775dcb51f64fR17 . This is different from OCI mediatypes that use + suffixes.