Support non-compressed/foreign image layer media types

[rvmiller89]

#261 added support for OCI Image Layout, but only supports Image Layers of two media types:

• application/vnd.oci.image.layer.v1.tar+gzip
• application/vnd.docker.image.rootfs.diff.tar.gzip

For any other value of media type it returns an error (see the TODO's):

go-containerregistry/pkg/v1/layout/image.go

Lines 96 to 106 in 34b7f00

	switch desc.MediaType {
	case types.OCILayer, types.DockerLayer:
	return &compressedBlob{
	path: li.path,
	desc: desc,
	}, nil
	default:
	// TODO: We assume everything is a compressed blob, but that might not be true.
	// TODO: Handle foreign layers.
	return nil, fmt.Errorf("unexpected media type: %v for layer: %v", desc.MediaType, desc.Digest)
	}

Issue 1. Supporting non-compressed layers

The OCI Image Manifest specification requires support for at least 4 types, including 2 non-compressed layer types (.tar):

• application/vnd.oci.image.layer.v1.tar
• application/vnd.oci.image.layer.v1.tar+gzip
• application/vnd.oci.image.layer.nondistributable.v1.tar
• application/vnd.oci.image.layer.nondistributable.v1.tar+gzip

Issue 2. Supporting foreign {compressed, non-compressed} layers

The OCI Artifacts project defines additional media types like Helm Charts and custom media types. Examples include:

• application/vnd.cncf.helm.chart.layer.v1.tar+gzip
• application/tar+gzip
• etc.

Possible work-arounds

1. Allow any media type: I verified removing the explicit check for types.OCILayer, types.DockerLayer allows us to push layers of arbitrary media types, but wasn't sure how this might conflict with compressedBlob.
2. Allow any media type ending in +gzip: (Image Manifest specification mentions "Entries in this field will frequently use the +gzip types.")
3. Something else?

Happy to open a PR to support this, but wanted to check how we might want to handle it.

[jonjohnsonjr]

In general, there's a lot I want to do with the layout package, I just haven't had the time. I welcome any discussion and feedback around what you're trying to do, but some more context here:

• Improve append command - ease the process to push manifests to remote registries from a set of images #891
• Feat(crane) support push tarball contains more than one image #1110
• Add support for OCI layouts in crane pull/push #1116

I think some of these might merit new commands in crane, but others could just be additional flags (if crane matters to you at all).

The OCI Artifacts project defines additional media types like Helm Charts and custom media types

In this case, the Compressed layer accessor is just a terrible name because this library was geared towards images more than anything else. It would be more accurate to call this Original or something.

Definitely open to better logic around producing an io.ReadCloser based on the layer's media type in general.

Supporting foreign

ACK, we need something like

go-containerregistry/pkg/v1/remote/image.go

Lines 175 to 194 in 308547a

	for _, u := range urls {
	req, err := http.NewRequest(http.MethodGet, u.String(), nil)
	if err != nil {
	return nil, err
	}
	resp, err := rl.ri.Client.Do(req.WithContext(ctx))
	if err != nil {
	lastErr = err
	continue
	}
	if err := transport.CheckError(resp, http.StatusOK); err != nil {
	resp.Body.Close()
	lastErr = err
	continue
	}
	return verify.ReadCloser(resp.Body, d.Size, rl.digest)
	}

inside the layout package. Ideally, we wouldn't have layout to take a dependency on remote or transport, so there might be some duplicated logic here. Unfortunately, plumbing options around becomes a little annoying (e.g. do we need a layout.WithTransport option?).

1. Allow any media type
   Happy to open a PR to support this

I don't see any harm in trying this. If it works for your use case, I'm ok with it. Feel free to send a PR.

The only issue is if you store an uncompressed blob in the layout and want to compress it before pushing to the registry, you're out of luck. I would say this is an astonishingly unlikely use case because we generally want to preserve the digests of the thing we're dealing with.

We may want to just make the Uncompressed accessor here do the right thing based on media types? Annoyingly, you'll want to look for mediaTypes with suffix +gzip and .gzip, as the docker media types have a layer as application/vnd.docker.image.rootfs.diff.tar.gzip.

While you're in there, you might want to make sure layout.compressedBlob implements partial.withDescriptor and partial.WithDiffID.

[jonjohnsonjr]

Does #1136 solve your issue or is there more to do here?

Edit: just realized lack of foreign layer support might be what's missing?

[rvmiller89]

We're unblocked, and currently aren't using any foreign layers. Not sure if this is a big deal for others.

I'm happy with closing this for now.