Add minimal read only registry support

[adrianmoye]

Proposed Changes

Adds a read-only docker image registry to the K3s server, this improves the experience for air-gapped K3s users and removes the need to pre-load containerd with images from the agent/images/ directory.

Types of Changes

New feature

Verification

Populate the /var/lib/rancher/k3s/server/registry/ directory with a container image and pull the container from the server.
and manually:

$ sudo mkdir -p /var/lib/rancher/k3s/server/registry/test/manifests
$ echo '{"mediaType":"Pass"}' | sudo tee /var/lib/rancher/k3s/server/registry/test/manifests/test
$ curl -kv https://127.0.0.1:6443/v2/test/manifests/test 2>&1 | grep -i "Content-Type: Pass" && echo OK || echo FAIL
< content-type: Pass
OK
$ echo HERE | sudo tee /var/lib/rancher/k3s/server/registry/blob
$ curl -kv https://127.0.0.1:6443/v2/blob -H "Accept: magic" 2>&1 | grep -i "Content-Type: magic" && echo OK || echo FAIL
< content-type: magic
OK

Testing

None

Linked Issues

User-Facing Change

NONE

Further Comments

This is designed to improve the experience bootstrapping an air gapped cluster, during both the initial install and reboots.
If the cluster is self hosted, you need to bootstrap a registry and the backing store, this solves that problem without having to load all of the artefacts at every K3s restart.

The registry is very simple, and was based of the /static/ path route. Each repo requires two directories: manifests and blobs, the manifests are JSON files with .mediaType specifying the Content-Type, so the patch reads the manifests and provides the correct Content-Type header. Otherwise it tries to respond with the Accept header.
Reading the entire manifest file could be a problem, however the cluster operator is responsible for the content of that directory.

This requires some documentation to be useful, and a helper script to populate the registry, I've created a stand alone example, documentation and a registry population script here: https://github.com/adrianmoye/minimal-registry/ I'm not sure what to do with this.

[brandond]

Thanks for the PR! This is an interesting way to wire up serving registry content. We can talk this over as a team, but I believe our product management folks have some long-term goals around built-in registry support that we need to do a better job solidifying and addressing before we ship any registry functionality in the project.

[harsimranmaan]

Thanks @adrianmoye . @brandond, can you share some insights into the full registry requirements from the product team? I am interested in this feature and would be happy to contribute if the requirements are laid out and a design is agreed upon.

I am interested in the following use case

1. Airgap - as @adrianmoye proposes
2. Pull through cache - Proxy all cluster image downloads through the server(s).

[harsimranmaan]

@brandond would you be able to share the specs for the desired registry. I am keen on getting this feature. Happy to collaborate on it

[brandond]

I would love to see something like this embedded in K3s, but I suspect it's not lightweight enough:
https://github.com/XenitAB/spegel

I am leaning on my manager and product management team to provide some better requirements before we move forward on this.

[caroline-suse-rancher]

Thank you for your contribution @adrianmoye ! Ultimately, we are choosing to move forward with our investigation into Spegel as an alternative registry, so I'm going to close this PR.