Add spegel distributed registry mirror

[brandond]

Proposed Changes

This embeds spegel, a distributed registry mirror, into the K3s supervisor. In addition to reducing traffic against upstream registries, this also allows airgap images preloaded onto one node to be transparently shared to other nodes as needed.

spegel consists of a registry API backed by the local containerd image store, and a distributed hash table that allows nodes to gossip about which images and blobs they have available.

The local registry mirror is injected into the containerd config as the first mirror endpoint, followed by the user-selected endpoints, and finally the default endpoint. If any node in the cluster has an image, it will be pulled from that node, instead of from the registry or registry mirror.

The embedded registry is enabled at a cluster level via a CLI flag. When enabled, all nodes will listen on port 5001 for P2P traffic, secured by a preshared key. Agents will have a new listener on port 6443 that hosts the registry endpoint. The registry API is served over HTTPS, and requires a valid client certificate for access.

This appears to add ~4.4MB to the size of the K3s release artifact.

Checklist:

• Write up ADR supporting this new feature and get team+PM approval
• Get spegel working embedded in supervisor endpoint
• Add agents to spegel cluster, instead of just proxying back to servers
• Remove hardcoded values and wire into node config
• Add CLI flag to enable, instead of always being enabled
• Require TLS auth for registry endpoint
• Require private network with preshared key
• Add tests.
• Add ipv6 support and push upstream.
  Handled by upstream in Fix support for ipv6 spegel-org/spegel#284
• Push embedding, https, and nonblocking containerd client support upstream
  spegel-org/spegel@main...k3s-io:spegel:k3s-main

Builds on:

• Add support for containerd cri registry config_path #8973

Inspired by:

• Add minimal read only registry support #8554
• docs: Add k3s to the Compatibility matrix spegel-org/spegel#281

Resolves:

• Images imported from airgap tarballs can be deleted by kubelet GC #8781
• Bundle or embed a registry mirror #9026

Use:

1. Start servers with --embedded-registry
2. In order to enable distributed mirroring of an upstream registry, reference it in the mirror section of registries.yaml on every node that you want to participate in the sharing of images. The registry does not need to have any endpoints, although it may. For example, this is a valid configuration:

   mirrors:
     docker.io:
     registry.k8s.io:
     gcr.io:
     quay.io:
     ghcr.io:

3. Nodes also may be started with --disable-default-registry-endpoint, in which case images will only be available via airgap tarball, distributed mirror, or user-configured mirror (in order of precendece)
4. Check metrics:
   kubectl get --raw /api/v1/nodes/NODENAME/proxy/metrics | grep -F 'spegel_

Types of Changes

new feature

Verification

See above

Testing

E2e test added.

Linked Issues

• TBD

User-Facing Change

Further Comments

[phillebaba]

We can check off ipv6 support as I have merged spegel-org/spegel#284.

[harsimranmaan]

Thanks Brandon. I am really happy to see such good progress on this one. I can help test this. If you need support convincing the team or the PMs, you can count my vote (and perhaps @adrianmoye's too) :)

[onedr0p]

It's truly amazing the work you put into integrating spegel @brandond and so quickly, can't wait to try it out. ❤️

[adrianmoye]

Thanks @brandond for the great work! I really appreciate it. This looks like the perfect solution for me, with pinning images (a feature I wasn't aware of) being the final point for my requirements.
@harsimranmaan thanks for ping me on this.

[twistedgrim]

I just learned about Spegel and tried out it out. Freaking amazing work! Looking forward to this!

[rlex]

spegel exposes some useful metrics, will they be exposed in case of running embedded spegel? Cannot find anything related in PR

[onedr0p]

It would be very helpful useful to have those metrics exposed, their grafana dashboard is quite nice for reviewing what's going on.