docs: Add k3s to the Compatibility matrix

[onedr0p]

Add k3s nuances to Compatibility matrix

Fixes #212

[phillebaba]

There is currently a PR underway that would make k3s support a lot simpler.

k3s-io/k3s#8973

Lets hold off on adding k3s to the matrix until it is released.

[onedr0p]

Oh wow that's fresh off the block, thanks for linking.

[brandond]

I have a secret plan to embed spegel in the k3s supervisor, to allow for sharing the preloaded airgap tarballs across the cluster using spegel as an embedded distributed registry.

I don't know if it'll go anywhere (I haven't talked to product management or anyone else on the team), but I'd love to make it work. Adding hosts.toml support is the first step in making that possible.

[onedr0p]

That would be amazing, hopefully you can get approval and time to work on it. I know Talos is working on something similar to have this built into their distro too.

[phillebaba]

@brandond we should talk. While I do not have this use case it is one that I have discussed with others. In theory I see no reason why this would not work, but it would be very interesting to see how it would perform in the real world.

[brandond]

@phillebaba the biggest hurdle at the moment is the fact that all of your packages are internal, and there is no public interface exposed to start up the various components in an embedding application.

K3s likes to embed components and run them in its main process - we successfully do this for etcd and all the various Kubernetes components, for example. Unfortunately, I can't take the bits from I need from main.go and embed them directly in k3s because of the internal package restrictions enforced by go - so my first move (at the moment) would involve forking this repo and renaming all the packages.

https://github.com/brandond/spegel/commits/main and https://github.com/brandond/k3s/commits/embed-spegel - although I literally just started tinkering with it this afternoon.

[brandond]

@phillebaba do y'all have a slack channel workspace, or can I find you on Kubernetes slack?

I have some questions about the DHT lookups; I'm seeing that they just don't work at all due to a defect in the async provider lookup, but I want to make sure I'm not doing something wrong first.

INFO[0393] spegel 2023/12/01 18:27:17 "level"=0 "msg"="handling mirror request from external node" "key"="docker.io/rancher/mirrored-pause:3.6" "path"="/v2/rancher/mirrored-pause/manifests/3.6" "ip"="172.17.0.1"
INFO[0393] spegel 2023/12/01 18:27:17 "level"=10 "msg"="resolving key" "key"="docker.io/rancher/mirrored-pause:3.6" "path"="/v2/rancher/mirrored-pause/manifests/3.6" "ip"="172.17.0.1" "host"="12D3KooWBQpJzRMuWz37aNW93ZRjSdh5Tc6GDY7XDpgLZNAi2eGF" "key"="docker.io/rancher/mirrored-pause:3.6" "allowSelf"=true "count"=0
2023-12-01T18:27:17.781Z	DEBUG	dht	go-libp2p-kad-dht@v0.25.2/routing.go:510	finding providers	{"cid": "bafkreig5lsn4dh3jstov5pvfqvqqjhjp2yq26zry5rk2maeqqxt5efsgee", "mh": "bciqn2xe3ygpwtfg5l27klblbasos7vrbv5tdr3cvuyajbbph2ilemii"}
INFO[0393] spegel 2023/12/01 18:27:17 "level"=10 "msg"="got provider" "key"="docker.io/rancher/mirrored-pause:3.6" "path"="/v2/rancher/mirrored-pause/manifests/3.6" "ip"="172.17.0.1" "host"="12D3KooWBQpJzRMuWz37aNW93ZRjSdh5Tc6GDY7XDpgLZNAi2eGF" "key"="docker.io/rancher/mirrored-pause:3.6" "peer"="{12D3KooWBQpJzRMuWz37aNW93ZRjSdh5Tc6GDY7XDpgLZNAi2eGF: [/ip4/172.17.0.7/tcp/5001]}"
INFO[0393] spegel 2023/12/01 18:27:17 "level"=10 "msg"="peer resolved" "key"="docker.io/rancher/mirrored-pause:3.6" "path"="/v2/rancher/mirrored-pause/manifests/3.6" "ip"="172.17.0.1" "mirror"="https://172.17.0.7:6443"
2023-12-01T18:27:17.782Z	DEBUG	dht	go-libp2p-kad-dht@v0.25.2/routing.go:581	1 provider entries
2023-12-01T18:27:17.782Z	DEBUG	dht	go-libp2p-kad-dht@v0.25.2/routing.go:586	got provider: {12D3KooWHGAstEM3bLAzGu4Z2KtADt2nKcp6ir7WCtN1TUHQuUBf: [/ip4/172.17.0.8/tcp/5001]}
2023-12-01T18:27:17.782Z	DEBUG	dht	go-libp2p-kad-dht@v0.25.2/routing.go:588	using provider: {12D3KooWHGAstEM3bLAzGu4Z2KtADt2nKcp6ir7WCtN1TUHQuUBf: [/ip4/172.17.0.8/tcp/5001]}
2023-12-01T18:27:17.782Z	DEBUG	dht	go-libp2p-kad-dht@v0.25.2/routing.go:607	got closer peers: 0 []
2023-12-01T18:27:17.782Z	DEBUG	dht	go-libp2p-kad-dht@v0.25.2/dht.go:707	peer found	{"peer": "12D3KooWHGAstEM3bLAzGu4Z2KtADt2nKcp6ir7WCtN1TUHQuUBf"}
INFO[0393] spegel 2023/12/01 18:27:17 "level"=10 "msg"="got provider" "key"="docker.io/rancher/mirrored-pause:3.6" "path"="/v2/rancher/mirrored-pause/manifests/3.6" "ip"="172.17.0.1" "host"="12D3KooWBQpJzRMuWz37aNW93ZRjSdh5Tc6GDY7XDpgLZNAi2eGF" "key"="docker.io/rancher/mirrored-pause:3.6" "peer"="{12D3KooWHGAstEM3bLAzGu4Z2KtADt2nKcp6ir7WCtN1TUHQuUBf: [/ip4/172.17.0.8/tcp/5001]}"
INFO[0393] spegel 2023/12/01 18:27:17 "level"=10 "msg"="peer resolved" "key"="docker.io/rancher/mirrored-pause:3.6" "path"="/v2/rancher/mirrored-pause/manifests/3.6" "ip"="172.17.0.1" "mirror"="https://172.17.0.8:6443"
INFO[0393] spegel 2023/12/01 18:27:17 "level"=10 "msg"="provider address channel closed" "key"="docker.io/rancher/mirrored-pause:3.6" "path"="/v2/rancher/mirrored-pause/manifests/3.6" "ip"="172.17.0.1" "host"="12D3KooWBQpJzRMuWz37aNW93ZRjSdh5Tc6GDY7XDpgLZNAi2eGF" "key"="docker.io/rancher/mirrored-pause:3.6"
INFO[0393] spegel 2023/12/01 18:27:17 "level"=10 "msg"="peer resolved" "key"="docker.io/rancher/mirrored-pause:3.6" "path"="/v2/rancher/mirrored-pause/manifests/3.6" "ip"="172.17.0.1" "mirror"=""
INFO[0393] spegel 2023/12/01 18:27:17 "level"=10 "msg"="mirror peer channel closed" "key"="docker.io/rancher/mirrored-pause:3.6" "path"="/v2/rancher/mirrored-pause/manifests/3.6" "ip"="172.17.0.1"
INFO[0393] spegel 2023/12/01 18:27:17 "msg"="" "error"="mirror resolve retries exhausted for key: docker.io/rancher/mirrored-pause:3.6" "path"="/v2/rancher/mirrored-pause/manifests/3.6" "status"=500 "method"="GET" "latency"="2.987583ms" "ip"="172.17.0.1" "handler"="mirror"

Edit: nevermind, the proxy's noop error handler was silently swallowing a TLS error. If I'm going to use https I need to pass through a transport with the CAs set up properly.

[phillebaba]

@brandond i messages you on the cncf slack. You can also find me on the Kubernetes slack if that's easier.