Add docs for embedded registry mirror

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>

Author: brandond

docs/cli/agent.md

@@ -79,6 +79,7 @@ Options are documented on this page as CLI flags, but can also be passed as conf
 | `--rootless` | Run rootless                          |
 | `--docker`   | Use cri-dockerd instead of containerd |
 | `--prefer-bundled-bin` | Prefer bundled userspace binaries over host binaries |
+| `--disable-default-registry-endpoint` | See "[Default Endpoint Fallback](../installation/private-registry.md#default-endpoint-fallback)"
 
 ### Deprecated
 

docs/cli/server.md

@@ -24,6 +24,7 @@ The following options must be set to the same value on all servers in the cluste
 * `--disable-network-policy`
 * `--disable=servicelb` *note: other packaged components may be disabled on a per-server basis*
 * `--egress-selector-mode`
+* `--embedded-registry`
 * `--flannel-backend`
 * `--flannel-external-ip`
 * `--flannel-ipv6-masq`
@@ -170,6 +171,7 @@ The following options must be set to the same value on all servers in the cluste
 | `--docker`             | Use cri-dockerd instead of containerd    |
 | `--prefer-bundled-bin` | Prefer bundled userspace binaries over host binaries |
 | `--disable-agent`      | See "[Running Agentless Servers (Experimental)](../advanced.md#running-agentless-servers-experimental)" |
+| `--embedded-registry`  | See "[Embedded Registry Mirror](../installation/registry-mirror.md)" |
 
 
 ### Deprecated Options

docs/installation/airgap.md

@@ -5,35 +5,54 @@ weight: 60
 
 You can install K3s in an air-gapped environment using two different methods. An air-gapped environment is any environment that is not directly connected to the Internet. You can either deploy a private registry and mirror docker.io, or you can manually deploy images such as for small clusters.
 
-## Private Registry Method
+## Load Images
 
-This document assumes you have already created your nodes in your air-gap environment and have a Docker private registry on your bastion host.
+### Private Registry Method
 
-If you have not yet set up a private Docker registry, refer to the official documentation [here](https://docs.docker.com/registry/deploying/#run-an-externally-accessible-registry).
+These steps assume you have already created nodes in your air-gap environment,
+are using the bundled containerd as the container runtime,
+and have a OCI-compliant private registry available in your environment.
 
-### Create the Registry YAML
+If you have not yet set up a private Docker registry, refer to the [official Registry documentation](https://docs.docker.com/registry/deploying/#run-an-externally-accessible-registry).
 
-Follow the [Private Registry Configuration](private-registry.md) guide to create and configure the registry.yaml file.
+#### Create the Registry YAML and Push Images
 
-Once you have completed this, you may now go to the [Install K3s](#install-k3s) section below.
+1. Obtain the images archive for your architecture from the [releases](https://github.com/k3s-io/k3s/releases) page for the version of K3s you will be running.
+2. Use `docker image load k3s-airgap-images-amd64.tar.zst` to import images from the tar file into docker.
+3. Use `docker tag` and `docker push` to retag and push the loaded images to your private registry.
+4. Follow the [Private Registry Configuration](private-registry.md) guide to create and configure the `registries.yaml` file.
+5. Proceed to the [Install K3s](#install-k3s) section below.
 
+### Manually Deploy Images Method
 
-## Manually Deploy Images Method
+These steps assume you have already created nodes in your air-gap environment,
+are using the bundled containerd as the container runtime,
+and cannot or do not want to use a private registry.
 
-We are assuming you have created your nodes in your air-gap environment and use containerd as container runtime.
-This method requires you to manually deploy the necessary images to each node and is appropriate for edge deployments where running a private registry is not practical.
+This method requires you to manually deploy the necessary images to each node, and is appropriate for edge deployments where running a private registry is not practical.
 
-### Prepare the Images Directory and K3s Binary
-Obtain the images tar file for your architecture from the [releases](https://github.com/k3s-io/k3s/releases) page for the version of K3s you will be running.
+#### Prepare the Images Directory and Airgap Image Tarball
 
-Place the tar file in the `images` directory, for example:
+1. Obtain the images archive for your architecture from the [releases](https://github.com/k3s-io/k3s/releases) page for the version of K3s you will be running.
+2. Download the imagess archive to the agent's images directory, for example:
+  ```bash
+  sudo mkdir -p /var/lib/rancher/k3s/agent/images/
+  sudo curl -L -O /var/lib/rancher/k3s/agent/images/k3s-airgap-images-amd64.tar.zst https://github.com/k3s-io/k3s/releases/download/v1.29.1-rc2%2Bk3s1/k3s-airgap-images-amd64.tar.zst
+  ```
+3. Proceed to the [Install K3s](#install-k3s) section below.
 
-```bash
-sudo mkdir -p /var/lib/rancher/k3s/agent/images/
-sudo cp ./k3s-airgap-images-$ARCH.tar /var/lib/rancher/k3s/agent/images/
-```
+### Embedded Registry Mirror
+
+:::info Version Gate
+The Embedded Registry Mirror is available as an experimental feature as of January 2024 releases: v1.26.13+k3s1, v1.27.10+k3s1, v1.28.6+k3s1, v1.29.1+k3s1
+:::
+
+K3s includes an embedded distributed OCI-compliant registry mirror.
+When enabled and properly configured, images available in the containerd image store on any node
+can be pulled by other cluster members without access to an external image registry.
 
-Once you have completed this, you may now go to the [Install K3s](#install-k3s) section below.
+The mirrored images may be sourced from an upstream registry, registry mirror, or airgap image tarball.
+For more information on enabling the embedded distributed registry mirror, see the [Embedded Registry Mirror](./registry-mirror.md) documentation.
 
 ## Install K3s
 
@@ -119,7 +138,7 @@ K3S_DATASTORE_ENDPOINT='mysql://username:password@tcp(hostname:3306)/database-na
 </Tabs>
 
 :::note
-K3s additionally provides a `--resolv-conf` flag for kubelets, which may help with configuring DNS in air-gap networks.
+K3s's `--resolv-conf` flag is passed through to the kubelet, which may help with configuring pod DNS resolution in air-gap networks where the host does not have upstream nameservers configured.
 :::
 
 ## Upgrading

docs/installation/installation.md

@@ -9,8 +9,14 @@ This section contains instructions for installing K3s in various environments. P
 
 [Network Options](network-options.md) provides guidance on the networking options available in k3s.
 
-[Air-Gap Installation](airgap.md) details how to set up K3s in environments that do not have direct access to the Internet.
+[Private Registry Configuration](private-registry.md) covers use of `registries.yaml` to configure container image registry mirrors.
+
+[Embedded Mirror](registry-mirror.md) shows how to enable the embedded distributed image registry mirror.
+
+[Air-Gap Install](airgap.md) details how to set up K3s in environments that do not have direct access to the Internet.
 
 [Managing Server Roles](server-roles.md) details how to set up K3s with dedicated `control-plane` or `etcd` servers.
 
 [Managing Packaged Components](packaged-components.md) details how to disable packaged components, or install your own using auto-deploying manifests.
+
+[Uninstalling K3s](uninstall.md) details how to remove K3s from a host.

docs/installation/network-options.md

@@ -35,11 +35,11 @@ K3s no longer includes strongSwan `swanctl` and `charon` binaries starting with
 
 :::
 
-#### Migrating from `wireguard` or `ipsec` to `wireguard-native`
+### Migrating from `wireguard` or `ipsec` to `wireguard-native`
 
-The legacy `wireguard` backend requires installation of the `wg` tool on the host. This backend will be removed in K3s v1.26, in favor of `wireguard-native` backend, which directly interfaces with the kernel.
+The legacy `wireguard` backend requires installation of the `wg` tool on the host. This backend is not available in K3s v1.26 and higher, in favor of `wireguard-native` backend, which directly interfaces with the kernel.
 
-The legacy `ipsec` backend requires installation of the `swanctl` and `charon` binaries on the host. This backend will be removed in K3s v1.27, in favor of the `wireguard-native` backend.
+The legacy `ipsec` backend requires installation of the `swanctl` and `charon` binaries on the host. This backend is not available in K3s v1.27 and higher, in favor of the `wireguard-native` backend.
 
 We recommend that users migrate to the new backend as soon as possible. The migration requires a short period of downtime while nodes come up with the new configuration. You should follow these two steps:
 
@@ -120,14 +120,16 @@ K3s agents and servers maintain websocket tunnels between nodes that are used to
 This allows agents to operate without exposing the kubelet and container runtime streaming ports to incoming connections, and for the control-plane to connect to cluster services when operating with the agent disabled.
 This functionality is equivalent to the [Konnectivity](https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/) service commonly used on other Kubernetes distributions, and is managed via the apiserver's egress selector configuration.
 
+The default mode is `agent`. `pod` or `cluster` modes are recommended when running [agentless servers](../advanced.md#running-agentless-servers-experimental), in order to provide the apiserver with access to cluster service endpoints in the absence of flannel and kube-proxy.
+
 The egress selector mode may be configured on servers via the `--egress-selector-mode` flag, and offers four modes:
 * `disabled`: The apiserver does not use agent tunnels to communicate with kubelets or cluster endpoints.
   This mode requires that servers run the kubelet, CNI, and kube-proxy, and have direct connectivity to agents, or the apiserver will not be able to access service endpoints or perform `kubectl exec` and `kubectl logs`.
 * `agent` (default): The apiserver uses agent tunnels to communicate with kubelets.
   This mode requires that the servers also run the kubelet, CNI, and kube-proxy, or the apiserver will not be able to access service endpoints.
-* `pod`: The apiserver uses agent tunnels to communicate with kubelets and service endpoints, routing endpoint connections to the correct agent by watching Nodes.
-  **NOTE**: This will not work when using a CNI that uses its own IPAM and does not respect the node's PodCIDR allocation. `cluster` or `agent` should be used with these CNIs instead.
-* `cluster`: The apiserver uses agent tunnels to communicate with kubelets and service endpoints, routing endpoint connections to the correct agent by watching Endpoints.
+* `pod`: The apiserver uses agent tunnels to communicate with kubelets and service endpoints, routing endpoint connections to the correct agent by watching Nodes and Endpoints.  
+  **NOTE**: This mode will not work when using a CNI that uses its own IPAM and does not respect the node's PodCIDR allocation. `cluster` or `agent` mode should be used with these CNIs instead.
+* `cluster`: The apiserver uses agent tunnels to communicate with kubelets and service endpoints, routing endpoint connections to the correct agent by watching Pods and Endpoints. This mode has the highest portability across different cluster configurations, at the cost of increased overhead.
 
 ## Dual-stack (IPv4 + IPv6) Networking