ability to include additional images in the kindest/node image

[dkoshkin]

What would you like to be added:
Currently the kind node images comes with the core kubernetes images and a few other images that it needs to start.
It would be great if it was possible to include additional images into the kind node image that it can have ready to be used.

Why is this needed:
In an air-gapped cluster where you may only need to have a few additional images at bootstrap, having pre-baked images in a kind image could be a simple way to get access to a Kubernetes API.

Some options here could be:

• Document a way to use the existing kindest/node image as a base image (I tried for a little bit but didn't find anything online and crictl pull wasn't working, but would love feedback if it is possible).
• Extend kind build node-image CLI command to add additional images.
• Rely on existing kind load command. More complicated since you would need to ship around multiple images, but an option that already works.
• Rely on a local registry. This adds an external dependency that complicates the setup.

[BenTheElder]

Document a way to use the existing kindest/node image as a base image (I tried for a little bit but didn't find anything online and crictl pull wasn't working, but would love feedback if it is possible).

It won't work for the same reason that kind build node-image isn't just a docker build, you need a privileged container to run container such that you can pull / import images. You can accomplish this external to kind but not via a normal image build.

Another approach might be trying to build a tool / command for this instead of doing it while building the rest of the node image (so users could start from a standard node image).

Rely on existing kind load command. More complicated since you would need to ship around multiple images, but an option that already works.

Did you know that you can have multiple images into a single tarball? docker save image1 image2 image3.
So if this is your only concern, you can do docker save kindest/node app-image1 app-image2 > images.tar and copy a single tarball around.

Rely on a local registry. This adds an external dependency that complicates the setup.

This one also persists image loading across clusters though. And is something we've already been tracking / documenting.

#2159 (comment) has some more questions for consideration

[joshrosso]

+1 to this desire of loading additional images to create a custom node image.

I've been working off a fork that has very similar modifications to @dkoshkin's work in #2159 (comment). The only difference is I input my images from a .yaml file, e.g:

preload:
  images:
    - "custom-repo/repo/cert-manager-controller:v0.16.1_customerver.1"
    - "custom-repo/repo/cluster-api/cluster-api-aws-controller:v0.6.4_customerver.1"
    - "custom-repo/repo/cluster-api/kubeadm-bootstrap-controller:v0.3.14_vmware.2"
    - "custom-repo/repo/cluster-api/kubeadm-control-plane-controller:v0.3.14_vmware.2"
    - "custom-repo/repo/cluster-api/kube-rbac-proxy:v0.4.1_vmware.2"
    - "custom-repo/repo/cert-manager-cainjector:v0.16.1_customerver.1"
    - "custom-repo/repo/cert-manager-webhook:v0.16.1_customerver.1"

Based on this issue and the open PR, it seems like the current open questions are:

• Should we instead, build a new tool / command for this instead of doing it while building the rest of the node image (so users could start from a standard node image).
• should we pull images if they're not present locally?
• if we're pulling images, should we just pull them directly to containerd instead of to the host?
• what about podman?
• what about tarballs / image archives?
• what about image eviction in the future? should kind be responsible for somehow preventing these images from being evicted if we re-enable eviction once we have a solution for the core images?
• could we instead support this in a way that involved extending an existing node image without having to build from source? would that be better?

[joshrosso]

Here are my initial thoughts on the above 😃 :

Should we instead, build a new tool / command for this instead of doing it while building the rest of the node image (so users could start from a standard node image).

could we instead support this in a way that involved extending an existing node image without having to build from source? would that be better?

Yeah, I think you're right that this makes the most sense for 2 reasons.

1. It does not pollute the node-image build process
2. The idea of "extending" the base node-image is (IMO) a better UX.

if we're pulling images, should we just pull them directly to containerd instead of to the host?

Yes, directly into containerd seems sensible.

what about podman?

If we are loading directly into containerd, does this matter?

what about tarballs / image archives?

Yes, this should be supported too. I'm not convinced you'd need it immediately, but an API could look roughly like

type AdditionalImages struct {
	Images []struct {
		imageName string
		imageTag  string
		// when pulling from a local image archive (.tar)
		imageLocation string
	}
}

what about image eviction in the future? should kind be responsible for somehow preventing these images from being evicted if we re-enable eviction once we have a solution for the core images?

Personally, I don't think doing anything special for image eviction is warranted.