Make custom image for KIC similar to minikube ISO

[afbjorklund]

Currently we are using the KIND base image for the KIC driver:

https://github.com/kubernetes/minikube/blob/master/hack/images/kicbase.Dockerfile

# which is an ubuntu 19.10 with an entry-point that helps running systemd
# could be changed to any debian that can run systemd
FROM kindest/base:v20200122-2dfe64b2 as base

This image (docker.io/kindest/base) is in turn based on Ubuntu:

https://github.com/kubernetes-sigs/kind/blob/master/images/base/Dockerfile

# start from ubuntu 19.10, this image is reasonably small as a starting point
# for a kubernetes node image, it doesn't contain much we don't need
FROM ubuntu:19.10

As with the other base images, this one starts from a rootfs tarball:

https://hub.docker.com/_/ubuntu

FROM scratch
ADD ubuntu-eoan-core-cloudimg-amd64-root.tar.gz /

See https://docs.docker.com/develop/develop-images/baseimages/

---

We might want to investigate using the same Linux that we are using for minikube.iso,
a custom distribution built using Buildroot which also includes systemd and the runtimes.

https://github.com/kubernetes/minikube/tree/master/deploy/iso/minikube-iso

The main difference would be that the regular ISO also includes a kernel (currently 4.19),
while the container image uses the host kernel (so it doesn't need to waste all that space)

Links:

• https://buildroot.org/downloads/manual/manual.html
• blog.docker.com/2013/06/create-light-weight-docker-containers-buildroot

There are lots of other small tricks needed, in order to make a running container image.
Including a lot of workaround and hacks, to be able to run systemd in a a container...

As per above, you can see several of these in the original Ubuntu image (vs Ubuntu ISO),
as well as from the KIND and KIC projects respectively. Some of these will need to be added.

https://github.com/kubernetes-sigs/kind/blob/master/images/base/files/usr/local/bin/entrypoint

[afbjorklund]

ping @medyagh

[medyagh]

This sounds like a good idea to do.

I support to have an option to run kic drivers with buildroot image in docker.
at least as an alternative, that users could choose.
same way that users choose their base ISO_URL
we should be able to support different base images either, Ubuntu (kind base) or Buildroot

[medyagh]

@afbjorklund would you be intereted to take this? could I add this to the milestone?

[afbjorklund]

@medyagh : I will investigate, but don't know yet if it is reasonable to do in two weeks

[afbjorklund]

I would like to postpone this for the minikube 2.0 release, to become a target for Buildroot 2020.02

[afbjorklund]

One thing that needs to be addressed here is settling on a kernel version to support.
Our current virtual machines use either 4.19 or 5.4 kernels, which are a bit "too new"

So if you just create a image from the same rootfs, it will not run on Ubuntu 16.04 or 18.04
FATAL: kernel too old

This is because the default settings in the glibc build is to run with the current kernel only.
--enable-kernel=$(call qstrip,$(BR2_TOOLCHAIN_HEADERS_AT_LEAST)) \

BR2_TOOLCHAIN_HEADERS_AT_LEAST="4.19"

Besides glibc, systemd now also requires Stack Smashing Protection

BR2_SSP_NONE=y
# BR2_SSP_REGULAR is not set
# BR2_SSP_STRONG is not set
# BR2_SSP_ALL is not set
BR2_TOOLCHAIN_HAS_SSP=y

We should be able to lower this to something like 3.10 (or perhaps 4.0):
(systemd requires BR2_TOOLCHAIN_HEADERS_AT_LEAST_3_10)

http://www.linuxfromscratch.org/lfs/view/systemd/chapter06/glibc.html

             --enable-kernel=3.2                    \
             --enable-stack-protector=strong        \

Not sure if we miss any major features by doing so, but can check.

LTS	Ubuntu	Kernel
20.04	Focal Fossa	5.4
18.04	Bionic Beaver	4.15
16.04	Xenial Xerus	4.4
14.04	Trusty Tahr	3.13
12.04	Precise Pangolin	3.2+
10.04	Lucid Lynx	2.6.32

https://en.wikipedia.org/wiki/Ubuntu_version_history#Table_of_versions

Not sure what LinuxKit (Docker Desktop VM) has, but it's 4.9 or 4.14
So it would not run an image built for 4.19 either, much less 5.4 LTS...

[medyagh]

we dont have to support ubuntu, we could just do whatever kernel version that makes sense for us.
the goal would be use same version of everything in ISO and Base Image

[afbjorklund]

we dont have to support ubuntu, we could just do whatever kernel version that makes sense for us.

This is the user laptop we are talking about... It would be perfectly fine with an arbitrary 4.0 as well.