TKLDev - support for docker in appliance builds

[CUCraigT]

[ note from @JedMeister ]

I've somewhat hijacked this issue to repurpose it to be a "feature request" for docker support in TKLDev builds.

---

I am attempting to build an appliance to basically just run Docker and a couple Docker containers. I have successfully gotten Docker installed based on TKL Core and installing those packages. However, I then want to pull the two Docker images I want using docker pull. This fails during the Make looking like the socket is not available. I have even tried to start the the docker service explicitly and still no luck. Here is the error I get:

+ echo Start Docker Service
Start Docker Service
+ systemctl start docker
[ ok ] Starting Docker: docker.
+ whoami
root
+ echo Pull Docker Images
Pull Docker Images
+ docker pull mcr.microsoft.com/mssql/server:2017-latest
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
make: *** [/usr/share/fab/product.mk:540: build/stamps/root.patched] Error 1

Here is my conf.d file:

#!/bin/sh -ex
SQL_VERSION="2017-latest"
echo "Install Docker"

# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable"

echo "deb [arch=amd64] https://download.docker.com/linux/debian $(lsb_release -cs) stable" >> /etc/apt/sources.list.d/sources.list

curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -

apt-get update
apt-get install docker-ce docker-ce-cli containerd.io

echo "Start Docker Service"
systemctl start docker

whoami
echo "Pull Docker Images"

#Pull MSSQL Server Image
docker pull mcr.microsoft.com/mssql/server:$SQL_VERSION

Any thoughts or tips on how I could do this?

[JedMeister]

As per the error message, is the docker daemon running?

systemctl status docker

Also FWIW here is the install script we use to install docker. It includes additional "best practice" security measures such as not including the repo key as a default "trusted key" and pinning the repository. It's possibly somewhat irrelevant now as you've added the key to your "trusted keys" but the pinning is still a good idea.

[CUCraigT]

Thanks for the quick response. I will give your script a try. I did add systemctl status docker to the conf.d/main file and got an error saying status was unknown:

+ echo Start Docker Service
Start Docker Service
+ systemctl start docker
[ ok ] Starting Docker: docker.
+ whoami
root
+ echo Docker Service Status
Docker Service Status
+ systemctl status docker
FATAL [/usr/local/bin/systemctl]: Unknown command 'status'
make: *** [/usr/share/fab/product.mk:540: build/stamps/root.patched] Error 1

[CUCraigT]

Decided to do service docker status instead and it returned that Docker is running. Tried a docker version and get the error on the server piece. So, yes it appears the service is running but issues connecting to it. I have seen this when I try to run the commands as non-root, but the build seems to be running as root:

+ echo Start Docker Service
Start Docker Service
+ systemctl start docker
[ ok ] Starting Docker: docker.
+ whoami
root
+ echo Docker Service Status
Docker Service Status
+ service docker status
[ ok ] Docker is running.
+ docker version
Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:45:50 2020
 OS/Arch:           linux/amd64
 Experimental:      false
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
make: *** [/usr/share/fab/product.mk:540: build/stamps/root.patched] Error 1

[JedMeister]

Ah ok. Apologies, in my haste, I missed that you were building an appliance... (For some reason I just thought that you were trying to install Docker on Core; rereading it now seems obvious that you're creating an appliance...).

So with that context in mind, I suspect that it's a limitation of our build system. Although I'm not 100% sure which one...

We use OverlayFS in the build process (the "decks" that are the make targets are overlaid over each other as "layers"). As Docker also uses OverlayFS (AFAIK?), that may be a factor?

The second, and probably more likely, is that our build environment is a chroot. Previous init systems would happily run within a chroot, but that is not the case for systemd. Other than 'enable' and 'disable' (i.e. to enable and disable a service autostart at boot) commands will fail. Some old services provided an SysVinit script (i.e. a start/stop script within /etc/init.d) and those can be leveraged via a direct call, e.g. /etc/init.d/script_name start or often a service call, e.g. service script_name start. (note that service calls will be picked by systemctl when the system is running "properly", but during build time will leverage init.d script when available).

However, what makes all this a somewhat moot point, is that to work around various issues with specific appliances, we actually provide build time wrappers for both the service and systemctl commands. You can find them in the common - systemd-chroot overlay. Note that there is some redundancy between the 2 scripts (I created one and my colleague Stefan created the other).

So what I would suggest is rather than trying to work out what is going on via the script, instead try resolving the issue interactively within the chroot. I.e. from within your appliance build code dir; /turnkey/fab/products/YOUR_APP:

fab-chroot build/root.patched

You will now be inside the build system. You can run commands interactively and test stuff out. Once you're done, you can exit to get back to the TKLDev host.

My suspicion is that you'll either want to start the required process manually (in your conf script - don't forget to capture the PID and kill it when you're done). Or alternatively, tweak the service wrapper script to do that (and explicitly copy it out and add it to the overlay). You'll find the service and systemctl wrapper scripts within the chroot at /usr/local/bin/.

As hinted above, don't forget to copy out any files that you want to keep. E.g. if you tweak the service script. Otherwise any modifications you made will be lost as soon as you re-run make.

[CUCraigT]

Thank you so much for the detailed response, and your assumption is correct. I am trying to build a simple appliance with Core+Docker+My Docker Image, so we can ship our Docker container to folks that don't have a Kubernetes or Docker environment.

Appreciate the command to get into chroot interactively; I had been struggling to figure that out from the docs. Seeing what is happening is extremely helpful.

With that being said, looks like you ARE correct in your assumption with the OverlayFS. When I just try to start the Docker Daemon with /usr/bin/dockerd, I in fact get error trying to mount the overlays:

ERRO[2020-08-05T17:52:15.756556350Z] failed to mount overlay: invalid argument     storage-driver=overlay2
ERRO[2020-08-05T17:52:15.762181148Z] AUFS was not found in /proc/filesystems       storage-driver=aufs
ERRO[2020-08-05T17:52:15.763081799Z] failed to mount overlay: invalid argument     storage-driver=overlay
ERRO[2020-08-05T17:52:15.802698034Z] Udev sync is not supported. This will lead to data loss and unexpected behavior. Install a more recent version of libdevmapper or select a different storage driver. For more information, see https://docs.docker.com/engine/reference/commandline/dockerd/#storage-driver-options  storage-driver=devicemapper

Going to play around with it a bit more, but have you ever done something similar getting Overlay to work on top of Overlay? The good thing is that Docker works great on my appliance; just really want to be able to seed it with the container images prior to shipping, so it works. If not, I may have to get a startup script on the actual appliance to do the load

[JedMeister]

Glad we're on the same page and that my post was useful. 😄

I wonder if it's worth trying to see if overlayFS and AUFs will play nice together? We use OverlayFS as it's included in the kernel. However Debian have an AUFs kernel DKMS module package. I can't guarantee it'll work, but it might? Also Docker has a vfs driver. I don't know much about it and apparently it's performance is poor, but for the purposes of building it might be sufficient? you'd probably only want to use the alternate driver at build time though.

[CUCraigT]

Worth trying @JedMeister. I'll take a look. I may also take a look at including the docker images as tar.gz files in an overlay directory and then doing a docker load command as part of inithooks. Will play with that a bit today as well. Thanks for the help working through this. This appliance should be pretty straightforward; just want to make sure the essential files are prepopulated, so they don't HAVE to update to get it running.

[JedMeister]

I may also take a look at including the docker images as tar.gz files in an overlay directory and then doing a docker load command as part of inithooks.

That may be a workable optiion. Although FWIW, for our appliances, we aim to have all install and initial config done at build time (then re-initialized as relevant on firstboot; e.g. secret regeneration, etc). Our experience suggests that this approach tends to lead to the fewest possible points of breakage for end users. Although obviously it doesn't eliminate chance of issues, it does reduce vectors.

just want to make sure the essential files are prepopulated, so they don't HAVE to update to get it running.

Good plan! 😄

[CUCraigT]

@JedMeister Got to looking at this again today. Apparently, if you specify one overlay filesystem to pull images and/or build containers and then change it, then those images become inaccessible, so couldn't specify one filesystem at build time and then switch it or we would lose those images (per this doc: https://docs.docker.com/storage/storagedriver/select-storage-driver/)

When you change the storage driver, any existing images and containers become inaccessible. This is because their layers cannot be used by the new storage driver. If you revert your changes, you can access the old images and containers again, but any that you pulled or created using the new driver are then inaccessible.

Any other options you can think of or am I stuck with inithooks, even understanding it is not best practice?

[JedMeister]

Argh!

Given that info, my suggestions is to give aufs a try?! TBH, I can't be sure how auFS will interact with OverlayFS, and whether it'll work. But I guess there's only one way to find out...

Be aware, that you'll need to install that on both the host (for usage during build time) and the guest (for usage when installed). It will unfortunately add a bit of cruft to your ISO (making it bigger) but hopefully it will work...

IIRC these are the packages that you'll need (install on TKLDev for use at build time and add to plan for later usage):

aufs-dkms dkms linux-headers-amd64

I suspect that you may also want aufs-tools installed on the guest too (but no harm installing on the host as well); but I'm not 100% sure.

Sorry I don't have a more precise answer...

[CUCraigT]

Thanks, @JedMeister. Just to be clear would this be using aufs for TKL or for Docker? I am assuming for Docker but just wanted to clarify.

[JedMeister]

@CUCraigT

Just to be clear would this be using aufs for TKL or for Docker?

TBH, you could go either way. TKLDev historically used to use auFS (it was the default union filesystem provided by Debian). IIRC that was v14.x and earlier. Once overlayFS was merged into the kernel mainline, Debian dropped default auFS support (Debian were patching their kernel to provide it). IIRC that was Debian Stretch (v15.x). When we first started development for that release, we initially just installed auFS in TKLDev as I noted above. However, we ended up refactoring to leverage overlayFS (thus reducing the size a bit). I'm pretty sure the main thing that changed was deck. Previously it was a python script but Alon rewrote it in bash for overlayFS support. It would probably be possible to rebuild the old python (2) package and it should work on v16.x, but it's not something I have time for ATM. There may be other bits and pieces.

TL;DR I anticipate that using Docker with auFS would be much less work (if it works at all that is). But would conversely increase the size of your resulting ISO and would mean that Docker in the resulting system would need to continue to use auFS.

One other thought that I've had is that IIRC one of the shortcomings of overlayFS is that it only allows a singe read/write layer. I suspect that is the underlaying cause of the issue you're experiencing. Assuming that's the case, then it may be possible to tweak fab to remount root.patched as read-only while the docker stuff happens, then remount as read/write once the docker stuff is done?!

[CUCraigT]

Interesting. A bit above my head, but I may try to figure that out. Really appreciate the help/ideas.

So, all of Docker stores the images in /var/lib/docker. So I just did a docker pull on my TKLDEV box and then copied /var/lib/docker all to the overlay directory. It actually worked and built an ISO! However, I did some cleanup and tried to build again, and now I am getting the following errors on some of those files when I do a make (same error with make clean).

cp: cannot create special file 'build/root.patched/var/lib/docker/overlay2/a01defc16e24fbcc849141a3492b71d00ba7b21f410c259c9194a0dab8797a4a/diff/etc/nginx/conf.d/default.conf': Operation not permitted

There are other files also; just including one so you can see. Any idea, @JedMeister?

Upon further investigation, looks like that file is trying to do a mknod. Tried tarballing that directory and then extracting in conf.d/main, but get that error upon extraction

[JedMeister]

So, all of Docker stores the images in /var/lib/docker. So I just did a docker pull on my TKLDEV box and then copied /var/lib/docker all to the overlay directory. It actually worked and built an ISO!

Ah, cool hack! I like the way you think! 😄

However, I did some cleanup and tried to build again, and now I am getting the following errors on some of those files when I do a make (same error with make clean).

Hmm, that's weird that it worked once, but now won't allow you to rerun any of the steps...

Also the message suggests that it's the copy (cp) command which is failing - but you said this occurs when you run make and/or make clean right? Or did I miss something?!

Although it's not ideal, when all else fails on TKLDev, I suggest a reboot. Then log back in, cd to your build code dir and re-run make clean.

As to how to "properly" copy in the /var/lib/docker dir from the host, I suggest not copying at all, and instead bind mounting it into the overlay directory. I.e.

cd YOUR_BUILD_CODE_DIR
mkdir -p overlay/var/lib/docker
mount --bind /var/lib/docker overlay/var/lib/docker

TBH, I'm not 100% sure that will work, but it should...

[CUCraigT]

Thanks, @JedMeister. Will try that route a bit more.

Also, tried the aufs route tonight. Seems that docker and the filesystem are not seeing it with I go into fab-chroot build/root.patched. Get the following error starting dockerd:

ERRO[2020-08-21T04:03:05.845580521Z] AUFS was not found in /proc/filesystems       storage-driver=aufs

Also, if I cat /proc/filesystems:

root@tkldev /# cat /proc/filesystems 
nodev	sysfs
nodev	rootfs
nodev	ramfs
nodev	bdev
nodev	proc
nodev	cpuset
nodev	cgroup
nodev	cgroup2
nodev	tmpfs
nodev	devtmpfs
nodev	debugfs
nodev	tracefs
nodev	securityfs
nodev	sockfs
nodev	dax
nodev	bpf
nodev	pipefs
nodev	hugetlbfs
nodev	devpts
nodev	mqueue
nodev	pstore
	ext3
	ext2
	ext4
nodev	autofs
nodev	overlay
nodev	binfmt_misc
root@tkldev /# 

I did install aufs-dkms dkms linux-headers-amd64 aufs-tools in host and guest.