Support for toolbox containers

[afbjorklund]

Description

I was looking into support for "toolbox containers", both the original root variants and the more recent rootless one:

https://github.com/coreos/toolbox/tree/0.0.9 (systemd)

https://github.com/coreos/toolbox (switched to podman)

https://github.com/flatcar/toolbox (went back to docker)

https://containertoolbx.org/ (rewritten, from Bash to Go)

Unfortunately the more recent toolbox requires podman, which in turn almost requires fedora (or get an old podman).

There is support for Ubuntu distribution (and a podman-toolbox package), and there is support for Podman in Ubuntu.

But it is not possible to use another container engine such as systemd or nerdctl, since this is not a goal of the project...

anders@lima-default:~$ toolbox
Command 'toolbox' not found, but can be installed with:
apt install podman-toolbox
Please ask your administrator.
anders@lima-default:~$ sudo apt install podman-toolbox
Installing:                     
  podman-toolbox
Installing dependencies:

aardvark-dns                     libabsl20230802          libjbig0                        libxcb-render0

adwaita-icon-theme               libasound2-data          libjpeg-turbo8                  libxcb-shm0

alsa-topology-conf               libasound2t64            libjpeg8                        libxcomposite1

alsa-ucm-conf                    libatk-bridge2.0-0t64    liblcms2-2                      libxcursor1

at-spi2-common                   libatk1.0-0t64           liblerc4                        libxdamage1

at-spi2-core                     libatspi2.0-0t64         libmalcontent-0-0               libxfixes3

bubblewrap                       libavahi-client3         libostree-1-1                   libxi6

buildah                          libavahi-common-data     libpango-1.0-0                  libxinerama1

catatonit                        libavahi-common3         libpangocairo-1.0-0             libxrandr2

conmon                           libavahi-glib1           libpangoft2-1.0-0               libxrender1

containernetworking-plugins      libcairo-gobject2        libpipewire-0.3-0t64            libxtst6

crun                             libcairo2                libpipewire-0.3-common          libyajl2

dconf-gsettings-backend          libcolord2               libpixman-1-0                   netavark

dconf-service                    libcups2t64              librsvg2-2                      p11-kit

desktop-file-utils               libdatrie1               librsvg2-common                 p11-kit-modules

flatpak                          libdconf1                libsharpyuv0                    passt

fontconfig                       libdeflate0              libslirp0                       podman

fontconfig-config                libepoxy0                libspa-0.2-modules              slirp4netns

fonts-dejavu-core                libfontconfig1           libsubid4                       ubuntu-mono

fonts-dejavu-mono                libgdk-pixbuf-2.0-0      libthai-data                    uidmap

fuse-overlayfs                   libgdk-pixbuf2.0-bin     libthai0                        x11-common

golang-github-containers-common  libgdk-pixbuf2.0-common  libtiff6                        xdg-dbus-proxy

golang-github-containers-image   libgraphite2-3           libwayland-client0              xdg-desktop-portal

gsettings-desktop-schemas        libgtk-3-0t64            libwayland-cursor0              xdg-desktop-portal-gtk

gtk-update-icon-cache            libgtk-3-bin             libwayland-egl1

hicolor-icon-theme               libgtk-3-common          libwebp7

humanity-icon-theme              libharfbuzz0b            libwebrtc-audio-processing-1-3
Suggested packages:

containers-storage  malcontent-gui      colord       liblcms2-utils  docker-compose   xdg-desktop-portal-gnome

libwasmedge0        alsa-utils          cups-common  pipewire        accountsservice

avahi-daemon        libasound2-plugins  gvfs         librsvg2-bin    evince
Summary:

Upgrading: 0, Installing: 106, Removing: 0, Not Upgrading: 20

Download size: 60.9 MB

Space needed: 262 MB / 100 GB available
Continue? [Y/n] n

Abort.

anders@lima-default:~$ toolbox enter
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download quay.io/toolbx-images/ubuntu-toolbox:24.10 (500MB)? [y/N]: y
Error: failed to pull image quay.io/toolbx-images/ubuntu-toolbox:24.10
If it was a private image, log in with: podman login quay.io
Use 'toolbox --verbose ...' for further details.
anders@lima-default:~$ toolbox enter --distro fedora
Error: option '--release' is needed
Distribution fedora doesn't match the host.
Run 'toolbox --help' for usage.
anders@lima-default:~$ toolbox --version
toolbox version 0.0.99.3
anders@lima-default:~$ podman --version
podman version 5.0.3

---

Anyway, the idea is that you use single VM (possibly read-only or spartan) - and then run all your work in containers.

The feature of the "toolbox" is that it preps the container for you, by selecting a common image and/or creating a user.

So that would make it more similar to WSL2, and the Hyper-V VM used: https://learn.microsoft.com/en-us/windows/wsl/

Where WSL actually does not add the user for you, just suggesting that you might want to look into adding one yourself.

Like so, example from Alma: https://wiki.almalinux.org/documentation/wsl.html (creating a system container)

Compared with: https://wiki.almalinux.org/cloud/Generic-cloud-on-local.html (creating an AlmaLinux VM/"lima")

So there could some integration with the "default" template and nerdctl, to access such a container running in Lima.

To start with, I made some small updates to the old toolbox (replaced rkt with crane) and it seems to run just fine...

---

I think that colima started on some similar features called "layers", but that term is somewhat overloaded (overlayfs)

They are called "distros" in WSL, but that term is even worse. And there seems to be plenty of "toolbox" around as well.

But the goal would be to have some variant of lima, that would drop you in a container instead of on the machine.

It is basically just running nerdctl run -it, the trick is setting up the parameters and the mounts - and the image.

Toolbox: https://github.com/afbjorklund/systemd-toolbox

limactl start --containerd none
apt install systemd-container
install crane, install toolbox

$ lima toolbox
░ Spawning container anders-fedora-latest on /var/lib/toolbox/anders-fedora-latest.
░ Press Ctrl-] three times within 1s to kill container.
[root@lima-default ~]# 

limactl start --containerd system

$ lima sudo nerdctl run -it -v /:/run/host fedora:latest
[root@494ddcce9f1c /]#

Archive size:
56M fedora_latest.tar (nerdctl save format)
55M fedora_latest.tar.gz
43M fedora_latest.tar.zst

---

With rootless and Lima it would look more similar to (but also needs some more setup):

limactl start

$ nerdctl.lima run -it -u `id -u`:`id -g` -v $HOME:$HOME -w $HOME fedora:latest
docker.io/library/fedora:latest:                                                  resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:3ec60eb34fa1a095c0c34dd37cead9fd38afb62612d43892fcf1d3425c32bc1e:    done           |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:9cfb3a7ad0a36a1e943409def613ec495571a5683c45addb5d608c2c29bb8248: done           |++++++++++++++++++++++++++++++++++++++| 
config-sha256:aa6787b90fe61e801687142277458584287469c9596c91766a43fa9f1e524c22:   done           |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:0c5a86865c5d3e78a4ab19ac7c516ffe93e41e0fd67f052a72f52d07cd2c59f9:    done           |++++++++++++++++++++++++++++++++++++++| 
elapsed: 18.0s                                                                    total:  55.9 M (3.1 MiB/s)                                       
bash-5.2$ whoami
whoami: cannot find name for user ID 1000

So the user, group and home directory needs adding. To the image, or the container at init.

[afbjorklund]

The apptainer experience is very similar to this, main difference is that it uses a squashfs image and not a directory.

But you remain as the same user, and you have access to your files and so on. Otherwise it calls it --containall.

[afbjorklund]

The "user" image looks something like this: (adding an entry to the system, with the user and group)

https://code.visualstudio.com/remote/advancedcontainers/add-nonroot-user#_creating-a-nonroot-user

[afbjorklund]

Creating a user image:

FROM ubuntu:latest
RUN userdel --remove ubuntu

RUN groupadd -g 1000 anders && useradd -u 1000 -g 1000 -d /home/anders.linux -s /bin/bash -m -N anders && rm -f /etc/.pwd.lock
WORKDIR /home/anders.linux
USER anders
CMD ["/bin/bash"]

Running user container:

$ nerdctl.lima run -it -u `id -u`:`id -g` -v $HOME:$HOME -w $HOME -e TERM anders-ubuntu-latest
anders@39ca671a1094:/home/anders$

Note: probably better to create the container in the background, and then exec into it with the shell?

It should also feature an --init process, to care of the zombies. And have a --name, like any pet would.

[afbjorklund]

A "toolbox" would typically run as a privileged container, and use the host namespaces for network and pid.

It would also mount the entire host filesystem (somewhere like /media/root or /run/host), and add sudo.

[afbjorklund]

Someone mentioned that there is also "distrobox", which is another system that is similar to toolbx...

Distrobox uses podman, docker or lilipod to create containers using the Linux distribution of your choice

Note: distrobox uses the GNU General Public License version 3, unlike toolbox which uses Apache License version 2

It is written in shell (/bin/sh), unlike toolbx which is using Go. There are lots of examples and different distributions.

---

EDIT: It will need some love to work with nerdctl and dash, though. It wants to use docker and /bin/sh as bash...

anders@lima-default:~$ distrobox create
Missing dependency: we need a container manager.
Please install one of podman,  docker or lilipod.
You can follow the documentation on:
	man distrobox-compatibility
or:
	https://github.com/89luca89/distrobox/blob/main/docs/compatibility.md

sudo ln -s nerdctl /usr/local/bin/docker
/usr/local/bin/distrobox-enter: 3: eval: Syntax error: ";" unexpected

The problems are with the integrated Go templates, that work with docker and podman - but not (yet) with nerdctl

[afbjorklund]

For the lima handbook

toolbx

limactl start template://podman
export LIMA_INSTANCE=podman
lima sudo dnf install -y toolbox

$ lima toolbox enter
No Toolbx containers found. Create now? [y/N] y
Image required to create Toolbx container.
Download registry.fedoraproject.org/fedora-toolbox:41 ( ... MB)? [y/N]: y
Error: directory /home/anders not found in container fedora-toolbox-41
Using /home/anders.linux instead.

Welcome to the Toolbx; a container where you can install and run
all your tools.

 - Use DNF in the usual manner to install command line tools.
 - To create a new tools container, run 'toolbox create'.

For more information, see the documentation.

⬢ [anders@toolbx ~]$ 

distrobox

limactl start --name=docker --containerd=none #needs "docker.io" deb
export LIMA_INSTANCE=docker
lima sudo apt update && lima sudo apt install -y docker.io distrobox
lima sudo usermod -aG docker $USER && ssh -F $(limactl ls --format='{{.SSHConfigFile}}' docker) lima-docker -O exit

$ lima distrobox enter
Error response from daemon: No such container: my-distrobox
Create it now, out of image registry.fedoraproject.org/fedora-toolbox:39? [Y/n]: 
Creating the container my-distrobox
39: Pulling from fedora-toolbox
40eb91ad8a1c: Pull complete 
Digest: sha256:11f3634d4a8f2d4a69c4ad8442133f69979be49fa6269eccc6ab0863c39d59d0
Status: Downloaded newer image for registry.fedoraproject.org/fedora-toolbox:39
registry.fedoraproject.org/fedora-toolbox:39
Creating 'my-distrobox' using image registry.fedoraproject.org/fedora-toolbox:39	 [ OK ]
Distrobox 'my-distrobox' successfully created.
To enter, run:

distrobox enter my-distrobox

uccessfully copied 2.56kB to /tmp/my-distrobox.os-release
Starting container...                   	 [ OK ]
Installing basic packages...            	 [ OK ]
Setting up devpts mounts...             	 [ OK ]
Setting up read-only mounts...          	 [ OK ]
Setting up read-write mounts...         	 [ OK ]
Setting up host's sockets integration...	 [ OK ]
Integrating host's themes, icons, fonts...	 [ OK ]
Setting up package manager exceptions...	 [ OK ]
Setting up rpm exceptions...            	 [ OK ]
Setting up distrobox profile...         	 [ OK ]
Setting up sudo...                      	 [ OK ]
Setting up user groups...               	 [ OK ]
Setting up kerberos integration...      	 [ OK ]
Setting up user's group list...         	 [ OK ]
Adding user...                          	 [ OK ]
Setting up user home...                 	 [ OK ]
Ensuring user's access...               	 [ OK ]

Container Setup Complete!
⚠️  First time user password setup ⚠️ 
Changing password for user anders.
New password: 
Retype new password: 
passwd: all authentication tokens updated successfully.
anders@lima-docker:/run/host/home/anders$ . /etc/profile.d/distrobox_profile.sh
📦[anders@my-distrobox anders]$

No lima changes need, for those to work.

EDIT: Except for the prompt being broken.

[afbjorklund]

Project has been added here: https://github.com/afbjorklund/nerdctl-toolbox

Too many toolbox around, so renamed it to nerdbox. No host access, either.

$ nerdbox create
579dd26aed42
$ nerdbox shell
anders@nerdbox:~$ 

Both fedora and ubuntu have weird setups, so went with plain old debian:

$ nerdctl diff nerdbox
A /etc/hosts
C /tmp
A /tmp/lima
C /usr/bin
C /usr/sbin
A /usr/sbin/tini

The others are there too, through changing $DISTRO environment variable.

No sudo, but it is possible to spawn a root shell with the usual exec -u 0...

$ nerdctl exec -u 0 -w /root -it nerdbox bash
root@nerdbox:~# 

No archives yet either, but then again you can save and load (as usual)

[afbjorklund]

Now the Containerfile can be previewed before building the image, after setup.

And then you can upgrade by removing the old container and creating a new...
Upgrading packages from within the container is discouraged, so I'll skip that.

RUN apt-get update && apt-get install -y procps psmisc && rm -rf /var/lib/apt/lists/*

RUN groupadd -g $gid $user && useradd -u $uid -g $gid -d $home -s $shell -m -N $user

The other toolboxes now adds user and home during the entrypoint (init) instead.
And then set up sudo, so that you can tweak the container instead of the image.

Bundled "ufetch", for good measure. https://github.com/beucismis/awesome-fetch

[afbjorklund]

I don't think there is any more to be done here? Works for me, at least. :-D

System toolbox: (like in CoreOS toolbox) - 72 lines

• https://github.com/afbjorklund/systemd-toolbox

User toolbox: (like in WSL and the others) - 73 lines

• https://github.com/afbjorklund/nerdctl-toolbox

Fixing toolbx and distrobox, for nerdctl support, belongs to those projects?