Epic: real-world experience migrating from docker to nerdctl

[apostasie]

I am using a home-grown deployment tooling (similar to terraform-docker) that heavily relies on a (docker) cli to bring up networks, volumes and containers to a desired state on remote iot nodes.

The tool connects over ssh to the node, evaluate the current state of the deployment then delete / create the necessary resources to bring it to the desired state.

I first tried to migrate from docker last year - quickly faced with a large number of failures - this is when I got involved with nerdctl to fix them.

Trying again a year after, things are better, but there are still a number of blocking issues.

This ticket is meant to keep track of all of them. Eventually, they can be broken down in individual tickets to be fixed when they have a simple reproducer or a diagnosis (some of them already are).

Currently blocking:

• Supports multiple IP address assignment for nerdctl run and nerdctl compose up #3663
• running rootful buildkitd inside a container in rootless mode: runc run failed: no cgroup mount found in mountinfo #3668 (while technically not a part of the deployment, this one is also hitting my lifecycle)

Problematic, but not blocking:

• CNI bridge: failed (add): failed to set bridge addr: could not set bridge's mac: invalid argument #4283
• Add Health Check Support in nerdctl #4157

Blocking using rootless:

• network -opt parent is not working rootless (failed to lookup master, link not found) #4276
• it is not clear that dhcp would work rootless

Blocking using DHCP:

• cni dhcp plugin does not seem to work well when there are multiple interfaces - in my context, eth0 is not connected, but wlan0 is - yet, dhcp insists on trying to bring up eth0 and get a lease of it, seemingly ignoring wlan0 - of course, it deadlines trying to do so
• cni dhcp plugin will block nerdctl run: in failing above, nerdctl run will get stuck (waiting for dhcp), with no timeout - note that the above is in the context of a vlan network which parent is wlan0 - either I do not know how to use the dhcp plugin properly, or this thing is just not ready

Minor / can be workaround / can live without for now:

• if uidmap is missing on the system, rootlesskit error is confusing / unhelpful - rootlesskit should test for the presence of the required binaries and provide a helpful message instead
• nerdctl inspect network does not show containers that are attached to multiple networks #4282

Fixed / patch pending:

• [PRIVATE PATCH] Subnet intersection check is preventing legit scenarios #4281
• somehow, nerdctl has a propensity to create broken containers (in unknown state), putting the system in a corrupted state that cannot be recovered from using nerdctl commands - and maybe show underlying containerd issues - this is being discussed in nerdctl rm -f does not work with containers in unknown state containerd#11890 (pending upstream fix, thanks to @ningmingxiao)

There might very well be more once I get past these. Will update this ticket accordingly.