Automatic network config propagation in the initramfs when using Tang

[jlebon]

Problem

When Tang is used, networking must be enabled in the initramfs on every boot. This is today done by the rootmap code. However, no network configuration is persisted by default. This works fine in the DHCP case, but if e.g. using static networking, it's up to the user to configure it using kernel arguments.

The current approaches for solving this are:

1. On bare metal, use coreos-installer install --append-karg ip=... to make the kernel arguments permanent rather than just on first boot.
2. On "cloud" platforms, use a backchannel for configuring networking (this currently only exists for VMware), and use Ignition kargs to also append the network kargs persistently (or before that, a systemd service which uses rpm-ostree kargs).

There are multiple concerns with this:

1. The UX is not great. In the bare metal case, this requires dropping down to scripting the install, and in the cloud case, it requires typing your network kargs twice (once for the backchannel, and once for e.g. Ignition kargs).
2. We would now have two sources of truth for networking: in the kernel command-line, and in the real root. Since we propagate network configs only on first boot, they can diverge afterwards and it's up to the user to edit them both: once in the real root in the NM configs, and once in the kargs using that syntax.
3. Network kargs are suboptimal for multiple reasons. We should keep working on improving the UX to make them less necessary.

Proposal

On first boot, we have code in the initramfs which detects if Tang is used. If so, then after NM keyfiles are propagated to the real root (either from coreos-installer's --copy-network, or from nm-initrd-generator) we automatically use rpm-ostree initramfs-etc --track /etc/NetworkManager/system-connections. The benefits of this are:

1. It requires no additional work at coreos-installer time: if users are currently relying on the "forwarded kargs" magic, then it will hook into this seamlessly since those kargs are translated into NM configs on first boot. If instead they're scripting --copy-network, then it will also hook into that seamlessly.
2. It defines a clear source of truth: after propagation at first boot, the files at /etc/NetworkManager/system-connections are now the source of truth, and simply kept in sync in the initramfs on each new deployment. This makes it much easier to change network configuration if necessary without worrying about also having to update the initramfs.
3. It involves writing less network kargs. In the bare metal case, one no longer needs to --append-karg. In the VMware case, the user would still need to initially use the backchannel to configure via kargs for bootstrapping network, though at least they'd no longer need to also encode it via Ignition kargs or systemd unit. Maybe down the line we can have a tool which is capable of embedding NM keyfiles inside of our VMware images to make this easier too.

[dustymabe]

xref: https://bugzilla.redhat.com/show_bug.cgi?id=1975701

One other thing I had thought of (before I considered initramfs-etc) was to add a mode to our network propagation code that would allow to switch the type of network propagation. One would be via persistent kernel arguments, the other being keyfiles (what we have today). I originally liked my proposal better, but it is sufficiently complex that I think I might like the initramfs-etc idea better.

One other question.. with initramfs-etc if I edit the keyfiles and then reboot (no rpm-ostree transaction) then the changes won't get picked up, correct?

[jlebon]

One other question.. with initramfs-etc if I edit the keyfiles and then reboot (no rpm-ostree transaction) then the changes won't get picked up, correct?

Right yeah, thanks for bringing this up. There's a --force-sync flag for this case, so documentation about changing network configs when using Tang would have to mention a workflow like:

$ vi /etc/NetworkManager/system-connections # or nmcli or nmtui
$ rpm-ostree initramfs-etc --force-sync

[jlebon]

coreos/fedora-coreos-config#1374 does a softer version of the proposal here (though more generalized in a way; we just check if rd.neednet=1 is enabled instead of specifically for Tang).