[RFE] Allow overriding the main kernel serial console on bare-metal

[jqueuniet]

Current situation

Flatcar seems to hardcode the following console kernel parameters with disk-based boot on bare-metal: console=ttyS0,115200n8 console=tty0. I haven't fully investigated the boot chain, but the likeliest source is /usr/boot/syslinux/root.A.cfg, which contains the following line:

  append console=ttyS0,115200n8 console=tty0 ro noswap cros_legacy  root=LABEL=ROOT rootflags=subvol=root usr=PARTLABEL=USR-A

My full cmdline currently looks like this:

rootflags=rw mount.usrflags=ro BOOT_IMAGE=/flatcar/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTUUID=7130c94a-213a-4e5a-8e26-6cce9662f132 rootflags=rw mount.usrflags=ro consoleblank=0 root=LABEL=ROOT console=ttyS0,115200n8 console=tty0 console=tty0 console=ttyS1,115200n8 verity.usrhash=5c215d2523556d4992ba36684815e8e6fad1d468795f4ed0868a855d0b76a607

Impact

Serial console stops working on hardware using ttyS1 instead of ttyS0 once the kernel is executed by the bootloader. ttyS1 is the default for some manufacturers (ie: Supermicro, Gigabyte), and some do not provide the necessary firmware configuration to change this setting. And even when they do provide this setting, changing the serial console parameters can be a tedious operation to do at scale depending on the hardware Redfish capabilities, and having the ability to adapt the OS provisioning to the default hardware settings would be a lot more flexible.

Sadly, the Linux kernel does not support multiple consoles of the same type and subsequent serial consoles are ignored, like adding console=ttyS1 after console=ttyS0 in my example above. See the kernel serial console documentation for further details : https://www.kernel.org/doc/html/v6.9/admin-guide/serial-console.html

Ideal future situation

Ability to override the serial console, ideally in a way that can be easily automated during provisioning.

Implementation options

Off the top of my head, I can think of the following:

• Flatcar bare-metal artifacts could omit console parameters entirely, let kernel automatic console detection do its work and leave further tuning to the provider/operator through Ignition. But it is a breaking change which could lead to regressions for current users though.
• Ignition provides a way to remove arguments, through kernel_arguments.should_not_exist. Using this option does not work currently for console flags as it can only act on a subset of the kernel cmdline, but could be expected to.
• The flatcar-install script could offer a way to customize this parameter as it is the one responsible for writing the initial image

Additional information

Note that the serial console does work with PXE ramdisks as those artifacts let the operator in control of the full kernel cmdline. It only becomes an issue after flatcar-install is run and the server switches to disk-based boot.

Also of note, starting the serial console from userland once boot is finished with systemctl start serial-getty@ttyS1.service does not work either. I tried adding ttyS1 to a copy of /usr/share/shadow/securetty file in /etc, to no avail. But even if I did manage to get this to work, it would only partially solve my issue as ttyS1 would not receive any kernel message if setup only with systemd.

[jqueuniet]

Another implementation option and probably the simplest, moving the hardcoded console parameters after the cmdline part controlled by Ignition. This way operators would be able to override them.

[jepio]

The logic regarding kernel parameters is here: https://github.com/flatcar/scripts/blob/main/build_library/grub.cfg#L71-L96:

if [ -z "$linux_console" ]; then
    if [ "$grub_platform" = pc ]; then
        set linux_console="console=ttyS0,115200n8 console=tty0"
        serial com0 --speed=115200 --word=8 --parity=no
        terminal_input console serial_com0
        terminal_output console serial_com0
    elif [ "$grub_platform" = efi ]; then
        if [ "$grub_cpu" = arm64 ]; then
            set linux_console="console=ttyAMA0,115200n8"
        else
            set linux_console="console=ttyS0,115200n8 console=tty0"
       fi
    elif [ "$grub_platform" = xen ]; then
        set linux_console="console=hvc0"
    fi
fi

set extra_options=""
if [ "$grub_cpu" = arm64 ]; then
	set extra_options="acpi=force"
fi

set suf=""

# Assemble the options applicable to all the kernels below
set linux_cmdline="rootflags=rw mount.usrflags=ro consoleblank=0 $linux_root $linux_console $first_boot $randomize_disk_guid $extra_options $oem $linux_append"

You can override the default console by providing a value for set linux_console in /oem/grub.cfg. This would require mounting the oem partition after running flatcar-install.

[jqueuniet]

I can indeed get a serial console working with the following post-install script:

# Override the default serial console
mount $(blkid -l -t "LABEL=OEM" -o device /dev/disk/by-path/{{ .InstallPath }}) /mnt
cat > /mnt/grub.cfg <<EOF
set linux_console="console=tty0 console={{ .SerialConsole }}"
umount /mnt
EOF

Thanks a lot for the help, this does solve my issue.