External disk support

[ibrokethecloud]

Problem:

Currently harvester can only be installed on local disks due to lack of multipath support and ability to configure additional kernel arguments as part of the installation process

Solution:

The PR attempts to address this by allowing user to perform the following via harvester config

• enable multipathd
• configure multipathd to support specific devices
• enable boot from multipath device

Related Issue:
harvester/harvester#5150
Test plan:

To test the following config.yaml, will enable and configure multipath even for a single local disk in a kvm environment

os:
  externalStorageConfig:
    enabled: true
    multiPathConfig:
    - vendor: "ATA"
      product: "QEMU HARDDISK"
  additionalKernelArguments: "rd.iscsi.firmware rd.iscsi.ibft"

post install, when the VM boots, the disks used will be corresponding multipathd mapper devices

Filesystem                              1K-blocks     Used Available Use% Mounted on
devtmpfs                                     4096        0      4096   0% /dev
tmpfs                                    10236112        4  10236108   1% /dev/shm
tmpfs                                     4094448     9200   4085248   1% /run
tmpfs                                        4096        0      4096   0% /sys/fs/cgroup
/dev/mapper/QEMU_HARDDISK_QM00003-part4  15375304  4866216   9706272  34% /run/initramfs/cos-state
/dev/loop0                                3091752  1609416   1325052  55% /
tmpfs                                     5118060    13208   5104852   1% /run/overlay
overlay                                   5118060    13208   5104852   1% /boot
overlay                                   5118060    13208   5104852   1% /etc
/dev/mapper/QEMU_HARDDISK_QM00003-part2     42851      285     38982   1% /oem
overlay                                   5118060    13208   5104852   1% /srv
/dev/mapper/QEMU_HARDDISK_QM00003-part5 153707984 29135744 116691536  20% /usr/local
overlay                                   5118060    13208   5104852   1% /var
tmpfs                                    10236116        0  10236116   0% /tmp
/dev/mapper/QEMU_HARDDISK_QM00003-part6  78841476       32  74790756   1% /var/lib/harvester/defaultdisk

without the additional config, there should be no impact to how harvester behaves post install

[tserong]

LGTM in general, just a couple of comments on comments ;-) and one question about the blacklist.

(Also I do wonder if we shouldn't just enable multipathd in the installer... But in that case we'd probably still need to add logic to hide multipath children so they couldn't be selected by mistake :-/ ... What you've got here is probably more straightforward and/or less risky).

[tserong]

Do we need a default devnode line here that blacklists everything? I was just looking at the manpage for multipath.conf, and it says:

The default blacklist consists of the regular expression "!^(sd[a-z]|dasd[a-z]|nvme[0-9])". This causes all device types other than scsi, dasd, and nvme to be excluded from multipath handling by default."

My reading is that with multipathd enabled all scsi, dasd and nvme devices are still up for grabs. Is this correct, or am I confused?

[ibrokethecloud]

from my experiments we are blacklisting everything apart from the negated expression in blacklist which whitelists the devices.

for example on my kvm setup the following config.yaml is used

os:
  externalStorageConfig:
    enabled: true
    multiPathConfig:
    - vendor: "QEMU"
      product: "QEMU HARDDISK"

This gets converted into multipath config

blacklist {

    device {
        vendor "!QEMU"
        product "!QEMU HARDDISK"
    }

}

end result is multipath is enabled on the qemu scsi disks

0QEMU_QEMU_HARDDISK_disk1 dm-0 QEMU,QEMU HARDDISK
size=250G features='0' hwhandler='0' wp=rw
|-+- policy='service-time 0' prio=1 status=active
| `- 0:0:0:1 sda 8:0  active ready running
`-+- policy='service-time 0' prio=1 status=enabled
  `- 0:0:0:2 sdb 8:16 active ready running

Longhorn disks in use by VM's are ignored by mutlipath since they do not match the expression

  *-disk
       description: SCSI Disk
       product: VIRTUAL-DISK
       vendor: IET
       physical id: 0.0.1
       bus info: scsi@7:0.0.1
       logical name: /dev/sdc
       version: 0001
       serial: beaf11
       size: 10GiB (10GB)
       capabilities: gpt-1.00 partitioned partitioned:gpt
       configuration: ansiversion=5 guid=57debec6-c414-4d56-a0a1-788da7c89c52 logicalsectorsize=512 sectorsize=512

[mergify]

This pull request is now in conflict. Could you fix it @ibrokethecloud? 🙏

[tserong]

LGTM!

[Vicente-Cheng]

just a nit, others LGTM!

BTW, could we squash some commits if they are similar?

[bk201]

@ibrokethecloud please help squash commits and we can merge this. Thanks!

[Vicente-Cheng]

thanks!

[ibrokethecloud]

@Mergifyio backport v1.4

[mergify]

backport v1.4

✅ Backports have been created

• #826 External disk support (backport #778) has been created for branch v1.4