Misc NDM Improvement

[Vicente-Cheng]

Problem:

1. Now we find out that NDM will remove the hot-plug device from the blockdevices that would cause the data to be lost when it add back.
2. Also, we allow the user to use the repair tool instead of the force format again.
3. Some improvement

Solution:

1. We changed the state to inactive for the hotplugged device instead of deleting it from blockdevices CR.
2. Add new field corrupted. For the corrupted device, we could choose the force format or try to use e2fsck to repair.
3. Log improvement
4. reduce the redundant CR update
5. remove the polling mechanism (But still keep two handling function now)

Related Issue:
harvester/harvester#3218
harvester/harvester#3412
harvester/harvester#4016

Test plan:
case 1:

1. add a single disk (should be external storage, that we could hot-plug)
2. remove this disk
3. check the blockdevices status is inactive
4. add this disk back
5. check that the blockdevices status is active and should work again.

case 2:
NOTE1: Please update the blockdevices CRD (on this PR) before we do this test
NOTE2: Please ensure you use the SCSI disk with wwn or the filesystem UUID will be erased when we try to break the filesystem.

1. add a single disk
2. create external storage with the above disk
3. remove this storage
4. make this filesystem corrupted (you could use the following command dd if=/dev/zero of=/dev/sda bs=512k count=1 oflag=sync
5. try to create external storage with this disk
6. you should see the mount error and check the Status.Filesystem.Corrupted is true
7. Check on the UI that you could force format again.
8. Add it back with force format, and it should work again.

[bk201]

Update
False alarm; there are duplicated WWN in the testing disks. Please ignore the following comment.

Try to attach some disks in a very short time, but only some are created:

for ((i=0;i<10;i++)); do ./attach-disk.sh node2 ; done

Block devices (please ignore node3)

NAMESPACE         NAME                               TYPE   DEVPATH    MOUNTPOINT   NODENAME   STATUS   AGE
longhorn-system   5819b466585e1e3b1c9ed398918a9cb1   disk   /dev/sdi                node2      Active   5m4s
longhorn-system   8a655634522b710cc4f2d3514f10b5c3   disk   /dev/sdc                node3      Active   10m
longhorn-system   91c3333ea91016f699f1641ba3228354   disk   /dev/sda                node2      Active   5m11s
longhorn-system   a7f262a2e9610974dd9904ee577f7515   disk   /dev/sdf                node2      Active   5m8s
longhorn-system   b3d51f437d5c71e7476929625234549e   disk   /dev/sdd                node3      Active   10m
longhorn-system   d390352fbf27678e80eed80342918336   disk   /dev/sde                node3      Active   10m

Virtual disks:

node2:~ # ls -alh /dev/sd*
brw-rw---- 1 root disk 8,   0 Jul  3 06:38 /dev/sda
brw-rw---- 1 root disk 8,  16 Jul  3 06:24 /dev/sdb
brw-rw---- 1 root disk 8,  32 Jul  3 06:38 /dev/sdc
brw-rw---- 1 root disk 8,  48 Jul  3 06:38 /dev/sdd
brw-rw---- 1 root disk 8,  64 Jul  3 06:38 /dev/sde
brw-rw---- 1 root disk 8,  80 Jul  3 06:38 /dev/sdf
brw-rw---- 1 root disk 8,  96 Jul  3 06:38 /dev/sdg
brw-rw---- 1 root disk 8, 112 Jul  3 06:38 /dev/sdh
brw-rw---- 1 root disk 8, 128 Jul  3 06:38 /dev/sdi
brw-rw---- 1 root disk 8, 144 Jul  3 06:38 /dev/sdj
brw-rw---- 1 root disk 8, 160 Jul  3 06:38 /dev/sdk

NDM log:
harvester-node-disk-manager-4v85m.log

[bk201]

LGTM