[TEST] v1.4.0 release testing - Four node upgrade with bonded nics

[TachunLin]

What's the test to develop? Please describe

Perform the four nodes upgrade test with bonded nics for v1.4.0 release testing

Prerequisite and dependency of test

1. VLAN 1 network on mgmt and 1 network on bonded NICs
2. 2 Virtual machines with data and md5sum computed- 1 running, 1 stopped
3. 2 VM backup, snapshots - 1 backup when VM is running and 1 backup when VM is stooped
4. Create a new storage class apart from default one. Use the new storage class for some basic operations.
5. Import to Rancher and create an RKE1 & RKE2 guest cluster. provisioned on Harvester VM before the upgrade.
6. Deploy Harvester cloud provider to RKE1 Cluster (prior to latest version)
7. Verify DHCP load balancer service
8. Install Harvester CSI Driver in RKE2 (prior to latest version)
9. Create a new Harvester PVC for nginx deployment
10. Create Load balancer and PVC in the guest cluster in RKE2 cluster.
11. Upgrade Rancher
12. Create another one node guest cluster.
13. Scale down up the guest cluster gc1.

Upgrade Test Path:

• H1.3.2 + R2.7.15 (K1.27) -> H1.4.0 + R2.8.9 (K1.28) -> (K1.29)

OS:

• SLES 15 SP4 (sles@ipv4)
• Windows 2022

Upgrade Test Check

1. Can correctly upgrade from v1.3.2 to v1.4.0-rc4

Post Upgrade Check

1. Dependencies Check
   • dependency_charts
   • Versions of all the crds
2. Virtual machines are in same state as before and accessible.
3. Restore the backups, check the data
4. Image and volume status
5. Monitoring chart status
6. VM operations are highlighted and working fine.
7. Add a node after the upgrade
8. Upgrade RKE1/RKE2 guest cluster
9. Upgrade cloud provider and CSI driver
10. Verify DHCP load balancer service and create a new Harvester PVC
11. Shutting off VM and then restarting VM

[TachunLin]

Four nodes upgrade test on SLES 15 SP6 from v1.3.2 to v1.4.0-rc4

Test environment

Harvester: v1.3.2 upgrade to v1.4.0-rc4
Rancher: v2.7.15 (Prime)
RKE2 version:
OS: SLES 15 SP6

Prerequisite and dependency setup

${\textsf{✅}}$ VLAN 1 network on mgmt and another vlan 100 network on bonded NICs $~~$

${\textsf{✅}}$ 2 Virtual machines with data and md5sum computed- 1 running, 1 stopped $~~$

sles@sles15-sp6-running:~> sha512sum running.txt
831658c6fa11148b1b6c91aeedf5dedb7eca9ed088b5561deaa1774d8bfce7bf322a1bcaf71b4457e688c64f2a1e6adbd71f273f9ae8bf7d482289963ac17f8a  running.txt

sles@sles15-sp6-stopped:~> sha512sum stopped.txt
6c9ea237c6001f86c0aeadaed585d9dc04ddfe704992284db7c56c5fdc9c874a01c9dc47d459cf82686aefa28dbadc836340d10a01731896c3ee05fd6f813ee7  stopped.txt

${\textsf{✅}}$ 2 VM backup, snapshots - 1 backup when VM is running and 1 backup when VM is stooped $~~$

${\textsf{✅}}$ Create a new storage class apart from default one. Use the new storage class for some basic operations. $~~$

${\textsf{✅}}$ Import to Rancher 2.7.15 and create an RKE2 v1.27.16 guest cluster. provisioned on Harvester VM before the upgrade. $~~$

${\textsf{✅}}$ Verify cloud provider and csi-driver version and running $~~$

• csi-driver: 0.1.18
• cloud provider: 0.2.4

${\textsf{✅}}$ Create a new Harvester PVC for nginx deployment $~~$

${\textsf{✅}}$ Verify DHCP load balancer service $~~$

Upgrade from v1.3.2 to v1.4.0-rc4

${\textsf{✅}}$ Successfully upgrade to v1.4.0-rc4 $~~$

• Remember to apply this workaround
  [BUG] Upgrade to v1.2-head stuck in pre-draining stage due to RKE2 guest cluster vm kept in Migrating state  harvester#5756 (comment)

  

Post Upgrade Check

${\textsf{✅}}$ Dependency check $~~$

Chart: https://github.com/harvester/harvester/tree/v1.4.0-rc4/deploy/charts/harvester/dependency_charts
Values: https://github.com/harvester/harvester/blob/v1.4.0-rc4/deploy/charts/harvester/values.yaml

• csi-snapshotter

  $\color{green}{\textsf{PASS}}$ $~~$

  • tag: v6.3.3
    https://github.com/harvester/harvester/blob/v1.4.0-rc4/deploy/charts/harvester/dependency_charts/csi-snapshotter/values.yaml
  • Harvester: csi-snapshotter
    

• kubevirt-operator

  $\color{green}{\textsf{PASS}}$ $~~$

  • tag: 1.2.2-150500.8.21.1
    https://github.com/harvester/harvester/blob/v1.4.0-rc4/deploy/charts/harvester/values.yaml

  • Harvester: virt-operator (deployment)

    

• kubevirt

  $\color{green}{\textsf{PASS}}$ $~~$

  • tag: 1.2.2-150500.8.21.1
    https://github.com/harvester/harvester/blob/v1.4.0-rc4/deploy/charts/harvester/values.yaml

  • Harvester: kubevirt (crd)

    

• snapshot-validation-webhook

  $\color{green}{\textsf{PASS}}$ $~~$

  • tag: 6.3.3
    https://github.com/harvester/harvester/blob/v1.4.0-rc4/deploy/charts/harvester/values.yaml

  • Harvester: harvester-snapshot-validation-webhook- (deployment)

    

• whereabouts

  $\color{green}{\textsf{PASS}}$ $~~$

  • tag: 0.6.3
    https://github.com/harvester/harvester/blob/v1.4.0-rc4/deploy/charts/harvester/values.yaml

  • Harvester: harvester-whereabouts- (pod)

    

${\textsf{✅}}$ Versions of all the crds $~~$

https://github.com/harvester/harvester/tree/v1.4.0-rc4/deploy/charts/harvester-crd/templates

• addons.harvesterhci.io

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_addons.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_addons.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• keypairs.harvesterhci.io

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_keypairs.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.3.2-rc2/deploy/charts/harvester-crd/templates/harvesterhci.io_keypairs.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_preferences

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_preferences.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_preferences.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_settings

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_settings.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_settings.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_supportbundles

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_supportbundles.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_supportbundles.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_upgradelogs

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_upgradelogs.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_upgradelogs.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_upgrades

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_upgrades.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_upgrades.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_versions

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_versions.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_versions.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_virtualmachinebackups

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinebackups.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinebackups.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_virtualmachineimages

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachineimages.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachineimages.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_virtualmachinerestores

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinerestores.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinerestores.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_virtualmachinetemplates

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinetemplates.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinetemplates.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

• harvesterhci.io_virtualmachinetemplateversions

  $\color{green}{\textsf{PASS}}$ $~~$

  https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinetemplateversions.yaml

  harvester-node-0:~ # ./check-harvester-crd.sh https://raw.githubusercontent.com/harvester/harvester/v1.4.0-rc4/deploy/charts/harvester-crd/templates/harvesterhci.io_virtualmachinetemplateversions.yaml
  Observation: v1beta1 
  Expectation: v1beta1 
  
  True
  

${\textsf{✅}}$ Virtual machines are in same state as before and accessible $~~$

${\textsf{✅}}$ Restore the backups, check the data $~~$

1. Restore the running backup vm to new, the data can be restored and exists

2. Restore the stopped backup vm to new, the data can be restored and exists

${\textsf{❌}}$ Image and volume status $~~$

• Volume state is fine
  

• One of the image display 0% progress and can't be used,
  

• Track in issue [BUG] After upgrade to v1.4.0-rc4, one image display 0% progress and can't be used anymore  harvester#6894

${\textsf{✅}}$ Monitoring chart status $~~$

${\textsf{✅}}$ VM operations are highlighted and working fine $~~$

• Start
• Stop
• Restart
• Soft reboot
• Delete
• Clone
• Migrate
• Add volume

${\textsf{✅}}$ Rancher v2.7.15 and RKE2 cluster running well $~~$

${\textsf{✅}}$ Upgrade Rancher to v2.8.9 $~~$

1. Can successfully upgrade Rancher to v2.8.9 and the RKE2 cluster is running

${\textsf{✅}}$ Upgrade RKE2 guest cluster to v1.28.14+rke2r2 $~~$

1. Can successfully upgrade the RKE2 cluster to v1.28.14+rke2r2

${\textsf{✅}}$ Upgrade cloud provider and CSI driver $~~$

1. Automatically upgrade cloud provider to v0.2.6 and csi-driver to 0.1.18
   

${\textsf{✅}}$ Create a new Harvester PVC $~~$

1. Create a new Harvester PVC for nginx deployment

${\textsf{✅}}$ Verify DHCP load balancer service $~~$

1. Verify DHCP load balancer service