[ReleaseTesting] Four node upgrade with bonded NICs #1254

albinsun · 2024-05-06T04:28:11Z

Description

Content refer to Qase HV-608, test following upgrade paths

Harvester

1.2.1 -> 1.2.2-rc2 -> 1.3-head

Harvester with Rancher guest cluster

H1.2.1 + R2.7.10 -> H1.2.1 + R2.7.11 -> H1.2.2-rc2 + R2.7.12 -> H1.3-head + R2.8.3

Test Steps

Prerequisites

VLAN 1 network on mgmt and 1 network on bonded NICs

2 Virtual machines with data and md5sum computed- 1 running, 1 stopped

2 VM backup, snapshots - 1 backup when VM is running and 1 backup when VM is stooped

Create a new storage class apart from default one. Use the new storage class for some basic operations.

Import to Rancher 2.7.10 and create an RKE2 guest cluster 1.26. provisioned on Harvester VM before the upgrade.

Deploy Harvester cloud provider to RKE1 Cluster (prior to latest version)

Verify DHCP load balancer service

Install Harvester CSI Driver (prior to latest version)

Create a new Harvester PVC for nginx deployment

Upgrade Rancher to v2.7.11

Guest cluster still works

Upgrade Harvester to v1.2.2-rc2

Guest cluster still works
Post-upgrade checks

Upgrade Rancher to v2.7.12

Guest cluster still works

Upgrade Harvester to v1.3-head

Guest cluster still works

Upgrade Rancher to v2.8.3

Guest cluster still works

Post upgrade checks

Dependencies Check

dependency_charts

Versions of all the crds

Virtual machines are in same state as before and accessible.

Restore the backups, check the data

Image and volume status

Monitoring chart status

VM operations are highlighted and working fine.

Add a node after the upgrade

Upgrade the RKE2 guest cluster to 1.27 or above

Upgrade cloud provider and CSI driver

Verify DHCP load balancer service and create a new Harvester PVC

Shutting off VM and then restarting VM

The text was updated successfully, but these errors were encountered:

albinsun · 2024-05-07T04:59:27Z

Prerequisites

🟢 VLAN 1 network on mgmt and 1 network on bonded NICs
🟢 2 Virtual machines with data and md5sum computed- 1 running, 1 stopped
🟢 2 VM backup, snapshots - 1 backup when VM is running and 1 backup when VM is stooped

backup-target

Backup

Snapshot
🟢 Create a new storage class apart from default one. Use the new storage class for some basic operations.

Both 2 VMs has an additional volume using new customized storage class
🟢 Import to Rancher 2.7.10 and create an RKE2 (1.26) guest cluster. provisioned on Harvester VM before the upgrade.

Import to rancher-v2.7.10

Create RKE2 (v1.26) cluster
⚪ Deploy Harvester cloud provider to ~~RKE1~~RKE2 Cluster (prior to latest version) ⚠️ RKE2 install harvester-cloud-provider by default
⚪ Install Harvester CSI Driver (prior to latest version) ⚠️ RKE2 install harvester-csi-driver by default
🟢 Create a new Harvester PVC for nginx deployment

Check default storage class

Create PVC

Nginx deployment
🟢 Verify DHCP load balancer service

albinsun · 2024-05-07T08:24:53Z

Upgrade Rancher to v2.7.11

Ref. https://ranchermanager.docs.rancher.com/v2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades#upgrade-outline

🟢 Upgrade successfully
🟢 Harvester is still Active
🟢 VM is still there and data not loss
🟡 RKE2 cluster is still running
- Hit known cloud-provider issue (w/ workaround)
  [BUG] A pending fail pod from different hub is generated after upgrade rancher harvester#5382
🟢 Nginx deployment and LB still works

Deployment

LB

albinsun · 2024-05-07T18:10:08Z

Upgrade Harvester to v1.2.2-rc

v1.2.2-rc2

Trial#1

Ref. https://docs.harvesterhci.io/v1.2/upgrade/index/#prepare-an-air-gapped-upgrade

🔴 Upgrade successfully

[BUG] Live migration fail when upgrade v1.2.1 to v1.2.2-rc2 due to virError harvester#5755

Trial#2

🔴 Upgrade successfully

[BUG] upgrade stuck in waiting plan restart-rancher-system-agent to complete harvester#5690

v1.2.2-rc3

Trial#1

🟢 Upgrade successfully
🟢 Harvester is still Active

Post upgrade checks

Dependencies Version Check
Chart: https://github.com/harvester/harvester/tree/master/deploy/charts/harvester/dependency_charts
Values: https://github.com/harvester/harvester/blob/v1.2/deploy/charts/harvester/values.yaml
- 🟢 csi-snapshotter
- 🟢 kubevirt-operator
- 🟢 kubevirt
- 🟢 snapshot-validation-webhook
- 🟢 whereabouts
Versions of all the CRDs
- dependency_charts
  - 🟢 snapshotter
    
    volumesnapshotclasses.snapshot.storage.k8s.io
    
    volumesnapshotcontents.snapshot.storage.k8s.io
    
    volumesnapshots.snapshot.storage.k8s.io
  - 🟢 kubevirt-operator
    
    kubevirts.kubevirt.io
  - 🟢 whereabouts
    
    ippools.whereabouts.cni.cncf.io
    
    overlappingrangeipreservations.whereabouts.cni.cncf.io
- harvester-crd
  https://github.com/harvester/harvester/tree/v1.2/deploy/charts/harvester-crd/templates
  - 🟢 addons.harvesterhci.io
  - 🟢 keypairs.harvesterhci.io
  - 🟢 preferences.harvesterhci.io
  - 🟢 settings.harvesterhci.io
  - 🟢 supportbundles.harvesterhci.io
  - 🟢 upgradelogs.harvesterhci.io
  - 🟢 upgrades.harvesterhci.io
  - 🟢 versions.harvesterhci.io
  - 🟢 virtualmachinebackups.harvesterhci.io
  - 🟢 virtualmachineimages.harvesterhci.io
  - 🟢 virtualmachinerestores.harvesterhci.io
  - 🟢 virtualmachinetemplates.harvesterhci.io
  - 🟢 virtualmachinetemplateversions.harvesterhci.io
🟢 Virtual machines are in same state as before and accessible.

VM is still there and data not loss

RKE2 cluster is still running

Nginx deployment and LB still works
🟡 Restore the backups, check the data

Running VM
1. Restore New
2. Restore Replace
3. Restore New again
Stopped VM
1. Restore New
2. Restore Replace
3. Restore New again
Successive restore hit known issue
[BUG] VMs from successive "Restore New" backup/snapshot tend to get the same IP. harvester#4474
🟢 Image and volume status
🟢 Monitoring chart status
VM operations are highlighted and working fine.
- 🟢 Restart
- 🟡 Snapshot
- 🟡 Backup
- 🟢 Migrate
- 🟡 Template
- 🟢 Soft Reboot
- 🟢 Add Volume
  
  Successive restore hit known issue
  [BUG] VMs from successive "Restore New" backup/snapshot tend to get the same IP. harvester#4474
🟢 Add a node after the upgrade

node-3 is added as Compute node

Can access harvester via rancher

LB and deployment still work
🟢 Upgrade the RKE2 guest cluster to 1.27 or above
Before ![image](https://github.com/harvester/tests/assets/2773781/817bcb76-b4ac-4319-928b-58c3d8d28c5c) ![image](https://github.com/harvester/tests/assets/2773781/9220108c-ed06-4cd4-888d-fbb5d5c3f2f4)
Upgrade

After

deployments looks good

Can access nginx deployment

LB still works
🟡 Upgrade cloud provider and CSI driver

cloud provider
- Before
- Upgrade
- After
csi-driver
- Before
- After
Known cloud-provider issue w/ workaround
[BUG] A pending fail pod from different hub is generated after upgrade rancher harvester#5382
🟢 Verify DHCP load balancer service and create a new Harvester PVC

DHCP LB

create a new Harvester PVC
🟢 Shutting off VM and then restarting VM
1. Before
2. Shutting off
3. Restart and check

albinsun · 2024-05-10T04:24:17Z

Upgrade Rancher to v2.7.12

Ref. https://ranchermanager.docs.rancher.com/v2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades#upgrade-outline

🟢 Upgrade successfully
🟢 Harvester is still Active
🟢 VM is still there and data not loss
🟢 RKE2 cluster is still running
🟢 Nginx deployment and LB still works

albinsun · 2024-05-10T12:43:03Z

Upgrade Harvester to v1.3-head (`v1.3-bf5847ad-head`)

🟢 Upgrade successfully
🟢 Harvester is still Active
🟢 Virtual machines are in same state as before and accessible.

VM is still there and data not loss

RKE2 cluster is still running

Nginx deployment and LB still works
🟡 Restore the backups, check the data

Running VM
1. Restore New
2. Restore Replace
Stopped VM
1. Restore New
2. Restore Replace
Successive restore hit known issue
[BUG] VMs from successive "Restore New" backup/snapshot tend to get the same IP. harvester#4474
🟢 Image and volume status

🔴 Monitoring chart status

No monitoring charts

Can NOT disable addon

helm-install-rancher-monitoring-vhh29 + echo 'Installing helm_v3 chart'                                                                              
helm-install-rancher-monitoring-vhh29 + helm_v3 install --version 103.0.3+up45.31.1 rancher-monitoring rancher-monitoring/rancher-monitoring --value
helm-install-rancher-monitoring-vhh29 Error: INSTALLATION FAILED: cannot re-use a name that is still in use

No charts, and unable to disable addon.

VM operations are highlighted and working fine.
- 🟢 Stop
- 🟢 Restart
- 🟢 Snapshot
- 🟢 Backup
- 🟢 Migrate
- 🟡 Soft Reboot
  
  IP addtress not shown on UI
- 🟢 Add Volume
  
  Successive restore hit known issue
  [BUG] VMs from successive "Restore New" backup/snapshot tend to get the same IP. harvester#4474
⚪ Upgrade the RKE2 guest cluster to 1.27 or above
Before
Upgrade

After

deployments looks good

Can access nginx deployment

LB still works

Skip due to RKE2 version do not change
🟢 Verify DHCP load balancer service and create a new Harvester PVC

DHCP LB

create a new Harvester PVC
🟢 Shutting off VM and then restarting VM

albinsun · 2024-05-10T13:39:34Z

Upgrade Rancher to v2.8.3

Ref. https://ranchermanager.docs.rancher.com/v2.7/getting-started/installation-and-upgrade/install-upgrade-on-a-kubernetes-cluster/upgrades#upgrade-outline

🟢 Upgrade successfully
🟢 Harvester is still Active
🟢 VM is still there and data not loss
🟡 RKE2 cluster is still running

After apply workardound

Known cloud-provider issue w/ workaround
[BUG] A pending fail pod from different hub is generated after upgrade rancher harvester#5382
🟢 Nginx deployment and LB still works

albinsun · 2024-05-13T07:46:22Z

Close as symptoms are found in v1.2.2-rc3 to v1.3-head phase and it's basically fine for v1.2.1 to v1.2.2-rc3 phase.

albinsun added this to the 1.2.2 milestone May 6, 2024

albinsun self-assigned this May 6, 2024

albinsun closed this as completed May 13, 2024

m-ildefons mentioned this issue Jun 5, 2024

[TEST] [Release Testing] Test installed CRD API versions of system charts #1314

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ReleaseTesting] Four node upgrade with bonded NICs #1254

[ReleaseTesting] Four node upgrade with bonded NICs #1254

albinsun commented May 6, 2024 •

edited

Loading

albinsun commented May 7, 2024 •

edited

Loading

albinsun commented May 7, 2024 •

edited

Loading

albinsun commented May 7, 2024 •

edited

Loading

albinsun commented May 10, 2024

albinsun commented May 10, 2024

albinsun commented May 10, 2024

albinsun commented May 13, 2024

[ReleaseTesting] Four node upgrade with bonded NICs #1254

[ReleaseTesting] Four node upgrade with bonded NICs #1254

Comments

albinsun commented May 6, 2024 • edited Loading

Description

Harvester

Harvester with Rancher guest cluster

Test Steps

Prerequisites

Upgrade Rancher to v2.7.11

Upgrade Harvester to v1.2.2-rc2

Upgrade Rancher to v2.7.12

Upgrade Harvester to v1.3-head

Upgrade Rancher to v2.8.3

Post upgrade checks

albinsun commented May 7, 2024 • edited Loading

Prerequisites

albinsun commented May 7, 2024 • edited Loading

Upgrade Rancher to v2.7.11

albinsun commented May 7, 2024 • edited Loading

Upgrade Harvester to v1.2.2-rc

v1.2.2-rc2

Trial#1

Trial#2

v1.2.2-rc3

Trial#1

Post upgrade checks

albinsun commented May 10, 2024

Upgrade Rancher to v2.7.12

albinsun commented May 10, 2024

Upgrade Harvester to v1.3-head (v1.3-bf5847ad-head)

albinsun commented May 10, 2024

Upgrade Rancher to v2.8.3

albinsun commented May 13, 2024

albinsun commented May 6, 2024 •

edited

Loading

albinsun commented May 7, 2024 •

edited

Loading

albinsun commented May 7, 2024 •

edited

Loading

albinsun commented May 7, 2024 •

edited

Loading

Upgrade Harvester to v1.3-head (`v1.3-bf5847ad-head`)