Addons are not updated correctly during upgrade from openstack-scs-1-30-v1 to main branch ClusterStack

[Nils98Ar]

/kind bug

What steps did you take and what happened:

1. Release openstack-scs-1-30-v1, e.g. to private OCI registry (from 87d65bb commit)
2. Release openstack-scs-1-30-v2, e.g. to private OCI registry (from current main branch)
3. Deploy openstack-scs-1-30-v1 cluster
4. Upgrade cluster to openstack-scs-1-30-v2 (from K8s 1.30.2 to 1.30.6)
5. See that the addons are not correctly updated to the new versions, see here and here. But there is at least something happening, e.g. there appear new cilium-envoy resources.

I cannot find anything useful like errors in the cso logs. Deploying a new openstack-scs-1-30-v2 cluster instead of upgrading a openstack-scs-1-30-v1 one seems to be successful.

To finish the update manually it seems to enough to apply the addons manually (can be found e.g. under /tmp/downloads in the cso container):

kubectl delete daemonset -n kube-system openstack-cinder-csi-nodeplugin openstack-cloud-controller-manager metrics-server openstack-cinder-csi-controllerplugin
kubectl delete deployments.apps -n kube-system metrics-server openstack-cinder-csi-controllerplugin
helm template -n kube-system --kube-version 1.30.6 . | kubectl apply -f -

Sometimes the upgrade gets stuck because the HelmChartApplied condition of the ClusterAddon object does not become true, see issue here. In this case it helps to set .spec.clusterStack and .spec.version in the ClusterAddon to openstack-scs-1-30-v1 / v1 again to re-trigger the ClusterAddon update and wait for HelmChartApplied to become true. But I am not sure if this is related, even if this does not happen the update does not finish successfully.

What did you expect to happen:

The upgrade completes successfully, or at least the cso logs show what went wrong.

Environment:

• OS on mgmt host (e.g. from /etc/os-release): Debian 12