feat(cluster-addons): Use clouds.yaml instead of cloud.conf

[jschoone]

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #188

Special notes for your reviewer:

Please confirm that if this PR changes any image versions, then that's the sole change this PR makes.

TODOs:

• squash commits
• include documentation
• add unit tests

[jschoone]

@garloff to add the cacert it's probably enough to add a second file called cacert (containing the cacert obviously) and point to that in the clouds.yaml with clouds.openstack.cacert=/etc/openstack/cacert since the clouds-yaml secret will be mounted to /etc/openstack for CCM and CSI

[garloff]

Questions:

• The idea is to save those two files and then inject them into the workload cluster manually later?
• How does capo then get its secrets?
• Any chance I can still tell the CCM to default to octavia-ovn rather than to octavia-amphora?

[jschoone]

Questions:

• The idea is to save those two files and then inject them into the workload cluster manually later?

No they will be applied by CAPI as CRS

• How does capo then get its secrets?

The dry-run can be removed than use that secret.

• Any chance I can still tell the CCM to default to octavia-ovn rather than to octavia-amphora?

Of course, can just be configured in the values.yaml
All these options are available then: https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/openstack-cloud-controller-manager/using-openstack-cloud-controller-manager.md#load-balancer

[garloff]

So I can generate a secret that capo can work with:

apiVersion: v1
data:
  clouds.yaml: $CL_YAML_B64
  cacert: $CACERT_B64
kind: Secret
metadata:
  name: openstack
  namespace: $CS_NAMESPACE
  labels:
    clusterctl.cluster.x-k8s.io/move: "true"
type: Opaque

Note that the CL_YAML_B64 contains cacert: /etc/certs/cacert which appears to be the place that capo maps it to.

I do not see the secrets propagated into the workload cluster yet, so OCCM and friends have no way to access the OpenStack API yet.

[garloff]

OK, the helper chart (which we no longer want to use) created a clusterresourceset,, will have to do something similar.

[jschoone]

Yes, it is created as connection between the Cluster and the ClusterResourceSet as soon as a Cluster is created with matching labels
It has the name of the cluster:

kubectl get clusterresourcesetbindings.addons.cluster.x-k8s.io my-cluster1 -oyaml
apiVersion: addons.cluster.x-k8s.io/v1beta1
kind: ClusterResourceSetBinding
metadata:
  name: my-cluster1
  namespace: default
spec:
  bindings:
  - clusterResourceSetName: testclouds-v1-clouds-yaml
  clusterName: my-cluster1

[garloff]

OK, the clusterSelector.matchLabels entry is wrong.
I apply the label managed-secret: cloud-config to the cluster, thus matchLabels.manged-secret: clouds-yaml can't work. Mystery solved, thanks @jschoone

[garloff]

OK, I have it working.
I now look for clusterstackrelease status.resources openstack-scs-$VERSION-clouds-yaml and if it exists, we add a label managed-secret: clouds-yaml instead of cloud-config to the cluster, so the CRS can do its matching and works.
I renamed the openstack-workload-cluster-secret and the opestack-crs-secret into ...-newsecret for the new way in my scripts, so I can handle clusters of both kinds coexisting in the same namespace on the mgmt cluster.

Cinder-CSI works out of the box with this, which is nice. (I had to do some patching before ...)

Questions:

• I understand that with the usage of clouds.yaml, we have more freedom to configure clusterstackreleases with different settings. This is good.
  I would like to take the decision which loadbalancer provider I use per cluster, for example.
  Is there a way to do this?
• I guess there is some complexity with upgrading clusters with cloud.config configuration to clusters with clouds.yaml configuration, with coexistence of both (in the same mgmt cluster namespace) and with autosubscribe ... I fear we need to create some documentation and/or automation for the upgrade case. Maybe only do this for 1.33.x and leave <= 1.32 alone?

[jschoone]

• I understand that with the usage of clouds.yaml, we have more freedom to configure clusterstackreleases with different settings. This is good.
  I would like to take the decision which loadbalancer provider I use per cluster, for example.
  Is there a way to do this?

Yes, using the overwrite.yaml, options/variables from the Cluster resource can be passed. We had
We can just set the default to ovn.
We had an approach before with the workload_loadbalancer variable and adjusting the values.yaml accordingly, see here. But since we did create=false this was a dummy. Now we are able to use that.

• I guess there is some complexity with upgrading clusters with cloud.config configuration to clusters with clouds.yaml configuration, with coexistence of both (in the same mgmt cluster namespace) and with autosubscribe ... I fear we need to create some documentation and/or automation for the upgrade case. Maybe only do this for 1.33.x and leave <= 1.32 alone?

Can also be done using the overwrite.yaml. e.g. checking which "managed secret" is used and turning on or off the create option depending on that. And yes I would use clouds-yaml in newer versions only.

[jschoone]

Better would be do create new Cluster Stacks. That would follow the idea of Cluster Stacks and no breaking changes exist which needed to be workarounded.
The whole SCS Cluster Stack can then be deprecated and until it reaches EOL the autosubscribe continues working.
Then it's also more clear there are differences which need to be handled which can easily be documented, such as ORC instead of CSPO or the change to use-clouds in the CPO Helm values.

So personally I'd prefer to have new Cluster Stacks.
Especially the use-clouds thing will be very tricky and breaking.
The ORC thing is already implemented, but luckily ORC must be installed anyway since CAPO 0.12 so there nothing breaks.

[jschoone]

Converted to draft since we need to decide where to put this.

[garloff]

Do we need to have this as addon?
I see:

  - error: 'failed to apply object: clusterresourcesets.addons.cluster.x-k8s.io "openstack-scs-1-32-v0-git.0e3e6e8-clouds-yaml"
      is forbidden: User "system:serviceaccount:cso-system:cso-controller-manager"
      cannot patch resource "clusterresourcesets" in API group "addons.cluster.x-k8s.io"
      in the namespace "ciabns2"'
    group: addons.cluster.x-k8s.io
    kind: ClusterResourceSet
    name: openstack-scs-1-32-v0-git.0e3e6e8-clouds-yaml
    status: not-synced
    version: v1beta1

when testing openstack-scs-1-32-v0-git.0e3e6e8. Not sure why it happens, though.
I do create the CRS from a script https://github.com/SovereignCloudStack/scs-training-kaas-scripts/blob/main/07-cluster-secret.sh when creating the secret ...

Isn't the