Get cacert into workload cluster

[jschoone]

/kind feature

Describe the solution you'd like
[A clear and concise description of what you want to happen.]

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Kubernetes version: (use kubectl version)
• OS (e.g. from /etc/os-release):

[garloff]

Half the solution is to add --set cacert="$(cat /path/to/single-ca-cert.pem)" on the helm call.
This gets capo to work with the custom CA.

Unfortunately not OCCM. OCCM not coming up still leaves you with a pretty disfunctional cluster.

[garloff]

To get OCCM to work, we probably need to tweak ca-file AND ensure we somehow get the secret mounted into OCCM's file system so we can reference it. That probably means tweaking the OCCM pod deployment file ...
(A working workaround is to change cloud.conf and add in tls-insecure=true in the helm _helpers.tpl -- not recommended of course!)

[garloff]

https://github.com/SovereignCloudStack/k8s-cluster-api-provider/blob/main/doc/usage/custom-ca.md might help here.

[garloff]

See also SovereignCloudStack/openstack-csp-helper#8

[garloff]

OK, so here's the analysis:

• The helm openstack-csp-helper creates a cloud-config secret with cacert and cloud.conf entries (files)[*] and the cloud.conf references the cacert file with a line ca-file=/etc/config/cacert IF --set cacert=... is passed during secret creation.
• That reference in cloud.conf obviously only works if the cloud-config secret is mounted to /etc/config/ which is what OCCM does.
• Cinder-CSI mounts it to /etc/kubernetes/, so the reference does not work.

[*] There is also an obsolete cloudProvider.conf file (identical to cloud.conf) that we no longer need.

Options:

1. Ensure that the secret gets mounted at the same directory in both cases.
2. Use a relative path IF this is supported.

I have tested 1 successfully by mounting to /etc/config/ also in CSI.

[garloff]

Additional comments:

• Submitted PR to align upstream helm chart.
• Workaround provided for scs-training-kaas-scripts in https://github.com/SovereignCloudStack/scs-training-kaas-scripts/blob/main/09-fixup-cinder.sh

@jschoone also suggested an alternative approach where we don't generate the cloud.conf using the helm helper chart, but submit a clouds.yaml to OCCM and let it use it with secret.create=true and cloudConfig.global.use-clouds=true. This may or may not address this issue also, depending how the cacert is passed and handled in that approach.

[jschoone]

Additional comments:

• Submitted PR to align upstream helm chart.
• Workaround provided for scs-training-kaas-scripts in https://github.com/SovereignCloudStack/scs-training-kaas-scripts/blob/main/09-fixup-cinder.sh

@jschoone also suggested an alternative approach where we don't generate the cloud.conf using the helm helper chart, but submit a clouds.yaml to OCCM and let it use it with secret.create=true and cloudConfig.global.use-clouds=true. This may or may not address this issue also, depending how the cacert is passed and handled in that approach.

I added that approach in #225. This does not exactly fix this issue yet, cacert must be added, I will add comments in the PR, where this might be placed