Refactor folders and remove CRS

[olga-mir]

Re-reviewing again approach outlined in this repo https://github.com/fluxcd/flux2-multi-tenancy I think I got it wrong. In my case clusters were no more than just pointers to kustomizations in infrastructure. infrastructure in this example as I understand it now is a collection of "lego" blocks, components that run on the clusters (not "payload"), and clusters is the assembly of the lego blocks. I also find infrastructure word a bit confusing in this context and I think that platform is more appropriate.

ClusterResourceSet is an experimental feature and was never meant to stay in CAPI, however currently there is no fully integrated solution to take its place. This is likely to be that solution one day: https://github.com/Jont828/cluster-api-addon-provider-helm
This will be used in this repo to install Cilium on workload clusters (including the permanent management cluster). It can't be installed with https://fluxcd.io/flux/components/helm/helmreleases/#remote-clusters--cluster-api because it needs dynamic value (api server host and port) which is only known at the runtime. CAAPH can handle this.

Note on cluster index

Cluster index is required for Cilium mesh, but current CAAPH implementation doesn't seem to be able to extract index from the labels.

Moreover, if using labels as a way to pass index there is type mismatch since labels are strings, but Cilium expects it as an integer.

The workaround for now is to duplicate HelmReleaseProxy and hard-code the index :(

id: {{ .ControlPlane.metadata.name | strings.TrimPrefix "cluster-0" }}
failed because "strings" not defined.

id: {{ .ControlPlane.metadata.labels.clusterindex }}
this works, but cilium needs it as integer, while labels need to be strings

[olga-mir]

minimal workflow is working. (permanent cluster-mgmt and 1 workload cluster-01, fully automated)

Clusters

% kubectx | head
cluster-01-admin@cluster-01
cluster-mgmt-admin@cluster-mgmt
docker-desktop
kind-kind

cluster-mgmt

% kubectx cluster-mgmt-admin@cluster-mgmt
Switched to context "cluster-mgmt-admin@cluster-mgmt".

% k get clusters -A
NAMESPACE      NAME           PHASE         AGE     VERSION
cluster-01     cluster-01     Provisioned   9m43s
cluster-mgmt   cluster-mgmt   Provisioned   12m

k8s-multi-cluster % flux get all -A
NAMESPACE       NAME                            REVISION        SUSPENDED       READY   MESSAGE
flux-system     gitrepository/flux-system       develop/a7113c6 False           True    stored artifact for revision 'develop/a7113c6fe24ab4258d0f97ba5fd50c908fafeb73'

NAMESPACE       NAME                            REVISION        SUSPENDED       READY   MESSAGE
cluster-01      kustomization/flux-remote       develop/a7113c6 False           True    Applied revision: develop/a7113c6
flux-system     kustomization/caaph             develop/a7113c6 False           True    Applied revision: develop/a7113c6
flux-system     kustomization/caaph-cni         develop/a7113c6 False           True    Applied revision: develop/a7113c6
flux-system     kustomization/flux-system       develop/a7113c6 False           True    Applied revision: develop/a7113c6

% k get po -A | grep -E "flux-system|cilium"
cilium                              cilium-6sf7r                                                              1/1     Running   0          31m
cilium                              cilium-kt6wf                                                              1/1     Running   0          31m
cilium                              cilium-operator-747b974bf5-5xddz                                          1/1     Running   0          31m
cilium                              cilium-operator-747b974bf5-pgls8                                          1/1     Running   0          31m
cilium                              hubble-relay-69b687f7c9-znhvc                                             1/1     Running   0          31m
cilium                              hubble-ui-5dc4d884b6-gcgm5                                                2/2     Running   0          31m
flux-system                         helm-controller-b95c4d69b-qf9pg                                           1/1     Running   0          17m
flux-system                         kustomize-controller-5c6c9f7dcc-mwpzd                                     1/1     Running   0          17m
flux-system                         notification-controller-5cd6f76fbb-4t69x                                  1/1     Running   0          17m
flux-system                         source-controller-84f6bcbfb8-wfn7x                                        1/1     Running   0          17m

cluster-01

% c cluster-01-admin@cluster-01
Switched to context "cluster-01-admin@cluster-01".

% k get po -A | grep -E "flux-system|cilium"
cilium        cilium-9t64n                                                             1/1     Running   0          6m43s
cilium        cilium-operator-5d67db599f-px4j9                                         1/1     Running   0          6m43s
cilium        cilium-operator-5d67db599f-v6qr9                                         1/1     Running   0          6m43s
cilium        cilium-vttj7                                                             1/1     Running   0          6m2s
cilium        hubble-relay-69b687f7c9-h7pdj                                            1/1     Running   0          6m43s
cilium        hubble-ui-5dc4d884b6-7b5b7                                               2/2     Running   0          6m43s
flux-system   helm-controller-b95c4d69b-g8slz                                          1/1     Running   0          5m58s
flux-system   kustomize-controller-5c6c9f7dcc-4znpq                                    1/1     Running   0          5m58s
flux-system   notification-controller-5cd6f76fbb-vctsz                                 1/1     Running   0          5m58s
flux-system   source-controller-84f6bcbfb8-rnnxx                                       1/1     Running   0          5m58s