feat: template Cilium, CAPI and other resources

[olga-mir]

Clean up and re-org config, template Cilium and CAPI resources and fix the complete 3 cluster setup e2e

Installation

deploy permanent management cluster on AWS (using temp kind cluster and then pivot)

./scripts/deploy.sh

flux on management cluster will apply CAPI manifests that are currently present in the repo.

When script is complete run script to finalize workload clusters (install cilium which currently is not vi CRS - due to dynamic KAS address) and flux secret (WIP to eliminate this step).
This script without arguments will discover all workload clusters and perform all necessary adjustments:

./scripts/workload-cluster.sh

Adding a new cluster

Hands free with just one script!

To add a new cluster create config env for it by copying existing file (./config/cluster-<num>.env) and modifying values. This is intended to be manual as script can't or shouldn't guess this values (or too difficult in bash like calc next CIDR)

./scripts/workload-cluster.sh -n cluster-02

This will generate all necessary files and add the cluster to mgmt kustomization list too. Then it will be pushed to the repo (example commit from the script: 92ee7e0), where flux will apply it and capi will provision it. The ./scripts/workload-cluster.sh is still waiting for the cluster to come up and finalize the installation.

on mgmt cluster:

% k get cluster -A
NAMESPACE      NAME           PHASE          AGE   VERSION
cluster-01     cluster-01     Provisioned    12m
cluster-02     cluster-02     Provisioning   60s
cluster-mgmt   cluster-mgmt   Provisioned    13m

Hubble

Current setup enables hubble relay and UI.

cilium hubble ui

https://docs.cilium.io/en/v1.11/gettingstarted/hubble/#hubble-ui
or in CLI with instructions from here: https://docs.cilium.io/en/v1.11/gettingstarted/hubble_setup/