Security Self Assessment: [STRIDE-INFODISCLOSE-1] RFE cluster addons

[fabriziopandini]

User Story

As a user/operator I would like Cluster API to get a comprehensive solution for Cluster addons lifecycle

Detailed Description

This builds up on #4166 and from recent discussions at SIG level.

There is a set of addons that have a lifecycle strictly linked to the cluster lifecycle managed by CAPI, afterwards Cluster addons.

Some of this addons should be lifecycle managed according with a combination of following requirements that can differ from addon to addon:

• They inherits some configurations from cluster configuration (e.g service or pod CIDR)
• They should be created during cluster creation (e.g immediately after the API server is installed)
• They should be upgraded before/during/after cluster (and more specifically control plane) upgrade
• There should be support support for out-of band upgrade, e.g. for CVE fix (not linked to the cluster lifecycle)
• They should be deleted before/during cluster deletion
• Possibly more...

Current answer for this problem space is CustomResourceSets, but this covers only some of the requirements above.
However this is falling short now that the number and the needs of Cluster addons are growing due to CSI/CPI plugins being moved out of three.

On top of that, most users have their own solution for addon management, and we should consider if/how to integrate with those solutions too

IMPORTANT: we are not seeking to reinvent an addon management solution with this issue, but instead we should focus on finding a way to lifecycle manage addons within the context of CAPI Cluster lifecycle

/kind feature

[vincepri]

/milestone v1.1
/priority important-soon

Bucketing in v1.1, although this probably needs more time for proposal, etc

[randomvariable]

Just adding to this one related to #3176. In EKS, GKE, AKS, VMware TKG-I (formerly Enterprise PKS), and VMware vSphere with Tanzu (formerly TKG-S), workload clusters do not require access to the cloud provider APIs. In the case of everything but TKG-S, cloud provider integrations and CSI are provisioned externally to the cluster and exist as a black box as far as the workload cluster goes. This prevents users of a workload cluster from stealing cloud provider credentials to cause damage on the underlying infrastructure, including elevation of privilege (workload --> management cluster) and arbitrary access to VMs and block storage.

We may want to investigate if certain addons can be run on the management cluster, or whether this should be a feature of CAPN, where CAPI can optionally generate workload cluster kubeconfigs for addons running on the mgmt cluster.

This is noted in the security self-assessment STRIDE-INFODISCLOSE-1.

[randomvariable]

/area security