Unable to upgrade providers when using custom fetchconfig

[arno-pons]

What steps did you take and what happened:

When trying to perform an upgrade of already deployed providers, configured for airgapped environment (custom config map), I have the following error in each provider's controller logs:

capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-23T17:52:29.626242023+02:00 E0723 15:52:29.626100       1 controller.go:329] "Reconciler error" err="failed to get configuration for the InfrastructureProvider with name infrastructure-azure. Please check the provider name and/or add configuration for new providers using the .clusterctl config file" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="5c096dda-b062-4d1b-9fbb-02ec7e27602f"

Here, controller AddonProvider seems to check configuration of an InfrastructureProvider, but I have same error for BootstrapProvider controller for example, and InfrastructureProvider is complaining about BootstrapProvider:

capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:25:39.452452000+02:00 E0724 06:25:39.452326       1 controller.go:329] "Reconciler error" err="failed to get configuration for the BootstrapProvider with name bootstrap-kubeadm. Please check the provider name and/or add configuration for new providers using the .clusterctl config file" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/infrastructure-azure" namespace="capz-system" name="infrastructure-azure" reconcileID="6bbf901c-baf7-412d-83b2-d3d90f37f9b2"

If i delete current provider version and apply new one, this provider is deployed without error, so the fetchConfig seems to be ok.

What did you expect to happen:

Upgrade of providers should happens no matter what kind of configuration is used (custom fetchconfig or "github manifest").

Anything else you would like to add:

Here is a full log about a controller reconciliation showing that it detects that version has changed but fails to perform upgrade:

capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:30.818619253+02:00 I0724 06:47:30.818454       1 genericprovider_controller.go:62] "Reconciling provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:30.818842756+02:00 I0724 06:47:30.818731       1 preflight_checks.go:58] "Performing preflight checks" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:30.819070959+02:00 I0724 06:47:30.819002       1 preflight_checks.go:199] "Preflight checks passed" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:30.819305962+02:00 I0724 06:47:30.819217       1 phases.go:240] "No configuration secret was specified" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:31.117517239+02:00 I0724 06:47:31.117378       1 phases.go:257] "Custom fetch configuration config map was provided" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:31.119239063+02:00 I0724 06:47:31.119171       1 phases.go:161] "Loading provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:31.122121203+02:00 I0724 06:47:31.122021       1 phases.go:420] "Fetching provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:31.419996575+02:00 I0724 06:47:31.419866       1 phases.go:475] "Version changes detected, updating existing components" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:31.627523882+02:00 E0724 06:47:31.627407       1 controller.go:329] "Reconciler error" err="failed to get configuration for the InfrastructureProvider with name infrastructure-azure. Please check the provider name and/or add configuration for new providers using the .clusterctl config file" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"

From my understanding issue come from the fact that known providers list remains the default one. The current controller is only aware of it's own provider custom configuration (see here) where it should have all providers custom infos to be able to check them here in this loop.

Environment:

• Cluster-api-operator version: 0.11.0
• Cluster-api version: 1.7.4
• Minikube/KIND version: N/A
• Kubernetes version: (use kubectl version): v1.28.7
• OS (e.g. from /etc/os-release): Ubuntu 22.04.4 LTS

/kind bug

[furkatgofurov7]

/triage accepted

[furkatgofurov7]

As discussed in the CAPI Operator office hours, we might want to reach out to CAPI Maintainers and discuss the possible solution.

[furkatgofurov7]

/assign @w21froster

[k8s-ci-robot]

@furkatgofurov7: GitHub didn't allow me to assign the following users: w21froster.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @w21froster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[dmvolod]

Probably, I understand the nature of this problem.
The combination of provider name and type must match the name of label cluster.x-k8s.io/provider in the manifest.
But in this case, seems to names are:

• for infrastructure azure name="infrastructure-azure", but should be azure
• for addon helm name="addon-helm", but should be helm

It's necessary to add a validation at the manifest parsing phase, for the following rule:
provider name should be the value of the cluster.x-k8s.io/provider label from manifest, cutting provider type prefix.

As alternative way, we could get a provider name from the manifest cluster.x-k8s.io/provider label and able to use any resource object name.

Will take care about this issue.

P.S. For example, if it validation is not passed, I can install any type of provider, i.e. helm addon as any type of the operator custom resource, as Infrastructure or Core, that's wrong behavior.

[dmvolod]

/assign

[w21froster]

@dmvolod sorry haven't had time to look at this recently. I went ahead and tried your suggestion though. I had used a different name for the Provider CR than was defined under cluster.x-k8s.io/provider in the YAML, so I was hopeful this would end up fixing the issue. But it looks like that didn't end up solving it (logs below).

E0222 00:07:36.107829       1 controller.go:316] "Reconciler error" err="failed to get configuration for the BootstrapProvider with name bootstrap-kubeadm. Please check the provider name and/or add configuration for new providers using the .clusterctl config file" controller="controlplaneprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="ControlPlaneProvider" ControlPlaneProvider="capi-kubeadm-control-plane-system/control-plane-kubeadm" namespace="capi-kubeadm-control-plane-system" name="control-plane-kubeadm" reconcileID="fcea1238-5d64-45b2-92e5-bd776bd0e828"
E0222 00:07:36.712165       1 controller.go:316] "Reconciler error" err="failed to get configuration for the ControlPlaneProvider with name control-plane-kubeadm. Please check the provider name and/or add configuration for new providers using the .clusterctl config file" controller="bootstrapprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="BootstrapProvider" BootstrapProvider="capi-kubeadm-bootstrap-system/bootstrap-kubeadm" namespace="capi-kubeadm-bootstrap-system" name="bootstrap-kubeadm" reconcileID="31652b90-74a9-4ba7-a1f3-678805860929"
E0222 00:07:43.900014       1 controller.go:316] "Reconciler error" err="failed to get configuration for the BootstrapProvider with name bootstrap-kubeadm. Please check the provider name and/or add configuration for new providers using the .clusterctl config file" controller="coreprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="CoreProvider" CoreProvider="capi-system/cluster-api" namespace="capi-system" name="cluster-api" reconcileID="4c299979-5dfa-4d87-864d-e5d57d0fd13c"

From what I understand, clusterctl runs a preflight check before upgrading any Provider to make sure all versions are compatible before upgrading so you don't reach an invalid state. I think what @arno-pons was mentioning was that we need to make clusterctl aware of all of the other providers using fetchConfig. If you notice on line 1, it shows that the controller complaining about the BootstrapProvider is the controlplaneprovider controller, which leads me to believe that we indeed need to pass the configuration of other fetchConfig providers to clusterctl. I went ahead and made a preliminary PR, and it looks like it may have fixed the problem (not sure if there are any other consequences to this approach though).