Infinite Fleet Cluster patching loop

[anmazzotti]

Occasionally, when the fleetaddonconfig.spec.cluster.patchResource is true, CAAPF will be stuck in a deadlock continuously patching the Fleet Cluster multiple times a second.

2025-05-28T11:04:13.368719Z  INFO reconciling object:reconcile:patch: controller::controllers::controller: Updated object object.ref=Cluster.v1beta1.cluster.x-k8s.io/pvala-capz-kubeadm-single.default object.reason=related object updated: Cluster.v1alpha1.fleet.cattle.io/pvala-capz-kubeadm-single.default name="pvala-capz-kubeadm-single" namespace="default" reconcile_id=853a9720607e2ff8c16d2a046445d80c name="pvala-capz-kubeadm-single" namespace="default" api_version="fleet.cattle.io/v1alpha1" kind="Cluster"
2025-05-28T11:04:13.405243Z  INFO reconciling object:reconcile:patch: controller::controllers::controller: Updated object object.ref=Cluster.v1beta1.cluster.x-k8s.io/pvala-capz-kubeadm-single.default object.reason=related object updated: Cluster.v1alpha1.fleet.cattle.io/pvala-capz-kubeadm-single.default name="pvala-capz-kubeadm-single" namespace="default" reconcile_id=8e3e9d7d1d3d9d661a05a3f0a2482720 name="pvala-capz-kubeadm-single" namespace="default" api_version="fleet.cattle.io/v1alpha1" kind="Cluster"
2025-05-28T11:04:13.448025Z  INFO reconciling object:reconcile:patch: controller::controllers::controller: Updated object object.ref=Cluster.v1beta1.cluster.x-k8s.io/pvala-capz-kubeadm-single.default object.reason=related object updated: Cluster.v1alpha1.fleet.cattle.io/pvala-capz-kubeadm-single.default name="pvala-capz-kubeadm-single" namespace="default" reconcile_id=0c0a5092af425cb5530ac2753c206b1f name="pvala-capz-kubeadm-single" namespace="default" api_version="fleet.cattle.io/v1alpha1" kind="Cluster"
2025-05-28T11:04:13.499983Z  INFO reconciling object:reconcile:patch: controller::controllers::controller: Updated object object.ref=Cluster.v1beta1.cluster.x-k8s.io/pvala-capz-kubeadm-single.default object.reason=related object updated: Cluster.v1alpha1.fleet.cattle.io/pvala-capz-kubeadm-single.default name="pvala-capz-kubeadm-single" namespace="default" reconcile_id=c0c64692fc94acbe2a5c063c7548b35c name="pvala-capz-kubeadm-single" namespace="default" api_version="fleet.cattle.io/v1alpha1" kind="Cluster"
2025-05-28T11:04:13.549389Z  INFO reconciling object:reconcile:patch: controller::controllers::controller: Updated object object.ref=Cluster.v1beta1.cluster.x-k8s.io/pvala-capz-kubeadm-single.default object.reason=related object updated: Cluster.v1alpha1.fleet.cattle.io/pvala-capz-kubeadm-single.default name="pvala-capz-kubeadm-single" namespace="default" reconcile_id=a8a5df77bf4a1123370b0ac5ef2514c5 name="pvala-capz-kubeadm-single" namespace="default" api_version="fleet.cattle.io/v1alpha1" kind="Cluster"
2025-05-28T11:04:13.591034Z  INFO reconciling object:reconcile:patch: controller::controllers::controller: Updated object object.ref=Cluster.v1beta1.cluster.x-k8s.io/pvala-capz-kubeadm-single.default object.reason=related object updated: Cluster.v1alpha1.fleet.cattle.io/pvala-capz-kubeadm-single.default name="pvala-capz-kubeadm-single" namespace="default" reconcile_id=935bc68bed8f6c10ec68bea091a3e45d name="pvala-capz-kubeadm-single" namespace="default" api_version="fleet.cattle.io/v1alpha1" kind="Cluster"

I did not see any changes in the Cluster resource itself.
The issue also stops if CAAPF is disabled for a while (for example setting 0 replicas), even with patchResource: true.

The side effect, other than CPU consumption, is Fleet creating and deleting continuously the fleet-agent Bundle before it can be applied downstream.

[anmazzotti]

So far this issue seems to be reproducible only on k3s.
Thanks to @valaparthvi it was tested on different versions: v1.31.4, v1.32.5, and it was originally reported on v1.30.10.

It is most likely related to the server-side apply patch from CAAPF.
I still can not understand the conflict however.

Interestingly, on k3s I get a different behavior when I apply any custom resource with kubectl:

Warning: resource helmapps/calico-cni is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.

Does the conflict originate on the last-applied-configuration annotation maybe?

[anmazzotti]

On a possibly related note, there seem to be an issue with the fleet.cattle.io/v1alpha1 last-applied-configuration, being a cluster.x-k8s.io/v1beta1 resource:

apiVersion: fleet.cattle.io/v1alpha1
kind: Cluster
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cluster.x-k8s.io/v1beta1","kind":"Cluster","metadata":{"annotations":{},"labels":{"cloud-provider":"azure","cluster-api.cattle.io/rancher-auto-import":"true","cni":"calico"},"name":"azure-quickstart","namespace":"default"},"spec":{"clusterNetwork":{"pods":{"cidrBlocks":["192.168.0.0/16"]}},"topology":{"class":"azure-kubeadm-example","controlPlane":{"replicas":1},"variables":[{"name":"subscriptionID","value":"foobar"},{"name":"location","value":"germanywestcentral"},{"name":"resourceGroup","value":"andrea-capi-playground"},{"name":"azureClusterIdentityName","value":"cluster-identity"}],"version":"v1.31.1","workers":{"machineDeployments":[{"class":"kubeadm-default-worker","name":"md-0","replicas":1}]}}}}

This is a result of annotation propagation, but I think it's going to lead to issues. CAAPF should most likely skip propagating this one (and including it into the Fleet Cluster spec as well).
This probably applies to other auto-generated annotations that should be filtered.

[Danil-Grigorev]

Note 1: The side-effect of CPU consumption is mitigated by establishing resource limits in CAAPF deployment: #282

Note 2: When it comes to k3s specific problem with SSA patch, a workaround is to unset patchResource option in the FleetAddonConfig. Snippet from the docs: https://rancher.github.io/cluster-api-addon-provider-fleet/04_reference/03_fleet-addon-config.html#spec

cluster.patchResource

• Description: Allow to patch resources, maintaining the desired state. If is not set, resources will only be re-created in case of removal.
• Type: boolean
• Optional: Yes

Example:

spec:
  cluster:
    patchResource: false

[anmazzotti]

For reference, it is possible to reproduce this issue by deploying a Turtles UI e2e environment, using the related action.
For example I was able to successfully reproduce it on: https://github.com/rancher/rancher-turtles-e2e/actions/runs/15385979760
I did it with an Azure Kubeadm Cluster, using the Turtles ClusterClass. All resources applied by kubectl (Class, HelmApps, Cluster).
Once the (CAPI) Cluster is provisioned successfully and the (Fleet) Cluster is created, the fleet-agent Bundle will constantly appear and disappear.

Also if it can help, this is a Fleet Cluster resource affected by the issue. The resourceVersion keeps increasing with every patch.
There are no spec changes however, only the managedFields and status are updated.
Note that this test case was reproduced with a #314 build, showing that the kubectl.kubernetes.io/last-applied-configuration are no longer propagated. This patch unfortunately did not solve the issue reported here.

cluster-fleet-dump.yaml.txt

[Danil-Grigorev]

I was able to reproduce the issue on GKE cluster in turtles e2e suite. Here is the source of the resource version changes - it raises due to managedFields getting a diff on time value for the CAAPF applied patch:

[
    [
        {
            "op": "replace",
            "path": "/metadata/managedFields/0/time",
            "value": "2025-06-05T08:55:01Z" <- this value changes each time
        },
        {
            "op": "add",
            "path": "/metadata/managedFields/2/fieldsV1/f:status/f:summary/f:nonReadyResources",
            "value": {}
        },
        {
            "op": "add",
            "path": "/metadata/managedFields/2/fieldsV1/f:status/f:summary/f:waitApplied",
            "value": {}
        },
        {
            "op": "replace",
            "path": "/metadata/managedFields/2/time",
            "value": "2025-06-05T08:55:02Z"
        },
        {
            "op": "replace",
            "path": "/metadata/resourceVersion",
            "value": "547172"
        },
        {
            "op": "replace",
            "path": "/status/conditions/1/lastUpdateTime",
            "value": "2025-06-05T08:55:02Z"
        },
        {
            "op": "add",
            "path": "/status/conditions/1/message",
            "value": "WaitApplied(1) [Bundle fleet-agent-cluster-gke-44fgdv]"
        },
        {
            "op": "replace",
            "path": "/status/conditions/1/status",
            "value": "False"
        },
        {
            "op": "replace",
            "path": "/status/display/readyBundles",
            "value": "0/1"
        },
        {
            "op": "replace",
            "path": "/status/summary/desiredReady",
            "value": 1
        },
        {
            "op": "add",
            "path": "/status/summary/nonReadyResources",
            "value": [
                {
                    "bundleState": "WaitApplied",
                    "name": "fleet-agent-cluster-gke-44fgdv"
                }
            ]
        },
        {
            "op": "add",
            "path": "/status/summary/waitApplied",
            "value": 1
        }
    ],
    [
        {
            "op": "replace",
            "path": "/metadata/managedFields/0/time",
            "value": "2025-06-05T08:55:02Z"
        },
        {
            "op": "add",
            "path": "/metadata/managedFields/2/fieldsV1/f:status/f:summary/f:nonReadyResources",
            "value": {}
        },
        {
            "op": "add",
            "path": "/metadata/managedFields/2/fieldsV1/f:status/f:summary/f:waitApplied",
            "value": {}
        },
        {
            "op": "replace",
            "path": "/metadata/managedFields/2/time",
            "value": "2025-06-05T08:55:02Z"
        },
        {
            "op": "replace",
            "path": "/metadata/resourceVersion",
            "value": "547173"
        },
        {
            "op": "replace",
            "path": "/status/conditions/1/lastUpdateTime",
            "value": "2025-06-05T08:55:02Z"
        },
        {
            "op": "add",
            "path": "/status/conditions/1/message",
            "value": "WaitApplied(1) [Bundle fleet-agent-cluster-gke-44fgdv]"
        },
        {
            "op": "replace",
            "path": "/status/conditions/1/status",
            "value": "False"
        },
        {
            "op": "replace",
            "path": "/status/display/readyBundles",
            "value": "0/1"
        },
        {
            "op": "replace",
            "path": "/status/summary/desiredReady",
            "value": 1
        },
        {
            "op": "add",
            "path": "/status/summary/nonReadyResources",
            "value": [
                {
                    "bundleState": "WaitApplied",
                    "name": "fleet-agent-cluster-gke-44fgdv"
                }
            ]
        },
        {
            "op": "add",
            "path": "/status/summary/waitApplied",
            "value": 1
        }
    ]
]

[Danil-Grigorev]

Have a fix for the issue in progress, but observing an interesting error in the fleet-controller pod while it is trying to apply fleet-agent bundle on the GKE cluster:

time="2025-06-05T13:44:53Z" level=error msg="error syncing 'creategitops-86kcjc/cluster-gke-44fgdv': handler import-cluster: failed to create cattle-fleet-system-fleet-agent-role rbac.authorization.k8s.io/v1, Kind=ClusterRole for fleet-agent-bootstrap-cattle-fleet-system: clusterroles.rbac.authorization.k8s.io is forbidden: User \"103653002940121800031\" cannot create resource \"clusterroles\" in API group \"rbac.authorization.k8s.io\" at the cluster scope: requires one of [\"container.clusterRoles.create\"] permission(s)., failed to create cattle-fleet-system-fleet-agent-role-binding rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding for fleet-agent-bootstrap-cattle-fleet-system: clusterrolebindings.rbac.authorization.k8s.io is forbidden: User \"103653002940121800031\" cannot create resource \"clusterrolebindings\" in API group \"rbac.authorization.k8s.io\" at the cluster scope: requires one of [\"container.clusterRoleBindings.create\"] permission(s)., requeuing"