Flux 2.5 seems to use old/deprecated versions of API

[erikgb]

After upgrading to Flux 2.5 in one of our early clusters we observed a new type of error. Example:

health check failed after 3m0.016091887s: timeout waiting for: [SubNamespace/strm/strm-kafka-dev2 status: 'Unknown': looking up status.observedGeneration from resource: .status.observedGeneration accessor error: ok is of the type string, expected map[string]interface, SubNamespace/strm/strm-cluster-monitoring status: 'Unknown': looking up status.observedGeneration from resource: .status.observedGeneration accessor error: ok is of the type string, expected map[string]interface, SubNamespace/strm/strm-kafka status: 'Unknown': looking up status.observedGeneration from resource: .status.observedGeneration accessor error: ok is of the type string, expected map[string]interface]

It seems like this error occurs because Flux somehow uses an old/deprecated version of the SubNamespace API. This wasn't an issue on Flux 2.4 and the error seems to occur a bit randomly.

After investigating this a bit more, it seems like kustomize-controller is also using other old/deprecated API versions. This example is from a bootstrap of our "fleet" repo on an ephemeral k3s cluster:

$ k logs -n flux-system kustomize-controller-5995c9c7f6-dg9qc | grep deprecated
{"level":"info","ts":"2025-02-24T11:12:10.399Z","logger":"KubeAPIWarningLogger","msg":"external-secrets.io/v1alpha1 ExternalSecret is deprecated; use external-secrets.io/v1beta1 ExternalSecret"}
{"level":"info","ts":"2025-02-24T11:12:10.404Z","logger":"KubeAPIWarningLogger","msg":"v1beta2 Kustomization is deprecated, upgrade to v1"}
{"level":"info","ts":"2025-02-24T11:12:15.402Z","logger":"KubeAPIWarningLogger","msg":"external-secrets.io/v1alpha1 ExternalSecret is deprecated; use external-secrets.io/v1beta1 ExternalSecret"}
{"level":"info","ts":"2025-02-24T11:12:15.407Z","logger":"KubeAPIWarningLogger","msg":"v1beta2 Kustomization is deprecated, upgrade to v1"}
{"level":"info","ts":"2025-02-24T11:12:20.404Z","logger":"KubeAPIWarningLogger","msg":"external-secrets.io/v1alpha1 ExternalSecret is deprecated; use external-secrets.io/v1beta1 ExternalSecret"}
{"level":"info","ts":"2025-02-24T11:12:20.412Z","logger":"KubeAPIWarningLogger","msg":"v1beta2 Kustomization is deprecated, upgrade to v1"}
{"level":"info","ts":"2025-02-24T11:12:20.930Z","logger":"KubeAPIWarningLogger","msg":"kyverno.io/v2beta1 ClusterCleanupPolicy is deprecated; use kyverno.io/v2 ClusterCleanupPolicy"}
{"level":"info","ts":"2025-02-24T11:12:20.933Z","logger":"KubeAPIWarningLogger","msg":"kyverno.io/v2beta1 ClusterCleanupPolicy is deprecated; use kyverno.io/v2 ClusterCleanupPolicy"}
{"level":"info","ts":"2025-02-24T11:12:25.413Z","logger":"KubeAPIWarningLogger","msg":"external-secrets.io/v1alpha1 ExternalSecret is deprecated; use external-secrets.io/v1beta1 ExternalSecret"}
{"level":"info","ts":"2025-02-24T11:12:25.421Z","logger":"KubeAPIWarningLogger","msg":"v1beta2 Kustomization is deprecated, upgrade to v1"}
{"level":"info","ts":"2025-02-24T11:12:30.406Z","logger":"KubeAPIWarningLogger","msg":"external-secrets.io/v1alpha1 ExternalSecret is deprecated; use external-secrets.io/v1beta1 ExternalSecret"}
{"level":"info","ts":"2025-02-24T11:12:30.415Z","logger":"KubeAPIWarningLogger","msg":"v1beta2 Kustomization is deprecated, upgrade to v1"}
{"level":"info","ts":"2025-02-24T11:22:25.117Z","logger":"KubeAPIWarningLogger","msg":"v1beta1 GitRepository is deprecated, upgrade to v1"}
{"level":"info","ts":"2025-02-24T11:22:25.507Z","logger":"KubeAPIWarningLogger","msg":"kyverno.io/v2beta1 ClusterCleanupPolicy is deprecated; use kyverno.io/v2 ClusterCleanupPolicy"}
{"level":"info","ts":"2025-02-24T11:22:25.515Z","logger":"KubeAPIWarningLogger","msg":"kyverno.io/v2beta1 ClusterCleanupPolicy is deprecated; use kyverno.io/v2 ClusterCleanupPolicy"}

And I have checked that we don't use any of these deprecated API versions in Git.

If re-bootstrapping the ephemeral cluster, the list appear to be different - making me believe there is some sort of randomness happening here.