EC2 controller broken after last upgrade to v1.0.5

[gecube]

Several days ago I begin to observe the next lines in logs of EC2 controller:

I0911 13:31:25.944518       1 request.go:690] Waited for 1.009831942s due to client-side throttling, not priority and fairness, request: GET:https://172.20.0.1:443/apis/mq.services.k8s.aws/v1alpha1?timeout=32s
2023-09-11T13:31:27.748Z	INFO	controller-runtime.metrics	Metrics server is starting to listen	{"addr": "0.0.0.0:8080"}
2023-09-11T13:31:27.834Z	INFO	setup	initializing service controller	{"aws.service": "ec2"}
2023-09-11T13:31:33.846Z	INFO	setup	starting manager	{"aws.service": "ec2"}
I0911 13:31:33.846476       1 leaderelection.go:248] attempting to acquire leader lease ack-system/ack-ec2.services.k8s.aws...
2023-09-11T13:31:33.846Z	INFO	Starting server	{"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
E0911 13:31:33.847855       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:31:36.120295       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:31:40.410968       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:31:43.712175       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:31:46.449041       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:31:48.581489       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:31:52.963643       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:31:55.680223       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:31:59.877513       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:32:04.251046       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:32:06.894045       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:32:10.942565       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:32:14.252065       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:32:18.543090       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:32:21.019628       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:32:24.974495       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"
E0911 13:32:27.450358       1 leaderelection.go:330] error retrieving resource lock ack-system/ack-ec2.services.k8s.aws: leases.coordination.k8s.io "ack-ec2.services.k8s.aws" is forbidden: User "system:serviceaccount:ack-system:ack-ec2-controller" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "ack-system"

In fact it does not working as it can't get the lease, so it doesn't go to the reconciliation loop.
I am observing this behaviour from the 7th of September.

I think that it is linked to the update here: https://github.com/aws-controllers-k8s/ec2-controller/blob/685cd7e0fec41dc0d7c9563f3e1c8e0a69a4c52d/helm/values.yaml#L128

So new values were introduced to the helm chart. It looks like that I forget to disable auto-update, but I want to emphasise that such an update must not break anything. So probably the correct value must be:

leaderElection:
  # Enable Controller Leader Election. Set this to true to enable leader election
  # for this controller.
  enabled: true
  # Leader election can be scoped to a specific namespace. By default, the controller
  # will attempt to use the namespace of the service account mounted to the Controller
  # pod.
  namespace: ""

as the binary itself expects the lease.

[gecube]

The same applies to other controllers like ECR, EKS, S3, and others, that were updated last week.

[govindj22]

I think the issue is because of typo at this location in code-generator repository.

I think this line should be changed to - --enable-leader-election flag=$(ENABLED_LEADER_ELECTION)
The env variable should be ENABLED instead of ENABLE.

[a-hilaly]

@govindj22 @gecube thank you for reporting this - we just merged aws-controllers-k8s/code-generator#466 . and we are aiming before EOW to get a new release.

[a-hilaly]

ec2-controller 1.0.6 should be live in few minutes and the rest of the controller will follow very soon. @gecube @govindj22 could any of you confirm that it's working properly?

[gecube]

With leaderElection.enabled=true it's working fine.

Also it looks like that this parameter must be true in any case. For instance, we are trying to roll out new version of EC2 controller. It means that during rollout two copies will simultaneously work in the cluster. As deployment by default is using rolling update strategy:

  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

Also I tried with removed leaderElection.enabled value from Helm values, and again - controller is working.

[a-hilaly]

Also it looks like that this parameter must be true in any case.

@gecube does turning it to false breaks the pods of the deployment (the functionality of the controller regarding leases/leader-election) ?

We currently have ready releases for all the ACK controllers, but i wanted to check with you before trigger the releases.

[gecube]

@a-hilaly it works with false, at least I did not notice any error messages, but I did not do a comprehensive testing. Sorry

[a-hilaly]

That's enough info - thanks a lot @gecube

I just wanted to make sure that we did not release any "hard" regressions with the new helm charts.
We've done some testing internally and all seems to be working fine.

For now, I will trigger releases for IAM and EKS. And will do the rest by monday morning...