Reconciling TargetGroup Removes ECS targets causing downtime

[jaguer0]

Describe the bug
I'm using Flux for gitops , using elvb2 to create loadbalancer , targetgroup, listeners & also using aws ack ECS for task definition and service.

When the targetgroup is reconciled , it's removing the targets that were registered from ECS via the targetGroupRef. This leads to the target group having 0 targets and being down. Eventually it comes back online ,

Steps to reproduce
Target group is created , example

apiVersion: elbv2.services.k8s.aws/v1alpha1
kind: TargetGroup
metadata:
  name: foo-bar-tg-staging
spec:
  name: foo-bar-tg-staging
  healthCheckEnabled: true
  healthCheckIntervalSeconds: 30
  healthCheckPath: /
  healthCheckPort: traffic-port
  healthCheckProtocol: HTTP
  healthCheckTimeoutSeconds: 5
  healthyThresholdCount: 5
  ipAddressType: ipv4
  matcher:
    httpCode: "200"
  port: 8080
  protocol: HTTP
  protocolVersion: HTTP1
  targetType: ip
  unhealthyThresholdCount: 2
  vpcID: xxxxx

It is created and working fine, it's referenced in the ECS service using targetGroupRef.

apiVersion: ecs.services.k8s.aws/v1alpha1
kind: Service
metadata:
  name: foo-bar
spec:
  name: foo-bar
  capacityProviderStrategy:
  - base: 0
    capacityProvider: FARGATE
    weight: 1
  cluster: staging
  deploymentConfiguration:
    alarms:
      alarmNames:
      - none
      enable: false
      rollback: false
    deploymentCircuitBreaker:
      enable: true
      rollback: true
    maximumPercent: 200
    minimumHealthyPercent: 100
  deploymentController:
    type: ECS
  desiredCount: 1
  enableECSManagedTags: true
  enableExecuteCommand: false
  healthCheckGracePeriodSeconds: 0
  loadBalancers:
  - containerName: foo-bar
    containerPort: 8080
    targetGroupRef:
      from:
        name: foo-bar-tg-staging
  networkConfiguration:
    awsVPCConfiguration:
      assignPublicIP: DISABLED
      securityGroups:
      - sg-xxxxx
      subnets:
      - sg-xxxxxx
      - sg-xxxxxx
  platformVersion: 1.4.0
  propagateTags: NONE
  schedulingStrategy: REPLICA
  taskDefinitionRef:
    from:
      name: foo-bar-staging

When the ELBv2 controller reconciles the target group, it removes the targets completely causing downtime. about ~15 min later ECS re-adds them and it's back online. It happens daily as the reconciler is set to every 10 hours and leads to downtime.

I do see an event for ECS noting task remained in deregistered state for too long

Log from elbv2 controller

2025-03-11T10:45:53.047941104Z stderr F 
{"level":"info",
"ts":"2025-03-11T10:45:53.047Z",
"logger":"ackrt",
"msg":"desired resource state has changed",
"kind":"TargetGroup",
"namespace":"foo-bar",
"name":"foo-bar-tg-staging",
"account":"xxxx",
"role":"",
"region":"us-west-2",
"is_adopted":false,"generation":1,"diff":[{"Path":{"Parts":["Spec",
"Targets"]},"A":null,"B":[{"availabilityZone":"us-west-2a",
"id":"xx.x.xx.xx","port":8080}]}]
}

you can see the metrics show unhealthy state at the same time as that log entry when it reconciles.

Expected outcome
target group to keep the targets when using ECS & targetGroupRef.

Environment

• Using EKS: Yes v1.29.13-eks-8cce635
• AWS service targeted: ECS, ELBV2

[michaelhtm]

Hi @jaguer0, since the elbv2 controller now manages individual targets, the spec is its source of truth.
Here ECS is adding targets to the targetGroup, which the elbv2 controller does not find in its spec, and will remove them.
As of now there isn't a straightforward workaround, except maybe by adding the target to the Spec.

[jaguer0]

Hey @michaelhtm, what should I set for the ID? Is it expecting an ARN? If so, that seems like it would lead to more hardcoding, which doesn’t align with GitOps principles, especially for services that self-register, like ECS and Auto Scaling. Or if I can do a reference like EcsServiceRef at least

In both CloudFormation and Terraform, you typically define the target group without specifying targets, allowing ECS to handle the self-registration. Here are some examples:

CloudFormation Example
Terraform Example

I looked at the P.R for the targetgroup change & it seems to be more of a use case for lambda, which I understand but I would expect the awk controller to account for this.

Not sure if it’s relevant, but it seems like the ECS ACK controller isn’t appending the targets to the target group spec in the YAML. It looks like this is happening behind the scenes. It might be by design, but perhaps if the controller could inject the targets and mark them as a managed field, another ACK controller might be able to persist them

[michaelvl]

This parallels some of our experiences with using TargetGroups together with EKS Pods #2355 i.e. its rather difficult to use TargetGroups in practice.

[michaelhtm]

Not sure if it’s relevant, but it seems like the ECS ACK controller isn’t appending the targets to the target group spec in the YAML. It looks like this is happening behind the scenes. It might be by design, but perhaps if the controller could inject the targets and mark them as a managed field, another ACK controller might be able to persist them

The ECS ACK controller is not the one appending those targets, rather it is the ECS service doing so behind the scenes.
Once solution that may help is to create the target group with ACK and annotate it to make it readonly. It's an alpha feature you can enable with a feature flag that allows you to reference the targetgroup in any other resource, such as EKS or ECS, and it will not remove any targets.

Here's more info on it: aws-controllers-k8s/runtime#145 https://aws-controllers-k8s.github.io/community/docs/user-docs/features/ https://github.com/aws-controllers-k8s/elbv2-controller/blob/1a79c825e462343f981f2eb35c2ea368df5e0436/helm/values.yaml#L160-L168 https://aws-controllers-k8s.github.io/community/docs/user-docs/features/#readonlyresources

If this is not applicable to your use case, how would you want to see this solved?

[michaelvl]

How about option 3 from #2355, which basically is to allow the TargetGroup definition to specify 'ignore external changes to targets' since this would allow for using external controllers to update the AWS targetgroup targets. This is probably a variation of the 'readonly' option above however limited to the targets part of the TargetGroup.

[michaelhtm]

Yes, this is part of a larger project we are considering, where we allow users to provide fields they want to be read-only, and you can specify the path to it. This would mean the controller would ignore external changes (happening via console or other controllers) and user changes.

Here's an idea of how we can ignore these fields:

annotations:
    services.k8s.aws/read-only-fields: |
         [
                 "Spec.Targets", "Spec.Tags"
         ]

Putting them in annotations is just a suggestion tho and is not final, still trying to find the best approach of what the user experience should be like..
What do you think @a-hilaly @rushmash91

[mkostrze-vs]

We found the same scenario recently (actually related to TargetGroupBindings described in #2355). In this scenario we find that the ACK TargetGroup resource is pretty much unusable (because only static targets are supported, and there is no good workaround I can think of)
Just to confirm, the annotation proposed above (services.k8s.aws/read-only-fields) - it's not implemented yet, is it?

[leonzaher]

Any update/workaround for this issue? We're also managing TargetGroups using ACK and want targets that point to Kubernetes Pods.