Skip to content

Keep aws-load-balancer-type when switching exposure to traefik - #850

Merged
WalBeh merged 1 commit into
masterfrom
fix/lb-orphan-on-traefik-switch
Jun 30, 2026
Merged

WalBeh merged 1 commit into
masterfrom
fix/lb-orphan-on-traefik-switch

Conversation

@WalBeh

@WalBeh WalBeh commented Jun 29, 2026 •

Copy link
Copy Markdown
Contributor

What

When spec.cluster.exposure is switched from loadbalancer to traefik, the operator patches the data service from type: LoadBalancer to type: ClusterIP in place. That same patch also nulled out every LB annotation — including service.beta.kubernetes.io/aws-load-balancer-type: nlb.

The in-tree service controller reads that annotation in EnsureLoadBalancerDeleted to distinguish an NLB from a Classic ELB. With it gone, the controller falls back to the Classic ELB API, can't find the NLB, removes the service.kubernetes.io/load-balancer-cleanup finalizer anyway, and the NLB is left orphaned on AWS.

Fix

Retain aws-load-balancer-type through the switch so the controller correctly identifies and deletes the NLB. The annotation is inert on a ClusterIP service (the controller only acts when type == LoadBalancer), so leaving it is safe. The remaining tuning annotations (connection draining, idle timeout, cross-zone) and the external-dns hostname are still removed.

This is also correct for legacy Classic-ELB clusters: with no annotation present the controller already uses the Classic ELB path, which deletes correctly.

Testing

  • pre-commit (black, flake8, isort, mypy, yamllint) passes.
  • Added TestServiceModels::test_lb_type_annotation_retained_when_switching_to_clusterip, pairing with the existing test_get_data_service which asserts the annotation is added on create.

Caveats for the reviewer

  • Not verifiable on minikube. The actual NLB deletion is AWS in-tree-controller behavior; the test asserts the annotation contract, not the AWS-side delete. Worth a real EKS smoke test before release.
  • Not retroactive. This prevents future orphans. NLBs already orphaned by the previous behavior need a one-time manual cleanup on AWS.

Refs: crate/cloud#2976, kubernetes/kubernetes#95042 #851

Switching `exposure: loadbalancer -> traefik` patches the data service from
type LoadBalancer to ClusterIP. We also nulled every LB annotation in that
patch, including `service.beta.kubernetes.io/aws-load-balancer-type: nlb`.

The in-tree service controller reads that annotation in
EnsureLoadBalancerDeleted to tell an NLB from a Classic ELB. Without it the
controller takes the Classic ELB path, fails to find the NLB, removes the
cleanup finalizer anyway, and leaves the NLB orphaned on AWS.

Retain the type annotation through the switch (it is inert on a ClusterIP
service) so Kubernetes deletes the NLB. The remaining tuning annotations are
still removed.

LLM assisted: Claude Opus 4.8 (1M context)
@WalBeh
WalBeh merged commit bba8b1c into master Jun 30, 2026
16 checks passed
@WalBeh
WalBeh deleted the fix/lb-orphan-on-traefik-switch branch June 30, 2026 12:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants