Fix orphaned AWS NLB on exposure loadbalancer->traefik switch #851

Tracking issue for the agentic fix in #850 (upstream bug report: crate/cloud#2976).

Problem

Switching spec.cluster.exposure from loadbalancer to traefik patches the data service LoadBalancer -> ClusterIP but, in the same patch, also stripped service.beta.kubernetes.io/aws-load-balancer-type: nlb. The in-tree service controller needs that annotation in EnsureLoadBalancerDeleted to recognise the LB as an NLB; without it, it takes the Classic ELB path, fails to find the LB, removes the load-balancer-cleanup finalizer, and orphans the NLB on AWS.

Investigation / decisions

• Confirmed via the captured service in crate/cloud#2976: finalizer is service.kubernetes.io/load-balancer-cleanup (in-tree service controller, KCM or standalone CCM) — not the AWS Load Balancer Controller (service.k8s.aws/resources). So the annotation-preservation fix applies.
• Chose Option A (keep the annotation), not delete-and-recreate the service. Avoids the recreate race (must wait for the LB to be fully gone), a new clusterIP/nodePorts, and external-dns churn. Smallest blast radius.
• NLB vs Classic ELB: only NLB is affected (the annotation is the only NLB signal); Classic ELB is the controller's default delete path and was never broken. The operator always tags AWS LBs as nlb, so the fix covers all current LBs.

Change

• _lb_annotations_to_remove() no longer nulls aws-load-balancer-type.
• Test pairing add-on-create with retain-on-switch.

Follow-ups (out of scope here)

• One-time manual cleanup of already-orphaned NLBs on AWS.
• Real EKS smoke test to confirm the NLB is deleted (minikube cannot exercise this path).

LLM assisted: Claude Opus 4.8 (1M context)