Using NodePort leaves proxy service externalIPs empty

[tommie]

Description:
I'm trying to run Gateway on bare metal with just public-IP nodes.
I.e. no load balancer or floating IPs. Part of #1035.

When creating a Gateway without spec.addresses and using NodePort for proxy services, I would expect the rendered service to have externalIPs set to the external IPs of the nodes where the pods are currently running. Right now, they are empty, likely because [1] and [2] means they are only set from GatewaySpec.addresses (this was recently added, as a start.) This should makeGatewayStatus.addresses empty, but it's actually populated with all node addresses [3]? This leads to ExternalDNS adding nodes to RRs, that have nothing to do with the Gateway.

Edit: If GatewaySpec.addresses are provided, the IP addresses in GatewayStatus.addresses end up with duplicates, because of [3] and then also adding the Service externalIPs. But until I understand [3], I don't know what that function should look like.

In the other direction, for NodePort, it would make sense if setting GatewaySpec.addresses caused the proxy pods to be scheduled on nodes that match those external IPs. Or rejecting that configuration.

Repro steps:

1. Create an EnvoyProxy with NodePort.
2. Create a Gateway without spec.addresses.
3. Look at the generated Service.

---

1. gateway/internal/infrastructure/kubernetes/proxy/resource_provider.go

   Line 100 in b1a7520

   serviceSpec.ExternalIPs = r.infra.Addresses

2. gateway/internal/gatewayapi/address.go

   Line 30 in b1a7520

   gwInfraIR.Proxy.Addresses = ipAddr

3. gateway/internal/status/gateway.go

   Line 54 in b1a7520

   addresses = nodeAddresses

[arkodg]

What is happening

• All the Node Addresses (Internal and External) are being appended to Gateway.Status.Addresses
• The Gateway.Spec.Addresses is also being appended to Gateway.Status.Addresses

What should happen

• We cannot tie location of where the Envoy pod is running to the IP that should be used to reach it. Kubernetes allows the Pod to be reachable from any node using iptables magic
• We currently append Internal IP of the node to Gateway.Status.Addresses if External IP is not present. It is debatable whether this is a good idea or not, and whether we should only limit it to External IP of the node ?
• We currently always append Gateway.Spec.Addresses to Gateway.Status.Addresses. Maybe this should be an explicit set/override ?

[tommie]

We cannot tie location of where the Envoy pod is running to the IP that should be used to reach it. Kubernetes allows the Pod to be reachable from any node using iptables magic

Doing internal routing doesn't feel very good when I actually know which nodes have Proxy running. Why can't we know the location? (This is mostly about using ExternalDNS to point to the relevant nodes directly.)

[tommie]

As I learn more about k8s port management, here are some notes:

• When proxier adds IPTables rules, it adds rules for a few different sources:
  • The clusterIP is bound on Service.port
  • The externalIPs are bound on Service.port
  • The status.loadBalancer.ingress.ip are bound on Service.port
  • The Service.nodePort is bound for any IP on any node.
• Since Service.nodePort is auto-generated (in current EG,) it cannot be used for ingress traffic.
  • That leaves us with externalIPs and status.loadBalancer. For NodePort, that leaves externalIPs.
  • I.e. the Service rendered by EG must have externalIPs.
  • The recent change here simply populates externalIPs from GatewaySpec.addresses, but otherwise leaves it empty.
• Node-to-node forwarding cannot reasonably work well for Service.port, since it's not unique for the cluster, unlike nodePort. Unless I'm missing some routing magic in K8s.
  • For simple bare-metal clusters, each node has an IP address, and that's all. We have the ability to send requests directly to a proxy Node/Pod, without node-to-node forwarding. Using DNS to store IP address of the nodes currently running the proxy Pods.
• What I think I'm after is EG defaulting Service.externalIPs to the IP addresses of the nodes where actual proxy Pods are running.
  • As before (and questioned earlier) this could use node external IP if available, and fall back to internal IP. This choice does not disappear; the logic just moves.
  • If GatewaySpec.addresses are provided, it probably means the cluster admin has some specific rules controlled elsewhere, e.g. Pod node affinity, and doing the current simple copying is enough.
  • We could know whether individual GatewaySpec.addresses are node addresses, and force the proxy Pods to match, but it would add yet another node affinity knob, which is probably not warranted.
• It is possible to build a Gateway-Pod controller that updates GatewaySpec.addresses based on where proxy Pods owned by the Gateway are actually running. It feels a bit backwards to have a controller for an individual spec field, so I'd prefer if EG did this, and we could see GatewaySpec as static.
  • However, this right now creates duplicate addresses in GatewayStatus.addresses, since all nodes are already added.
  • Which also means it's meaningless.

[arkodg]

closing this issue since #1465 allows a way for the user to explicitly control the value in gateway.Status.Addressesusinggateway.Spec.Addresses`.
@tommie please feel free to reopen this issue if the issue remains unresolved.