EndpointSlice Support

[andrewsykim]

EndpointSlice is now enabled by default in Kubernetes and offers some solid performance benefits for clusters with large Endpoint sets. It might be worthwhile to support consuming EndpointSlice, at least as optional.

[youngnick]

Thanks for raising this @andrewsykim, totally agree that we should support consuming EndpointSlice in some fashion.

[zianke]

Hi Team. Agree that Contour should support EndpointSlice in the future. I just discussed with @stevesloka and would like to start working on an experimental implementation. Will send you an update once it's done.

[stevesloka]

@zianke has a branch which implements endpoint slices (master...zianke:endpointslice) which is fantastic! 🎉

Before pushing up a PR to implement to better help shape the PR, when would we want to add support for EndpointSlices? It's still beta in Kubernetes v1.17, so we. can't fully move to it. Should probably add a feature flag to enable which would disable the current Endpoints implementation.

After we get this merged we'd also need to build out some E2E tests which mock up the behavior.

[youngnick]

Great work on getting something done so quickly @zianke!

However, given that we're working on the Endpoint translator at the moment, this may be something that will need refactoring if we change anything about EndpointTranslator or ClusterLoadAssignment.

I'd also like to do our best not to add feature flags for enabling Kinds - in particular, EndpointSlice is designed so that if it's available, you can use either it or Endpoints, relatively transparently. I'd prefer to see detection of enabled Kinds used rather than a feature flag.

[dprotaso]

Hey following up what's the status of EndpointSlices? and does Contour currently support External Service Routing with EndpointSlices with the CNAME address type?

[youngnick]

Thinking more about this, I think that we should confirm what K8s version EndpointSlice is GA in, and if that's less than our minimum supported (which it almost certainly is), we should move to using EndpointSlices exclusively (since Kube will ensure that changes are propagated as necessary back to Endpoints).

There are some things that wee may need to be careful about, since I believe that EndpointSlices can be used for other stuff (like External Service Routing and locality routing), so we will also need to validate if we need to do anything special with the resource, or if we can just migrate the simple logic we have now to the new resource.

But I agree we should look at this one soon. @xaleeks, what do you think?

[dprotaso]

I think that we should confirm what K8s version EndpointSlice is GA

Beta in 1.17, GA in 1.21

GA doesn't necessarily mean all things are supported since it depends on the DNS provider for the cluster. I think it just means the API is stable.

For example - CoreDNS EndpointSlice support was added 1.8.4 which is bundled with K8s 1.22 (via kubeadm).

Note you may need to watch a subset of Endpoints objects that aren't mirrored as EndpointSlices - details here: https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/#endpointslice-mirroring

[xaleeks]

@youngnick Yup, we should support this. However, EndpointSlice started in k8s 1.17 it seems. Do we need to add a check that the version of K8s running supports it before Contour is deployed? I guess all the supported versions of Contour today would necessitate a version of k8s that supports it so we should be in the clear.

Tagging it for 1.20 for now to start investigating but we can also slot this in the 1.21

[xaleeks]

Also according to https://kubernetes.io/docs/concepts/services-networking/endpoint-slices/, it allows for setting a "topology.kubernetes.io/zone" key. Which opens us to this https://kubernetes.io/docs/concepts/services-networking/service-topology/

So a client can hit his preferred node specified in the topologyKeys for his service when using Ingress . I’m not sure if that means this ServiceTopology feature is unsupported unless we support EndpointSlice. Interesting opportunity for anyone looking to utilize this.

[youngnick]

If we are already mandating a newer version of Kubernetes than 1.17 (which I think we are), then we just need to call out when we start doing EndpointSlices that 1.17 (or whatever version) is now required.

The zone topology is an interesting idea, yeah.

[howardjohn]

topologyKeys is deprecated fwiw. replaced by https://kubernetes.io/docs/concepts/services-networking/topology-aware-hints/

[arjunsalyan]

I can take this up. I was also able to write an EndpointSliceTranslator, however I am a bit confused about the part where we observe the k8s resources (endpoints etc.). The name of EndpointSlice has an additional suffix and I am not really sure if that will come into picture while reading the slice. If someone could help me with any dev document or simply where to look in the code for reading the k8s resources - that would be helpful.

[dprotaso]

Related Gateway Issue hopes to exercise different Service/Endpoint[Slice] variants - kubernetes-sigs/gateway-api#1718

[dprotaso]

Also related EndpointSlices/FQDN could be a replacement for Service type=ExternalName

#3950

[orenwolf]

Confirming that this appears to create a hard cap of 1000 replicas that contour can track for a given service, as non-sliced endpoints are limited to 1000 members.

[Quentin-M]

This would also allow usage of Topology Aware Routing, which was added back in Kubernetes as EndpointSlice hints since 1.23.

[rajatvig]

Will plan to work on adding support for this.