Support Kubernetes multi-network

[plwhite]

Issue type

• Feature request

Support for Kubernetes multi-network

We would like Linkerd2 to support multi-network in Kubernetes. For context:

Spec: https://docs.google.com/document/d/1Ny03h6IDVy_e_vmElOqR7UdTPAG_RNydhVE1Kx54kFQ/edit
Reference implementation: https://github.com/intel/multus-cni

Roughly this works by having a management network which is the existing pod network, and additional networks that are created independent of Kubernetes and are only known to the K8s API as annotations to pods and services. Endpoints objects for services are created by an extra controller (still at prototype level - see https://github.com/K8sNetworkPlumbingWG/k8s-net-attach-def-controller).

From the Linkerd point of view, I think the changes required are:

1. Some careful configuration of the network and Linkerd so that the proxy only picks up traffic for the correct network (not a code change, just config documentation).

2. For outbound connections, Linkerd should explicitly bind to the same source IP address as the pod bound to (since there may be multiple IPs in play).

3. For incoming connections, Linkerd should explicitly connect to the correct IP, based on the IP on the incoming request (before iptables got its hands on it). It may already do what I would hope.

It's quite possible that my limited Rust knowledge has led me to miss that it already does what we want in at least some of these cases, which would obviously be great news.

Next steps

I'd like to understand

• if there is an objection to this in principle;

• if anybody else has done anything in this area.

In due course, I hope to prototype this and see how hard it seems to be, with perhaps a PR down the line.

[grampelberg]

@plwhite how is CNI setting this up? What interfaces are provided to the pods?

[plwhite]

The model is roughly this.

• You create a CRD for each additional network, defining whatever information that network needs, including which CNI plugin it uses.
• When you create a pod, you can add annotations for any additional networks that it should be in.
• The master CNI plugin (multus, normally) gets called by Kubernetes.
• Multus calls a configured default plugin to create the pod network. Then for each additional network in the pod annotations it reads the CRD and calls the CNI for that additional network to create the additional network interfaces and do anything else required to the pod namespace for that network (setting up routes, mostly).
• Multus only returns information to Kubernetes from the default plugin, i.e. the normal pod network. Additional network interfaces and IPAM get written to the pod as annotations.

The net of all this is that pods are always on the management network, but may have arbitrary interfaces on other networks. Service meshes generally don't play nicely with this model because they at some level assume that traffic uses only the management (pod) network. I'd like to figure out in more detail what changes would be required in linkerd 2, and then (ideally) get an enhancement PR in for it.