There are several situations in which kube-router will use tunnels in order to perform certain forms of overlay / underlay routing within the cluster. To accomplish this, kube-router makes use of IPIP overlay tunnels that are built into the Linux kernel and instrumented with iproute2.
By default, kube-router enables the option --enable-overlay which will perform overlay networking based upon the
--overlay-type setting (by default set to subnet). So out of the box, kube-router will create a tunnel for
pod-to-pod traffic any time it comes across a kube-router enabled node that is not within the subnet of it's primary
interface.
Additionally, if --overlay-type is set to full kube-router will create an tunnel for all pod-to-pod traffic and
attempt to transit any pod traffic in the cluster via an IPIP overlay network between nodes.
Finally, kube-router also uses tunnels for DSR (Direct Server Return). In this case, the inbound traffic is encapsulated in an IPIP packet by IPVS after it reaches the node and before it is set to the pod for processing. This allows the return IP address of the sender to be preserved at the pod level so that it can be sent directly back to the requestor (rather than being routed in a synchronous fashion).
- IPIP (IP in IP) - This is the default method of encapsulation that kube-router uses
- FoU (Foo over UDP) - This is an optional type of IPIP encapsulation that kube-router uses if the user enables it
Specifically, kube-router uses GUE
(Generic UDP Encapsulation)
in order to support both IPv4 and IPv6 FoU tunnels. This option can be enabled via the kube-router parameter
--overlay-encap=fou. Optionally, the user can also specify a desired port for this traffic via the
--overlay-encap-port parameter (by default set to 5555).
Unfortunately, Azure doesn't allow IPIP encapsulation on their network. So users that want to use an overlay network
will need to enable fou support in order to deploy kube-router in an Azure environment.
While it is possible to change a running cluster between ipip and fou type tunnels, administrators should beware
that during the rollout it will cause pod-to-pod traffic to be dropped between nodes. Since, in almost all rollout
scenarios, kube-router would be rolled out gracefully one pod or host to the next, during this rollout there will be
mismatches between the encapsulation support between the two nodes as invariably one node will have an upgraded
kube-router and another node may have a previous deployment.
When this happens, they will have conflicting encapsulation setup on their tunnels and traffic will not be able to be sent between the two nodes until they are using a consistent encapsulation protocal between them.
Once all nodes have upgraded with the destination configuration, pod-to-pod traffic patterns should return to normal.