RFC - Attaching cluster nodes to complex upstream networks

[nelljerram]

Hi - this is a collection of hackery that I've developed in order to model a Kubernetes cluster that is split across racks, and with redundant planes of connectivity between all the nodes.

• Splitting across racks means we need to be able to specify a Docker network to attach each node to, instead of assuming the default 'bridge'.

• Redundant planes of connectivity means:

  • We want to be able to attach each node to multiple specified Docker networks.
  • We want to configure a loopback address for each node (i.e. an address that is not tied to a particular network) and arrange for those addresses to be used as source and destination IPs by Kubernetes, instead of interface-specific addresses.
  • As part of the previous point, we need to add routing for the loopback addresses, after the node containers have been created and before kubeadm starts running.

Here's an example config with the new fields I've added support for:

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  disableDefaultCNI: true
nodes:
- role: control-plane
  networks: [ra1, ra2]
  loopback: 172.31.10.3
  routes:
  - "172.31.10.0/24 src 172.31.10.3 nexthop dev eth0 nexthop dev eth1"
  - "172.31.20.0/24 src 172.31.10.3 nexthop via 172.31.11.1 nexthop via 172.31.12.1"
- role: worker
  networks: [ra1, ra2]
  loopback: 172.31.10.4
  routes:
  - "172.31.10.0/24 src 172.31.10.4 nexthop dev eth0 nexthop dev eth1"
  - "172.31.20.0/24 src 172.31.10.4 nexthop via 172.31.11.1 nexthop via 172.31.12.1"
- role: worker
  networks: [rb1, rb2]
  loopback: 172.31.20.3
  routes:
  - "172.31.20.0/24 src 172.31.20.3 nexthop dev eth0 nexthop dev eth1"
  - "172.31.10.0/24 src 172.31.20.3 nexthop via 172.31.21.1 nexthop via 172.31.22.1"
- role: worker
  networks: [rb1, rb2]
  loopback: 172.31.20.4
  routes:
  - "172.31.20.0/24 src 172.31.20.4 nexthop dev eth0 nexthop dev eth1"
  - "172.31.10.0/24 src 172.31.20.4 nexthop via 172.31.21.1 nexthop via 172.31.22.1"

I'm sure I haven't done this all right first time, but I guess that enhancements like these will be of wider interest than just for me. Hence I'm posting this as an RFC to ask if it would make sense to upstream this kind of support, and if so to get your advice on how to put into a potentially mergeable form.

[k8s-ci-robot]

Welcome @neiljerram!

It looks like this is your first PR to kubernetes-sigs/kind 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/kind has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @neiljerram. Thanks for your PR.

I'm waiting for a kubernetes-sigs or kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: neiljerram
To complete the pull request process, please assign munnerz
You can assign the PR to them by writing /assign @munnerz in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[BenTheElder]

RFC - Attaching cluster nodes to complex upstream networks #939

usually an RFC comes before sending an implementation 🙃

/hold

• loopback is generally not a configurable IP?
• writing routes inline seems brittle and very specific to your use case
• configurable docker network does not work safely yet because the default network has different behavior, please see the issues in the issue tracker regarding this

[aojea]

hi @neiljerram , I think that all this configuration can be done with a script once kind finish the deployment, configuring the routes and loopback interfaces on nodes.
Also, it's adding too many dependencies on docker network that will be bad if we want to support more providers,
I have several examples of creating complex topologies with containers in this repo https://github.com/aojea/frr-lab , using tools like koko, you can use something similar scripting your complex topologies and using kind to deploy the different clusters

[k8s-ci-robot]

@neiljerram: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

I think #939 (comment) is the advice to go with here.

Thanks for the PR, but let's discuss this in an issue first if you want to follow up. I think the end goal can be done without these changes and the complexity will not help the existing use cases.

[nelljerram]

Thanks @BenTheElder . I'm not sure you are right about the end goal being achievable without any KIND changes, but I'm happy for this PR to be closed for now, and will come back to this when I have time, hopefully soon.