Calico Route Reflector Operator

Use your own risk !!!

Proposal documentation found here: https://github.com/mhmxs/calico-route-reflector-operator-proposal. Please feel free to share your ideas !!!

Prerequisites

• Kubernetes cluster up and running
• Calico network on Kubernetes data store a.k.a. KDD.

This Kubernetes operator can monitor and scale Calico route refloctor pods based on node number per zone. The operator owns a few environment variables:

• DATASTORE_TYPE Calico datastore [incluster, kubernetes, etcd], default incluster
• K8S_API_ENDPOINT Kubernetes API endpoint, default https://kubernetes.default
• ROUTE_REFLECTOR_CLUSTER_ID Route reflector cluster ID, default 224.0.0.%d
• ROUTE_REFLECTOR_MIN Minimum number of route reflector pods per zone, default 3
• ROUTE_REFLECTOR_MAX Maximum number of route reflector pods per zone, default 25
• ROUTE_REFLECTOR_RATIO Node / route reflector pod ratio, default 0.005 (1000 * 0.005 = 5)
• ROUTE_REFLECTOR_NODE_LABEL Node label of the route reflector nodes, default calico-route-reflector=
• ROUTE_REFLECTOR_ZONE_LABEL Node label of the zone, default ``
• ROUTE_REFLECTOR_TOPOLOGY Selected topology of route reflectors [simple, multi], default simple

You can edit or add those environment variables at the manager manifest. You can add Calico client config related variables, Calico lib will parse it in the background.

During the api/core/v1/Node reconcile phases it calculates the right number of route refloctor nodes per zone. It supports linear scaling only and it multiplies the number of nodes with the given ratio and than updates the route reflector replicas to the expected number. After all the nodes are labeled correctly it regenerates BGP peer topology for the cluster.

Usage

This is a standard Kubebuilder opertor so building and deploying process is similar as a stock Kubebuilder project.

After first reconcile phase is done don not forget to disable the node-to-node mesh!

Use latest release:

kustomize build config/crd | kubectl apply -f -
$(cd config/manager && kustomize edit set image controller=quay.io/mhmxs/calico-route-reflector-controller:v0.0.3)
kustomize build config/default | kubectl apply -f -

Use official latest master image:

kustomize build config/crd | kubectl apply -f -
kustomize build config/default | kubectl apply -f -

Build your own image: IMG_REPO=[IMG_REPO] IMG_NAME=[IMG_NAME] IMG_VERSION=[IMG_VERSION] make test docker-push install deploy

Limitations

• In the current implementation each reconcile loop fetches all nodes and all BGP peer configurations which could take too much time in large clusters.
• Multi cluster topology doesn't select route reflectors from multiple zones, so on case of full zone outages you have a chance some network outage if a node all 3 connections was established toward of that zone.
• Multi cluster topology rebalances the whole cluster on case of nodes are added. If you are unlicky it could drop all 3 route reflector sessions which chause 1-2 sec network outage.
• Multi cluster topology generates 3 BGP peers per node, which can grow in large cluster. Would be better to create BGP peer configuration for eahc route reflector combination to decrease number of BGP peer configs. For example: [1,2,3], [1,2,4], [2,3,4], [1,3,4]

Roadmap

• Etcd data store support (You have to add Etcd related env vars into manager manually)
• Use custom resource instead of environment variables
• Dedicated or preferred node label
• Disallow node label
• Handle taints and tolerations

Contributing

We appreciate your help!

To contribute, please read our contribution guidelines: CONTRIBUTION.md