Gardener comparison

[luisdavim]

This project seems similar to https://github.com/gardener/gardener what are the main differences? Why should we choose one over the other?

[prometherion]

I'm a bit biased about it since the main maintainer and contributor to Kamaji, hope someone else can join the discussion.

Kamaji was born due to my personal experience in managing a fleet of Kubernetes clusters, mostly on-prem, thus with a higher cognitive load and responsibilities compared to the cloud (sic). Creating clusters was a cumbersome task for many reasons, although some automation can easily solve it: the real challenge was operating it in the long term, such as:

• How can I scale the Control Plane?
• How can I easily rotate certificates and roll out the Control Plane without downtime?
• How can I increase the density of the Control Plane virtual machines?
• How can I avoid getting paged if one or more VMs are failing?

Back in the day, I was mostly involved in writing operators and the idea of an Operator that manages Kubernetes control planes as regular applications popped into my mind. Nothing new if you look for kube-ception or kubernetes-in-kubernetes on Google search.

Great software is expected to make easy something difficult, that has been our deliverable (pretty sure @bsctl agrees on that), and the complexity I found with Gardener has been terrific the first time I saw it.

In Kamaji, essentially, you have just two resources:

• TenantControlPlane, a namespace resource that declares the desired state for a given Control Plane
• DataStore, a cluster-scoped resource that declares the connection details for a data store, mostly known as etcd.

Before delving into the concepts, describing Kamaji, in a nutshell, would be Kubernetes API Server as a Service.
Kamaji offers Control Planes in a similar way to NFV, it means the worker nodes join the desired Control Plane that is hosted in a different infrastructure, even if these nodes don't have a public and reachable IP thanks to the Konnectivity integration.

Having one etcd per Control Plane is good and solves many issues, especially from the throttling, quota, and bandwidth allocation, however, we wanted to optimize one use-case we saw several times. Imagine a cluster made of a few compute nodes (e.g.: 2): if you want HA on the CP, at least 3 nodes must be available, and you're going to spend more on the Control Plane rather than the compute nodes.

This is what we call the "Kubernetes Tax" (credits, @bsctl), and the reason resides on the raft algorithm used by etcd to ensure the HA, and it's ok.

Now you can image an organization where multiple small teams need to manage their own small clusters: besides the cluster sprawl, the amount of etcd clusters to maintain is cumbersome, it's much better having a pool of these where a multi-tenancy is available, and each CP is writing their data according to a prefix.

In the end, with Kamaji you can pick if having a 1:1 ownership between the CP and the DataStore, or by having a pool according to your design.

However, we noticed that using etcd in the long term could be dangerous, especially when the K/V size is exceeding 8 GB: it means we need to provide a different backing data store, pluggable. And here comes the integration with kine, and shifting the multi-tenancy from etcd (the key prefix) to databases (using the schema).

I know this could be a wall of text, tl;dr; our aim was to provide software addressing Day-2 problems (such as the migration of the datastore, or the linear minor update of the Kubernetes version) in the simplest way, by abstracting the underlying infrastructure and focusing only on the most critical part of maintaining a cluster, the Control Plane.

Also, the design of the multi-tenant DataStore could be saw as opinionated, however, it's optional and nobody is blocking you from having a 1:1 DS and TCP relation.

Last but not least, Kamaji aims to be perfectly integrated with the Cluster API ecosystem, and we started supporting it by working with the community and the other infrastructure working group by providing the CAPI Control Plane Provider.

[luisdavim]

Thanks, this helps.

[bsctl]

@luisdavim Thanks for asking that and for looking at Kamaji. Totally agree with what @prometherion said.

Kamaji has been designed according to our experience and conversations from professional consultancy with several MSPs/CSPs looking for a simple and effective way to manage Kubernetes clusters at scale. And with cost saving requirements, as "infrastructure can scale but people managing it do not" (credits to @prometherion). We have been supported those customers in scouting several solutions and Gardener was an option we evaluated. However, what discouraged our customers to use Gardner was the inherent complexity that would not suit their operational burden saving requirements. So we built Kamaji having customers requirements in mind.

That said, we reconize Gardner as pioneer in the hyper-scale kubernetes space. However, we want to keep Kamaji simpler as much as possible and perfectly integrated with the Cluster API ecosystem.

Hope this helps.

[luisdavim]

Thanks for the reply, I was looking for a more technical comparison