diff --git a/docs/README.md b/docs/README.md index 72837bfc764..77cccc5fc7c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -18,6 +18,7 @@ * [Gardener Seed Admission Controller](concepts/seed-admission-controller.md) * [Gardenlet](concepts/gardenlet.md) * [Backup Restore](concepts/backup-restore.md) +* [ETCD](concepts/etcd.md) * [Network Policies](concepts/network_policies.md) * [Relation between Gardener API and Cluster API](concepts/cluster-api.md) @@ -29,6 +30,7 @@ * [Cleanup of Shoot clusters in deletion](usage/shoot_cleanup.md) * [Custom `CoreDNS` configuration](usage/custom-dns.md) * [(Custom) CSI components](usage/csi_components.md) +* [Dockershim removal](usage/docker-shim-removal.md) * [ExposureClasses](usage/exposureclasses.md) * [Gardener configuration and usage](usage/configuration.md) * [`ManagedIstio` feature](usage/istio.md) diff --git a/docs/concepts/etcd.md b/docs/concepts/etcd.md new file mode 100644 index 00000000000..53d9bd16a6e --- /dev/null +++ b/docs/concepts/etcd.md @@ -0,0 +1,63 @@ +# etcd - Key-Value Store for Kubernetes + +[etcd](https://etcd.io/) is a strongly consistent key-value store and the most prevalent choice for the Kubernetes +persistence layer. All API cluster objects like `Pod`s, `Deployment`s, `Secret`s, etc. are stored in `etcd` which +makes it an essential part of a [Kubernetes control plane](https://kubernetes.io/docs/concepts/overview/components/#control-plane-components). + +## Shoot cluster persistence + +Each shoot cluster gets its very own persistence for the control plane. It runs in the shoot namespace on the respective +seed cluster. Concretely, there are two etcd instances per shoot cluster which the `Kube-Apiserver` is configured +to use in the following way: + +* etcd-main + +A store that contains all "cluster critical" or "long-term" objects. These object kinds are typically considered +for a backup to prevent any data loss. + +* etcd-events + +A store that contains all `Event` objects (`events.k8s.io`) of a cluster. `Events` have usually a short retention +period, occur frequently but are not essential for a disaster recovery. + +The setup above prevents both, the critical `etcd-main` is not flooded by Kubernetes `Events` as well as backup space is +not occupied by non-critical data. This segmentation saves time and resources. + +## etcd Operator + +Configuring, maintaining and health-checking `etcd` is outsourced to a dedicated operator called [ETCD Druid](https://github.com/gardener/etcd-druid/). +When [Gardenlet](../concepts/gardenlet.md) reconciles a `Shoot` resource, it creates or updates an [Etcd](https://github.com/gardener/etcd-druid/blob/1d427e9167adac1476d1847c0e265c2c09d6bc62/config/samples/druid_v1alpha1_etcd.yaml) +resources in the seed cluster, containing necessary information (backup information, defragmentation schedule, resources, etc.) `etcd-druid` +needs to manage the lifecycle of the desired etcd instance (today `main` or `events`). Likewise, when the shoot is deleted, +Gardenlet deletes the `Etcd` resource and [ETCD Druid](https://github.com/gardener/etcd-druid/) takes care about cleaning up +all related objects, e.g. the backing `StatefulSet`. + +## Autoscaling + +Gardenlet maintains [HVPA](https://github.com/gardener/hvpa-controller/blob/master/config/samples/autoscaling_v1alpha1_hvpa.yaml) +objects for etcd `StatefulSet`s if the corresponding [feature gate](../deployment/feature_gates.md) is enabled. This enables +a vertical scaling for `etcd`. Downscaling is handled more pessimistic to prevent many subsequent `etcd` restarts. Thus, +for `production` clusters downscaling is deactivated and for all other clusters lower advertised requests/limits are only +applied during a shoot's maintenance time window. + +## Backup + +If `Seed`s specify backups for etcd ([example](https://github.com/gardener/gardener/blob/e9bf88a7a091a8cf8c495bef298bdada17a03c7f/example/50-seed.yaml#L19)), +then Gardener and the respective [provider extensions](../extensions/overview.md) are responsible for creating a bucket +on the cloud provider's side (modelled through [BackupBucket resource](../extensions/backupbucket.md)). The bucket stores +backups of shoots scheduled on that seed. Furthermore, Gardener creates a [BackupEntry](../extensions/backupentry.md) +which subdivides the bucket and thus makes it possible to store backups of multiple shoot clusters. + +The `etcd-main` instance itself is configured to run with a special backup-restore _sidecar_. It takes care about regularly +backing up etcd data and restoring it in case of data loss. More information can be found on the component's GitHub +page https://github.com/gardener/etcd-backup-restore. + +How long backups are stored in the bucket after a shoot has been deleted, depends on the configured _retention period_ in the +`Seed` resource. Please see this [example configuration](https://github.com/gardener/gardener/blob/849cd857d0d20e5dde26b9740ca2814603a56dfd/example/20-componentconfig-gardenlet.yaml#L20) for more information. + +## Housekeeping + +[etcd maintenance tasks](https://etcd.io/docs/v3.3/op-guide/maintenance/) must be performed from time to time in order +to re-gain database storage and to ensure the system's reliability. The [backup-restore](https://github.com/gardener/etcd-backup-restore) +_sidecar_ takes care about this job as well. Gardener chooses a random time **within the shoot's maintenance time** to +schedule these tasks. \ No newline at end of file diff --git a/docs/extensions/controlplane-webhooks.md b/docs/extensions/controlplane-webhooks.md index df013a3879b..cfcb81306ea 100644 --- a/docs/extensions/controlplane-webhooks.md +++ b/docs/extensions/controlplane-webhooks.md @@ -12,7 +12,6 @@ In order to apply any provider-specific changes to the configuration provided by In order to support a new cloud provider you should install "controlplane" mutating webhooks for any of the following resources: * Deployment with name `kube-apiserver`, `kube-controller-manager`, or `kube-scheduler` -* StatefulSet with name `etcd-main` or `etcd-events` * Service with name `kube-apiserver` * `OperatingSystemConfig` with any name and purpose `reconcile` @@ -123,13 +122,9 @@ The kube-scheduler command line can't contain provider-specific flags, and it ma ### etcd-main and etcd-events -To deploy etcd, Gardener **shall** create 2 StatefulSets named `etcd-main` and `etcd-events` in the Shoot namespace. They can be mutated by webhooks to apply any provider-specific changes to the standard configuration provided by Gardener. +To deploy etcd, Gardener **shall** create 2 [Etcd](https://github.com/gardener/etcd-druid/blob/1d427e9167adac1476d1847c0e265c2c09d6bc62/config/samples/druid_v1alpha1_etcd.yaml) named `etcd-main` and `etcd-events` in the Shoot namespace. They can be mutated by webhooks to apply any provider-specific changes to the standard configuration provided by Gardener. -The pod template of these 2 deployments **shall** contain a container named `etcd`. It **shall not** contain a sidecar container for etcd backups. If such a container is needed, it should be added by webhooks, together with any volumes it may need to mount. - -The `command` field of the `etcd` container **shall** contain the etcd command line. It **shall** contain only provider-independent flags that should be ignored by webhooks. It can't contain provider-specific flags, and it makes no sense to specify provider-specific environment variables or mount provider-specific `Secret` or `ConfigMap` resources as volumes. - -The `volumeClaimTemplates` section of these 2 StatefulSets **shall** contain a template named `etcd-main` or `etcd-events`. This template **shall** use the default storage class. The corresponding claim is mounted into the `etcd` container at `/var/etcd/data`. If it is desirable to use a non-default storage class, this should be done by webhooks. +Gardener **shall** configure the `Etcd` resource completely to set up an etcd cluster which uses the default storage class of the seed cluster. ### cloud-controller-manager