From 489aaaa2ccb4ac05f35d57bcec11a18d1a0ecc12 Mon Sep 17 00:00:00 2001 From: Rafael Franzke Date: Wed, 25 Sep 2019 15:51:10 +0200 Subject: [PATCH] Documentation for Gardener config/usage --- docs/README.md | 22 ++- docs/concepts/configuration.md | 60 -------- docs/testing/integration_tests.md | 4 +- docs/usage/configuration.md | 131 ++++++++++++++++++ .../openidconnect-presets.md | 0 ...ng-provider.yaml => 80-secretbinding.yaml} | 0 pkg/operation/seed/seed.go | 1 - 7 files changed, 151 insertions(+), 67 deletions(-) delete mode 100644 docs/concepts/configuration.md create mode 100644 docs/usage/configuration.md rename docs/{concepts => usage}/openidconnect-presets.md (100%) rename example/{80-secretbinding-provider.yaml => 80-secretbinding.yaml} (100%) diff --git a/docs/README.md b/docs/README.md index 38bd38a8098..1a7cef26539 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,5 +1,9 @@ # Documentation Index +* [General architecture](https://github.com/gardener/documentation/wiki/Architecture) +* [Gardener landing page `gardener.cloud`](https://gardener.cloud/) +* ["Gardener, the Kubernetes Botanist" blog on kubernetes.io](https://kubernetes.io/blog/2018/05/17/gardener/) + ## Development * [Setting up a local development environment](development/local_setup.md) @@ -9,9 +13,13 @@ * [Adding New Cloud Providers](development/new-cloud-provider.md) * [Extending the Monitoring Stack](development/monitoring-stack.md) +## Testing + +* [Integration Testing Manual](testing/integration_tests.md) + ## Concepts -* [Configuration and Secrets](concepts/configuration.md) +* [Gardener Scheduler](concepts/scheduler.md) ## Extensions @@ -44,10 +52,16 @@ ## Usage -* [Audit a Kubernetes Cluster](usage/shoot_auditpolicy.md) +* [Gardener configuration and usage](usage/configuration.md) +* [OpenIDConnect presets](usage/openidconnect-presets.md) * [Supported Kubernetes versions](usage/supported_k8s_versions.md) -* [Trouble Shooting Guide](usage/trouble_shooting_guide.md) +* [Audit a Kubernetes cluster](usage/shoot_auditpolicy.md) +* [Trigger shoot operations](usage/shoot_operations.md) +* [Troubleshooting guide](usage/trouble_shooting_guide.md) ## Proposals -* [Gardener extensibility and extraction of cloud-specific/OS-specific knowledge](proposals/01-extensibility.md) +* [GEP-1: Gardener extensibility and extraction of cloud-specific/OS-specific knowledge](proposals/01-extensibility.md) +* [GEP-2: `BackupInfrastructure` CRD and Controller Redesign](proposals/02-backupinfra.md) +* [GEP-3: Network extensibility](proposals/03-networking-extensibility.md) +* [GEP-4: New `core.gardener.cloud/v1alpha1` APIs required to extract cloud-specific/OS-specific knowledge out of Gardener core](proposals/04-new-core-gardener-cloud-apis.md) diff --git a/docs/concepts/configuration.md b/docs/concepts/configuration.md deleted file mode 100644 index 44d252b1b89..00000000000 --- a/docs/concepts/configuration.md +++ /dev/null @@ -1,60 +0,0 @@ -# Configuration - -In order to establish different configuration settings for the same cloud environment, one has to define [`CloudProfiles`](../../example/30-cloudprofile-aws.yaml). These profiles define configuration and constraints for allowed values in the Shoot manifest as well. - -Seed clusters have their [own resource](../../example/50-seed-aws.yaml) as well. These resources contain metadata about the respective Seed cluster and a reference to a secret holding the credentials (see below). Also, it contains the backup provider configuration and a reference to a secret holding the credentials to the object store (see below). - -The Gardener requires some secrets in order to work properly. These secrets are: -* *Seed cluster secrets*, contain the credentials of the cloud provider account in which the Seed cluster is deployed, and a Kubeconfig which can be used to authenticate against the Seed cluster's kube-apiserver, please see [this](../../example/40-secret-seed-aws.yaml) for an example. - -* *Object store provider secrets*, contains the credentials of the cloud provider account with object store privileges in which backup of the Shoot clusters (currently only etcd backing Shoot cluster) can be stored, please see [this](../../example/45-secret-seed-backup-aws.yaml) for an example. - * For each `Seed`, Gardener needs to create a backup infrastructure (AWS S3 bucket equivalent) for the configured backup provider. As mentioned above, the `Seed` has a reference to this secret. - -* *Internal domain secrets* (optional), contain the DNS provider credentials (with appropriate privileges) which will be used to create/delete internal DNS records for the Shoot clusters (e.g., `example.com`), please see [this](../../example/10-secret-internal-domain.yaml) for an example. - * These secrets are used in order to establish a stable endpoint for Shoot clusters which is used internally by all control plane components. - * It is forbidden to change the internal domain secret if there are existing Shoot clusters. - -* *Default domain secrets* (optional), contain the DNS provider credentials (with appropriate privileges) which will be used to create/delete DNS records for the default domain (e.g., `example.com`), please see [this](../../example/10-secret-default-domain.yaml) for an example. - * These secrets are used in order to allow not specifying a hosted zone when creating a Shoot cluster in the `.spec.dns.hostedZoneID` field (useful when a user does not have an own domain/hosted zone but want us to manage it). In this case, based on the provided `.spec.dns.domain` value, the Gardener tries to find an appropriate secret holding the credentials for the hosted zone of this domain. It will use them to manage the relevant DNS records. Currently, we have implemented AWS Route53 and Google CloudDNS as DNS providers. For Google CloudDNS you need to provide a service account with the `DNS Administrator` role. For AWS you need to provide a user being assigned to this IAM policy document: - ```bash - { - "Version": "2012-10-17", - "Statement": [ - { - "Effect": "Allow", - "Action": [ - "route53:GetChange", - "route53:GetHostedZone", - "route53:ListResourceRecordSets", - "route53:ChangeResourceRecordSets" - ], - "Resource": "*" - } - ] - } - ``` - -* *Alerting SMTP secrets* (optional), contain the SMTP credentials which will be used by the [Alertmanager](https://prometheus.io/docs/alerting/alertmanager/) to send emails on alerts, please see [this](../../example/10-secret-alerting-smtp.yaml) for an example. - * These secrets are used by the Alertmanager which is deployed next to the Kubernetes control plane of a Shoot cluster in Seed clusters. In case there have been alerting SMTP secrets configured, the Gardener will inject the credentials in the configuration of the Alertmanager. It will use them to send mails to the stated email address in case anything is wrong with the Shoot clusters. - -* *Cloud provider secrets*, contains the credentials of the cloud provider account in which Shoot clusters can be deployed, please see [this](../../example/70-secret-cloudprovider-aws.yaml) for an example. - * For each Shoot cluster, the Gardener needs to create infrastructure (networks, security groups, technical users, ...) and worker nodes in the desired cloud provider account. - -The described secrets are expected to be stored in the so-called **Garden namespace**. In case the Gardener runs inside a Kubernetes cluster, the Garden namespace is the namespace the Gardener is deployed in. In case it runs outside (local development), the Garden namespace must be specified via a command line flag (see below). -The secrets are determined based on labels with key `garden.sapcloud.io/role`. Please take a look on the above linked examples. - -The Seed cluster which is used to deploy the control plane of a Shoot cluster can be specified by the user in the Shoot manifest (see [here](../../example/90-deprecated-shoot-azure.yaml#L10)). If it is not specified, the Gardener will try to find an adequate Seed cluster (one deployed in the same region at the same cloud provider) automatically. - -The cloud provider secrets can be stored in any namespace. With [`SecretBindings`](../../example/80-secretbinding-cloudprovider-aws.yaml) one can reference a secret in the same or in another namespace. These binding objects can also be used to reference `Quotas` for the specific secret. - -## Configuration file for Gardener controller manager -The Gardener controller manager does only support one command line flag which should be a path to a valid configuration file. - -Please take a look at [this](../../example/20-componentconfig-gardener-controller-manager.yaml) example configuration. - -## Configuration file for Gardener scheduler -The Gardener scheduler also only supports one command line flag which should be a path to a valid scheduler configuration file. - -Please take a look at [this](../../example/20-componentconfig-gardener-scheduler.yaml) example configuration. - -Information about the concepts of the gardener scheduler can be found [here](./scheduler.md) diff --git a/docs/testing/integration_tests.md b/docs/testing/integration_tests.md index c0fa3fd29a7..08c67d127b8 100644 --- a/docs/testing/integration_tests.md +++ b/docs/testing/integration_tests.md @@ -1,6 +1,6 @@ -# Testing Manual +# Integration Testing Manual -This manual describes the Gardener integration test directory structure, +This manual describes the Gardener integration test directory structure. ## Structure diff --git a/docs/usage/configuration.md b/docs/usage/configuration.md new file mode 100644 index 00000000000..8c854fc98d6 --- /dev/null +++ b/docs/usage/configuration.md @@ -0,0 +1,131 @@ +# Gardener Configuration and Usage + +Gardener automates the full lifecycle of Kubernetes clusters as a service. +Additionally, it has several extension points allowing external controllers to plug-in to the lifecycle. +As a consequence, there are several configuration options for the various custom resources that are partially required. + +This document describes the + +1. [configuration and usage of Gardener as operator/administrator](#configuration-and-usage-of-gardener-as-operatoradministrator). +2. [configuration and usage of Gardener as end-user/stakeholder/customer](#configuration-and-usage-of-gardener-as-end-userstakeholdercustomer). + +## Configuration and Usage of Gardener as Operator/Administrator + +When we use the terms "operator/administrator" we refer to both the people deploying and operating Gardener. +Gardener consists out of three components: + +1. `gardener-apiserver`, a Kubernetes-native API extension that serves custom resources in the Kubernetes-style (like `Seed`s and `Shoot`s), and a component that contains multiple admission plugins. +1. `gardener-controller-manager`, a component consisting out of multiple controllers that implement reconciliation and deletion flows for the various custom resources (e.g., it contains the logic for reconciliation and deletion of `Shoot`s). +1. `gardener-scheduler`, a component that assigns newly created `Shoot` clusters to appropriate `Seed` clusters. + +Each of these components have various configuration options. +The `gardener-apiserver` uses the standard API server library maintained by the Kubernetes community, and as such it mainly supports command line flags. +The two other components are using so-called componentconfig files that describe their configuration in a Kubernetes-style versioned object. + +### Configuration file for Gardener controller manager + +The Gardener controller manager does only support one command line flag which should be a path to a valid configuration file. +Please take a look at [this](../../example/20-componentconfig-gardener-controller-manager.yaml) example configuration. + +### Configuration file for Gardener scheduler + +The Gardener scheduler also only supports one command line flag which should be a path to a valid scheduler configuration file. +Please take a look at [this](../../example/20-componentconfig-gardener-scheduler.yaml) example configuration. +Information about the concepts of the gardener scheduler can be found [here](./scheduler.md) + +### System configuration + +After successful deployment of the three components you need to setup the system. +Let's first focus on some "static" configuration. +When the `gardener-controller-manager` starts it scans the `garden` namespace of the garden cluster for `Secret`s that have influence on its reconciliation loops, mainly the `Shoot` reconciliation: + +* **Internal domain secret**, contains the DNS provider credentials (having appropriate privileges) which will be used to create/delete so-called "internal" DNS records for the Shoot clusters, please see [this](../../example/10-secret-internal-domain.yaml) for an example. + * This secret is used in order to establish a stable endpoint for shoot clusters which is used internally by all control plane components. + * The DNS records are normal DNS records but called "internal" in our scenario because only the kubeconfigs for the control plane components use this endpoint when talking to the shoot clusters. + * It is forbidden to change the internal domain secret if there are existing shoot clusters. + +* **Default domain secrets** (optional), contain the DNS provider credentials (having appropriate privileges) which will be used to create/delete DNS records for a default domain for shoots (e.g., `example.com`), please see [this](../../example/10-secret-default-domain.yaml) for an example. + * Not every end-user/stakeholder/customer has its own domain, however, Gardener needs to create a DNS record for every shoot cluster. + * As landscape operator you might want to define a default domain owned and controlled by you that is used for all shoot clusters that don't specify their own domain. + +* **Alerting SMTP secrets** (optional), contain the SMTP credentials which will be used by the [AlertmMnager](https://prometheus.io/docs/alerting/alertmanager/) to send emails for alerts, please see [this](../../example/10-secret-alerting-smtp.yaml) for an example. + * These secrets are used by the AlertManager which is deployed next to the Kubernetes control plane of a shoot cluster in seed clusters. + * In case there have been alerting SMTP secrets configured, the Gardener will inject the credentials in the configuration of the AlertManager. + * It will use them to send mails to the stated email address in case anything is wrong with the Shoot clusters. + +* **OpenVPN Diffie-Hellmann Key secret** (optional), contains the self-generated Diffie-Hellmann key used by OpenVPN in your landscape, please see [this](../../example/10-secret-openvpn-diffie-hellman.yaml) for an example. + * If you don't specify a custom key then a default key is used, but for productive landscapes it's recommend to create a landscape-specific key and define it. + +* **Global monitoring secrets** (optional), contains basic authentication credentials for the Prometheus aggregating metrics for all clusters. + * These secrets are synced to each seed cluster and used to gain access to the aggregate monitoring components. + +Apart from this "static" configuration there are several custom resources extending the Kubernetes API and used by Gardener. +As an operator/administrator you have to configure some of them to make the system work. + +### `CloudProfile`s + +`CloudProfile`s are resources that describe a specific environment of an underlying infrastructure provider, e.g. AWS, Azure, etc. +Each shoot has to reference a `CloudProfile` to declare the environment it should be created in. +In a `CloudProfile` you specify certain constraints like available machine types, regions, which Kubernetes versions you want to offer, etc. +End-users can read `CloudProfile`s to see these values, but only operators can change the content or create/delete them. +When a shoot is created or updated then an admission plugin checks that only values are used that are allowed via the referenced `CloudProfile`. + +Additionally, a `CloudProfile` may contain a `providerConfig` which is a special configuration dedicated for the infrastructure provider. +Gardener does not evaluate or understand this config, but extension controllers might need for declaration of provider-specific constraints, or global settings. + +Please see [this](../../example/30-cloudprofile.yaml) example manifest and consult the documentation of your provider extension controller to get information about its `providerConfig`. + +### `Seed`s + +`Seed`s are resources that represent seed clusters. +Gardener does not care about how a seed cluster got created - the only requirement is that it is of at least Kubernetes v1.11 and passes the Kubernetes conformance tests. +You have to provide the seed's kubeconfig inside a secret that is referenced by the `Seed` resource. + +Please see [this](../../example/40-secret-seed.yaml), [this](../../example/40-secret-seed-backup.yaml), and [this](../../example/50-seed.yaml) example manifest. + +### `Quota`s + +In order to allow end-user not having their own dedicated infrastructure account to try out Gardener you can register an account owned by you that you use for trial clusters. +Trial clusters can be put under quota such that they don't consume too many resources (resulting in costs), and so that one user cannot consume all resources on his own. +These clusters are automatically terminated after a specified time, but end-users may extend the lifetime manually if needed. + +Please see [this](../../example/60-quota.yaml) example manifest. + +## Configuration and Usage of Gardener as End-User/Stakeholder/Customer + +As an end-user/stakeholder/customer you are using a Gardener landscape that has been setup for you by another team. +You don't need to care about how Gardener itself has to be configured or how it has to be deployed. + +### `Project`s + +The first thing before creating a shoot cluster is to create a `Project`. +A project is used to group multiple shoot clusters together. +You can invite colleagues to the project to enable collaboration, and you can either make them `admin` or `viewer`. +After you have created a project you will get a dedicated namespace in the garden cluster for all your shoots. + +Please see [this](../../example/05-project-dev.yaml) example manifest. + +### `SecretBinding`s + +Now that you have a namespace the next step is registering your infrastructure provider account. + +Please see [this](../../example/70-secret-provider.yaml) example manifest and consult the documentation of the extension controller for your infrastructure provider to get information about which keys are required in this secret. + +After the secret has been created you have to create a special `SecretBinding` resource that binds this secret. +Later when creating shoot clusters you will reference such a binding. + +Please see [this](../../example/80-secretbinding.yaml) example manifest. + +### `Shoot`s + +Shoot cluster contain various settings that influence how your Kubernetes cluster will look like in the end. +As Gardener heavily relies on extension controllers for operating system configuration, networking, and infrastructure specifics, you have the possibility (and duty) to provide these provider-specific configurations as well. +Such configurations are not evaluated by Gardener (because it doesn't know/understand them), but they are only transported to the respective extension controller. + +:warning: This means that any configuration issues/mistake on your side that relates to a provider-specific flag or setting cannot be caught during the update request itself but only later during the reconciliation. + +Please see [this](../../example/90-shoot.yaml) example manifest and consult the documentation of the provider extension controller to get information about its `spec.provider.controlPlaneConfig`, `.spec.provider.infrastructureConfig`, and `.spec.provider.workers[].providerConfig`. + +### `(Cluster)OpenIDConnectPreset`s + +Please see [this](./openidconnect-presets.md) separate documentation file. diff --git a/docs/concepts/openidconnect-presets.md b/docs/usage/openidconnect-presets.md similarity index 100% rename from docs/concepts/openidconnect-presets.md rename to docs/usage/openidconnect-presets.md diff --git a/example/80-secretbinding-provider.yaml b/example/80-secretbinding.yaml similarity index 100% rename from example/80-secretbinding-provider.yaml rename to example/80-secretbinding.yaml diff --git a/pkg/operation/seed/seed.go b/pkg/operation/seed/seed.go index 1b8989f759a..5171bf70c8c 100644 --- a/pkg/operation/seed/seed.go +++ b/pkg/operation/seed/seed.go @@ -22,7 +22,6 @@ import ( "strings" v1alpha1constants "github.com/gardener/gardener/pkg/apis/core/v1alpha1/constants" - "github.com/gardener/gardener/pkg/apis/garden" gardenv1beta1 "github.com/gardener/gardener/pkg/apis/garden/v1beta1" "github.com/gardener/gardener/pkg/apis/garden/v1beta1/helper"