[gardenlet] Switch shootsecret controller to controller-runtime (ga…

…rdener#6819)

* Minor documentation fix

* Add documentation for shoot secret controller

* Switch shoot secret controller to native controller-runtime controller

* Update skaffold depedencies

* Add unit test

* Address review

* Address review

docs/concepts/gardenlet.md

@@ -261,6 +261,16 @@ It checks the `.status.conditions` of the backing `ManagedResource` created in t
 
 A `ControllerInstallation` is considered "healthy" if `Applied=Healthy=true` and `Progressing=False`.
 
+### [`ShootState` Controller](../../pkg/gardenlet/controller/shootstate)
+
+The `ShootState` controller in the `gardenlet` reconciles resources containing information that have to be synced to the `ShootState`.
+This information is used when a [control plane migration](../usage/control_plane_migration.md) is performed.
+
+#### "Secret" Reconciler
+
+This reconciler reconciles `Secret`s having labels `managed-by=secrets-manager` and `persist=true` in the shoot namespaces in the seed cluster.
+It syncs them to the `ShootState` so that the secrets can be restored from there in case a shoot control plane has to be restored to another seed cluster (in case of migration).
+
 ## Managed Seeds
 
 Gardener users can use shoot clusters as seed clusters, so-called "managed seeds" (aka "shooted seeds"),
@@ -295,7 +305,6 @@ More information: [Deploy a Gardenlet](../deployment/deploy_gardenlet.md) for al
 
 ## Related Links
 
-- [Gardener Architecture](https://github.com/gardener/documentation/wiki/Architecture)
+- [Gardener Architecture](architecture.md)
 - [#356: Implement Gardener Scheduler](https://github.com/gardener/gardener/issues/356)
 - [#2309: Add /healthz endpoint for Gardenlet](https://github.com/gardener/gardener/pull/2309)
-

docs/deployment/getting_started_locally.md

@@ -145,7 +145,7 @@ In order to deploy required resources in the KinD cluster that you just created,
 make gardenlet-kind2-up
 ```
 
-The following steps assume that your are using the kubeconfig that points to the `gardener-local`  cluster (first KinD cluster): `export KUBECONFIG=example/gardener-local/kind/kubeconfig`.
+The following steps assume that you are using the kubeconfig that points to the `gardener-local` cluster (first KinD cluster): `export KUBECONFIG=example/gardener-local/kind/kubeconfig`.
 
 You can wait for the `local2` `Seed` to be ready by running:
 

docs/proposals/15-manage-bastions-and-ssh-key-pair-rotation.md

@@ -206,7 +206,7 @@ Currently, the `ssh` key pair for the shoot nodes are created once during shoot
 
 ### Rotation Proposal
 - `gardeneruser` original user data [component](https://github.com/gardener/gardener/tree/master/pkg/operation/botanist/component/extensions/operatingsystemconfig/original/components/gardeneruser):
-    - The `gardeneruser` create script should be changed into a reconcile script script, and renamed accordingly. It needs to be adapted so that the `authorized_keys` file will be updated / overwritten with the current and old `ssh` public key from the cloud-config user data.
+    - The `gardeneruser` create script should be changed into a reconcile script, and renamed accordingly. It needs to be adapted so that the `authorized_keys` file will be updated / overwritten with the current and old `ssh` public key from the cloud-config user data.
 - Rotation trigger:
     - Once in the maintenance time window
     - On demand, by annotating the shoot with `gardener.cloud/operation: rotate-ssh-keypair`

docs/proposals/16-adminkubeconfig-subresource.md

@@ -119,7 +119,7 @@ status:
 
 This is needed, because the Gardener API server might not know on which IP address the API server is advertised on (e.g. DNS is disabled).
 
-If there are multiple entries, each would be added in a separate `cluster` in the `kubeconfig` and a `context` with the same name would be added added as well. The current context would be selected as the first entry in the `advertisedAddresses` list (`.status.advertisedAddresses[0]`).
+If there are multiple entries, each would be added in a separate `cluster` in the `kubeconfig` and a `context` with the same name would be added as well. The current context would be selected as the first entry in the `advertisedAddresses` list (`.status.advertisedAddresses[0]`).
 
 ## Alternatives
 

docs/usage/logging.md

@@ -6,7 +6,7 @@ Kubernetes uses the underlying container runtime logging, which does not persist
 
 ### Components:
 ![](images/logging-architecture.png)
-* A Fluent-bit daemonset which works like a log collector and custom custom Golang plugin which spreads log messages to their Loki instances
+* A Fluent-bit daemonset which works like a log collector and custom Golang plugin which spreads log messages to their Loki instances
 * One Loki Statefulset in the `garden` namespace which contains logs for the seed cluster and one per shoot namespace which contains logs for shoot's controlplane.
 * One Grafana Deployment in `garden` namespace and two Deployments per shoot namespace (one exposed to the end users and one for the operators). Grafana is the UI component used in the logging stack.
 

docs/usage/managed_seed.md

@@ -82,7 +82,7 @@ There are few configuration options that are not supported in a `Shoot` resource
 Option | Description
 --- | ---
 `apiServer.autoscaler.minReplicas` | Controls the minimum number of `kube-apiserver` replicas for the shoot registered as seed cluster.
-`apiServer.autoscaler.maxReplicas` | Controls the maximum number of `kube-apiserver` replicas for the the shoot registered as seed cluster.
+`apiServer.autoscaler.maxReplicas` | Controls the maximum number of `kube-apiserver` replicas for the shoot registered as seed cluster.
 `apiServer.replicas` | Controls how many `kube-apiserver` replicas the shoot registered as seed cluster gets by default.
 
 It is possible to specify these options via the `shoot.gardener.cloud/managed-seed-api-server` annotation on the Shoot resource. Example configuration:

docs/usage/shoot_credentials_rotation.md

@@ -19,7 +19,7 @@ The following steps are required to perform the rotation:
 
 1. Update the data in the `Secret` with new credentials.
 2. ⚠️ Wait until all `Shoot`s using the `Secret` are reconciled before you disable the old credentials in your cloud provider account! Otherwise, the `Shoot`s will no longer work as expected. Check out [this document](shoot_operations.md#immediate-reconciliation) to learn how to trigger a reconciliation of your `Shoot`s.
-3. After all `Shoot`s using the `Secret` were reconciled, you can go ahead and deactivate the old credentials in your provider account account.
+3. After all `Shoot`s using the `Secret` were reconciled, you can go ahead and deactivate the old credentials in your provider account.
 
 ## Gardener-Provided Credentials
 

docs/usage/shoot_versions.md

@@ -53,7 +53,7 @@ Typically for Kubernetes versions, the latest Kubernetes patch versions of the a
 
 - **deprecated:** A `deprecated` version is a version that approaches the end of its lifecycle and can contain issues which are probably resolved in a supported version.
 New Shoots should not use this version any more.
-Existing Shoots will be updated to a newer version if `auto-update` is enabled (`.spec.maintenance.autoUpdate.kubernetesVersion` for Kubernetes version `auto-update`, or `.spec.maintenance.autoUpdate.machineImageVersion` for machine machine image version `auto-update`).
+Existing Shoots will be updated to a newer version if `auto-update` is enabled (`.spec.maintenance.autoUpdate.kubernetesVersion` for Kubernetes version `auto-update`, or `.spec.maintenance.autoUpdate.machineImageVersion` for machine image version `auto-update`).
 Using automatic upgrades, however, does not guarantee that a Shoot runs a non-deprecated version, as the latest version (overall or of the minor version) can be deprecated as well.
 Deprecated versions **should** have an expiration date set for eventual expiration.
 

pkg/gardenlet/controller/add.go

@@ -24,6 +24,7 @@ import (
 	"github.com/gardener/gardener/pkg/client/kubernetes"
 	"github.com/gardener/gardener/pkg/gardenlet/apis/config"
 	"github.com/gardener/gardener/pkg/gardenlet/controller/controllerinstallation"
+	"github.com/gardener/gardener/pkg/gardenlet/controller/shootstate"
 )
 
 // AddControllersToManager adds all gardenlet controllers to the given manager.
@@ -45,5 +46,9 @@ func AddControllersToManager(
 		return fmt.Errorf("failed adding ControllerInstallation controller: %w", err)
 	}
 
+	if err := shootstate.AddToManager(mgr, gardenCluster, seedCluster, *cfg); err != nil {
+		return fmt.Errorf("failed adding ShootState controller: %w", err)
+	}
+
 	return nil
 }

pkg/gardenlet/controller/factory.go

@@ -33,7 +33,6 @@ import (
 	networkpolicycontroller "github.com/gardener/gardener/pkg/gardenlet/controller/networkpolicy"
 	seedcontroller "github.com/gardener/gardener/pkg/gardenlet/controller/seed"
 	shootcontroller "github.com/gardener/gardener/pkg/gardenlet/controller/shoot"
-	shootsecretcontroller "github.com/gardener/gardener/pkg/gardenlet/controller/shootsecret"
 	"github.com/gardener/gardener/pkg/healthz"
 	"github.com/gardener/gardener/pkg/utils/imagevector"
 	"github.com/gardener/gardener/pkg/utils/retry"
@@ -108,11 +107,6 @@ func (f *LegacyControllerFactory) Start(ctx context.Context) error {
 		return fmt.Errorf("failed initializing NetworkPolicy controller: %w", err)
 	}
 
-	secretController, err := shootsecretcontroller.NewController(ctx, log, f.GardenCluster, f.SeedCluster)
-	if err != nil {
-		return fmt.Errorf("failed initializing Secret controller: %w", err)
-	}
-
 	seedController, err := seedcontroller.NewSeedController(ctx, log, f.GardenCluster, f.SeedClientSet, f.HealthManager, imageVector, componentImageVectors, identity, f.Config)
 	if err != nil {
 		return fmt.Errorf("failed initializing Seed controller: %w", err)
@@ -131,7 +125,6 @@ func (f *LegacyControllerFactory) Start(ctx context.Context) error {
 	go bastionController.Run(controllerCtx, *f.Config.Controllers.Bastion.ConcurrentSyncs)
 	go managedSeedController.Run(controllerCtx, *f.Config.Controllers.ManagedSeed.ConcurrentSyncs)
 	go networkPolicyController.Run(controllerCtx, *f.Config.Controllers.SeedAPIServerNetworkPolicy.ConcurrentSyncs)
-	go secretController.Run(controllerCtx, *f.Config.Controllers.ShootSecret.ConcurrentSyncs)
 	go seedController.Run(controllerCtx, *f.Config.Controllers.Seed.ConcurrentSyncs)
 	go shootController.Run(controllerCtx, *f.Config.Controllers.Shoot.ConcurrentSyncs, *f.Config.Controllers.ShootCare.ConcurrentSyncs, *f.Config.Controllers.ShootMigration.ConcurrentSyncs)
 

pkg/gardenlet/controller/shootstate/add.go

@@ -0,0 +1,41 @@
+// Copyright (c) 2022 SAP SE or an SAP affiliate company. All rights reserved. This file is licensed under the Apache Software License, v. 2 except as noted otherwise in the LICENSE file
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package shootstate
+
+import (
+	"fmt"
+
+	"sigs.k8s.io/controller-runtime/pkg/cluster"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+
+	"github.com/gardener/gardener/pkg/gardenlet/apis/config"
+	"github.com/gardener/gardener/pkg/gardenlet/controller/shootstate/secret"
+)
+
+// AddToManager adds Controllers to the given manager.
+func AddToManager(
+	mgr manager.Manager,
+	gardenCluster cluster.Cluster,
+	seedCluster cluster.Cluster,
+	cfg config.GardenletConfiguration,
+) error {
+	if err := (&secret.Reconciler{
+		Config: *cfg.Controllers.ShootSecret,
+	}).AddToManager(mgr, gardenCluster, seedCluster); err != nil {
+		return fmt.Errorf("failed adding secret reconciler: %w", err)
+	}
+
+	return nil
+}

pkg/gardenlet/controller/shootstate/secret/add.go

@@ -0,0 +1,61 @@
+// Copyright (c) 2022 SAP SE or an SAP affiliate company. All rights reserved. This file is licensed under the Apache Software License, v. 2 except as noted otherwise in the LICENSE file
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package secret
+
+import (
+	secretsmanager "github.com/gardener/gardener/pkg/utils/secrets/manager"
+
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/client-go/util/workqueue"
+	"sigs.k8s.io/controller-runtime/pkg/builder"
+	"sigs.k8s.io/controller-runtime/pkg/client"
+	"sigs.k8s.io/controller-runtime/pkg/cluster"
+	"sigs.k8s.io/controller-runtime/pkg/controller"
+	"sigs.k8s.io/controller-runtime/pkg/manager"
+	"sigs.k8s.io/controller-runtime/pkg/predicate"
+)
+
+// ControllerName is the name of this controller.
+const ControllerName = "shootstate-secret"
+
+// AddToManager adds Reconciler to the given manager.
+func (r *Reconciler) AddToManager(mgr manager.Manager, gardenCluster, seedCluster cluster.Cluster) error {
+	if r.GardenClient == nil {
+		r.GardenClient = gardenCluster.GetClient()
+	}
+	if r.SeedClient == nil {
+		r.SeedClient = seedCluster.GetClient()
+	}
+
+	return builder.
+		ControllerManagedBy(mgr).
+		Named(ControllerName).
+		For(&corev1.Secret{}, builder.WithPredicates(r.SecretPredicate())).
+		WithOptions(controller.Options{
+			MaxConcurrentReconciles: *r.Config.ConcurrentSyncs,
+			RecoverPanic:            true,
+			RateLimiter:             workqueue.DefaultControllerRateLimiter(),
+		}).
+		Complete(r)
+}
+
+// SecretPredicate returns the predicates for the secret watch.
+func (r *Reconciler) SecretPredicate() predicate.Predicate {
+	return predicate.NewPredicateFuncs(func(object client.Object) bool {
+		labels := object.GetLabels()
+		return labels[secretsmanager.LabelKeyManagedBy] == secretsmanager.LabelValueSecretsManager &&
+			labels[secretsmanager.LabelKeyPersist] == secretsmanager.LabelValueTrue
+	})
+}