Enhance minimal distance algorithm in gardener-scheduler (gardener#…

…8277)

* Replace mock client in scheduler test

* Add scheduler region config support

Co-authored-by: Rafael Franzke <rafael.franzke@sap.com>

* Refactor scheduler integration tests

* Allow referencing multiple cloudprofiles

* Add documentation

* Address review comments

* Address review comments(II)

* Assume mininmal distance to region itself

* Add required permissions

* Address nits

* Address yet another nit

---------

Co-authored-by: Rafael Franzke <rafael.franzke@sap.com>

charts/gardener/controlplane/charts/application/templates/clusterrole-scheduler.yaml

@@ -33,6 +33,8 @@ rules:
   - create
   - delete
   - get
+  - list
+  - watch
   - patch
   - update
 - apiGroups:

cmd/gardener-scheduler/app/app.go

@@ -26,22 +26,29 @@ import (
 	"github.com/go-logr/logr"
 	"github.com/spf13/cobra"
 	"github.com/spf13/pflag"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/labels"
+	"k8s.io/apimachinery/pkg/selection"
 	"k8s.io/component-base/version"
 	"k8s.io/component-base/version/verflag"
 	"k8s.io/klog/v2"
 	"k8s.io/utils/pointer"
+	"sigs.k8s.io/controller-runtime/pkg/cache"
+	"sigs.k8s.io/controller-runtime/pkg/client"
 	controllerconfigv1alpha1 "sigs.k8s.io/controller-runtime/pkg/config/v1alpha1"
 	"sigs.k8s.io/controller-runtime/pkg/healthz"
 	logf "sigs.k8s.io/controller-runtime/pkg/log"
 	"sigs.k8s.io/controller-runtime/pkg/manager"
 
+	v1beta1constants "github.com/gardener/gardener/pkg/apis/core/v1beta1/constants"
 	"github.com/gardener/gardener/pkg/client/kubernetes"
 	"github.com/gardener/gardener/pkg/controllerutils/routes"
 	"github.com/gardener/gardener/pkg/features"
 	gardenerhealthz "github.com/gardener/gardener/pkg/healthz"
 	"github.com/gardener/gardener/pkg/logger"
 	"github.com/gardener/gardener/pkg/scheduler/apis/config"
 	"github.com/gardener/gardener/pkg/scheduler/controller"
+	"github.com/gardener/gardener/pkg/utils"
 )
 
 // Name is a const for the name of this component.
@@ -116,6 +123,13 @@ func run(ctx context.Context, log logr.Logger, cfg *config.SchedulerConfiguratio
 		HealthProbeBindAddress: net.JoinHostPort(cfg.Server.HealthProbes.BindAddress, strconv.Itoa(cfg.Server.HealthProbes.Port)),
 		MetricsBindAddress:     net.JoinHostPort(cfg.Server.Metrics.BindAddress, strconv.Itoa(cfg.Server.Metrics.Port)),
 
+		NewCache: cache.BuilderWithOptions(cache.Options{
+			SelectorsByObject: map[client.Object]cache.ObjectSelector{
+				&corev1.ConfigMap{}: {
+					Label: labels.NewSelector().Add(utils.MustNewRequirement(v1beta1constants.SchedulingPurpose, selection.Equals, v1beta1constants.SchedulingPurposeRegionConfig)),
+				},
+			},
+		}),
 		LeaderElection:                cfg.LeaderElection.LeaderElect,
 		LeaderElectionResourceLock:    cfg.LeaderElection.ResourceLock,
 		LeaderElectionID:              cfg.LeaderElection.ResourceName,

docs/concepts/scheduler.md

@@ -12,7 +12,7 @@ The following sections explain the configuration and flow in greater detail.
 
 Previously, an admission plugin in the Gardener API server conducted the scheduling decisions.
 This implies changes to the API server whenever adjustments of the scheduling are needed.
-Decoupling the API server and the scheduler comes with greater flexibility to develop these components independently from each other.
+Decoupling the API server and the scheduler comes with greater flexibility to develop these components independently.
 
 ### 2. Extensibility
 
@@ -38,6 +38,10 @@ The following **sequence** describes the steps involved to determine a seed cand
 1. Apply active [strategy](#strategies) e.g., _Minimal Distance strategy_
 1. Choose least utilized seed, i.e., the one with the least number of shoot control planes, will be the winner and written to the `.spec.seedName` field of the `Shoot`.
 
+In order to put the scheduling decision into effect, the scheduler sends an update request for the `Shoot` resource to
+the API server. After validation, the `gardener-apiserver` updates the `Shoot` to have the `spec.seedName` field set.
+Subsequently, the `gardenlet` picks up and starts to create the cluster on the specified seed.
+
 ## Configuration
 
 The Gardener Scheduler configuration has to be supplied on startup. It is a mandatory and also the only available flag.
@@ -51,31 +55,59 @@ However, the Gardener Scheduler on the other hand does not need a TLS configurat
 The scheduling strategy is defined in the _**candidateDeterminationStrategy**_ of the scheduler's configuration and can have the possible values `SameRegion` and `MinimalDistance`.
 The `SameRegion` strategy is the default strategy.
 
-1. *Same Region strategy*
+### Same Region strategy
 
-   The Gardener Scheduler reads the `spec.provider.type` and `.spec.region` fields from the `Shoot` resource.
+The Gardener Scheduler reads the `spec.provider.type` and `.spec.region` fields from the `Shoot` resource.
 It tries to find a seed that has the identical `.spec.provider.type` and `.spec.provider.region` fields set.
 If it cannot find a suitable seed, it adds an event to the shoot stating that it is unschedulable.
 
-2. *Minimal Distance strategy*
-
-   The Gardener Scheduler tries to find a valid seed with minimal distance to the shoot's intended region.
-The distance is calculated based on the Levenshtein distance of the region. Therefore, the region name
+### Minimal Distance strategy
+
+The Gardener Scheduler tries to find a valid seed with minimal distance to the shoot's intended region.
+Distances are configured via `ConfigMap`(s), usually per cloud provider in a Gardener landscape.
+The configuration is structured like this:
+- It refers to one or multiple `CloudProfile`s via annotation `scheduling.gardener.cloud/cloudprofiles`.
+- It contains the declaration as `region-config` via label `scheduling.gardener.cloud/purpose`.
+- If a `CloudProfile` is referred by multiple `ConfigMap`s, only the first one is considered.
+- The `data` fields configure actual distances, where _key_ relates to the `Shoot` region and _value_ contains distances to `Seed` regions.
+
+```yaml
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: <name>
+  namespace: garden
+  annotations:
+    scheduling.gardener.cloud/cloudprofiles: cloudprofile-name-1{,optional-cloudprofile-name-2,...}
+  labels:
+    scheduling.gardener.cloud/purpose: region-config
+data:
+  region-1: |
+    region-2: 10
+    region-3: 20
+    ...
+  region-2: |
+    region-1: 10
+    region-3: 10
+    ...
+```
+
+> Gardener provider extensions for public cloud providers usually have an example weight `ConfigMap` in their repositories.
+> We suggest to check them out before defining your own data.
+
+If a valid seed candidate cannot be found after consulting the distance configuration, the scheduler will fall back to 
+the Levenshtein distance to find the closest region. Therefore, the region name
 is split into a base name and an orientation. Possible orientations are `north`, `south`, `east`, `west` and `central`.
 The distance then is twice the Levenshtein distance of the region's base name plus a correction value based on the
 orientation and the provider.
 
-   If the orientations of shoot and seed candidate match, the correction value is 0, if they differ it is 2 and if
+If the orientations of shoot and seed candidate match, the correction value is 0, if they differ it is 2 and if
 either the seed's or the shoot's region does not have an orientation it is 1.
 If the provider differs, the correction value is additionally incremented by 2.
 
-   Because of this, a matching region with a matching provider is always prefered.
-
-In order to put the scheduling decision into effect, the scheduler sends an update request for the `Shoot` resource to
-the API server. After validation, the Gardener Aggregated API server updates the shoot to have the `spec.seedName` field set.
-Subsequently, the Gardenlet picks up and starts to create the cluster on the specified seed.
+Because of this, a matching region with a matching provider is always prefered.
 
-3. *Special handling based on shoot cluster purpose*
+### Special handling based on shoot cluster purpose
 
 Every shoot cluster can have a purpose that describes what the cluster is used for, and also influences how the cluster is setup (see [Shoot Cluster Purpose](../usage/shoot_purposes.md) for more information).
 

example/31-scheduler-region-configmap.yaml

@@ -0,0 +1,16 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: cloudprofile1-scheduling-regions
+  namespace: garden
+  annotations:
+    scheduling.gardener.cloud/cloudprofiles: cloudprofile1{,optional-cloudprofile-name-2,...}
+  labels:
+    scheduling.gardener.cloud/purpose: region-config
+data:
+  europe-west-1: |
+    europe-central-1: 50
+    europe-north-1: 60
+    europe-south-1: 70
+    europe-east-1: 120
+    us-west-1: 220

pkg/apis/core/v1beta1/constants/types_constants.go

@@ -567,6 +567,13 @@ const (
 
 	// DefaultSchedulerName is the name of the default scheduler.
 	DefaultSchedulerName = "default-scheduler"
+	// SchedulingPurpose is a constant for the key in a label describing the purpose of the scheduler related object.
+	SchedulingPurpose = "scheduling.gardener.cloud/purpose"
+	// SchedulingPurposeRegionConfig is a constant for a label value indicating that the object should be considered as a region config.
+	SchedulingPurposeRegionConfig = "region-config"
+	// AnnotationSchedulingCloudProfiles is a constant for an annotation key on a configmap which denotes
+	// the linked cloudprofiles containing the region distances.
+	AnnotationSchedulingCloudProfiles = "scheduling.gardener.cloud/cloudprofiles"
 
 	// AnnotationManagedSeedAPIServer is a constant for an annotation on a Shoot resource containing the API server settings for a managed seed.
 	AnnotationManagedSeedAPIServer = "shoot.gardener.cloud/managed-seed-api-server"

pkg/component/gardenerscheduler/gardener_scheduler_test.go

@@ -208,7 +208,7 @@ var _ = Describe("GardenerScheduler", func() {
 				{
 					APIGroups: []string{""},
 					Resources: []string{"configmaps"},
-					Verbs:     []string{"create", "delete", "get", "patch", "update"},
+					Verbs:     []string{"create", "delete", "get", "list", "watch", "patch", "update"},
 				},
 				{
 					APIGroups: []string{gardencorev1beta1.GroupName},

pkg/component/gardenerscheduler/rbac.go

@@ -43,7 +43,7 @@ func (g *gardenerScheduler) clusterRole() *rbacv1.ClusterRole {
 			{
 				APIGroups: []string{""},
 				Resources: []string{"configmaps"},
-				Verbs:     []string{"create", "delete", "get", "patch", "update"},
+				Verbs:     []string{"create", "delete", "get", "list", "watch", "patch", "update"},
 			},
 			{
 				APIGroups: []string{gardencorev1beta1.GroupName},

pkg/scheduler/controller/shoot/add.go

@@ -38,6 +38,9 @@ func (r *Reconciler) AddToManager(mgr manager.Manager) error {
 	if r.Recorder == nil {
 		r.Recorder = mgr.GetEventRecorderFor(ControllerName + "-scheduler")
 	}
+	if r.GardenNamespace == "" {
+		r.GardenNamespace = v1beta1constants.GardenNamespace
+	}
 
 	return builder.
 		ControllerManagedBy(mgr).