LoadBalancer toggles for master and replica pooler pods (#1799)

* Add support for pooler load balancer

Signed-off-by: Sergey Shatunov <me@prok.pw>

* Rename to enable_master_pooler_load_balancer

Signed-off-by: Sergey Shatunov <me@prok.pw>

* target port should be intval
* enhance pooler e2e test
* add new options to crds.go

Co-authored-by: Sergey Shatunov <me@prok.pw>

Author: FxKu

charts/postgres-operator/crds/operatorconfigurations.yaml

@@ -380,9 +380,15 @@ spec:
                   enable_master_load_balancer:
                     type: boolean
                     default: true
+                  enable_master_pooler_load_balancer:
+                    type: boolean
+                    default: false
                   enable_replica_load_balancer:
                     type: boolean
                     default: false
+                  enable_replica_pooler_load_balancer:
+                    type: boolean
+                    default: false
                   external_traffic_policy:
                     type: string
                     enum:

charts/postgres-operator/crds/postgresqls.yaml

@@ -197,8 +197,12 @@ spec:
                 type: boolean
               enableMasterLoadBalancer:
                 type: boolean
+              enableMasterPoolerLoadBalancer:
+                type: boolean
               enableReplicaLoadBalancer:
                 type: boolean
+              enableReplicaPoolerLoadBalancer:
+                type: boolean
               enableShmVolume:
                 type: boolean
               init_containers:

charts/postgres-operator/values.yaml

@@ -228,8 +228,12 @@ configLoadBalancer:
 
   # toggles service type load balancer pointing to the master pod of the cluster
   enable_master_load_balancer: false
+  # toggles service type load balancer pointing to the master pooler pod of the cluster
+  enable_master_pooler_load_balancer: false
   # toggles service type load balancer pointing to the replica pod of the cluster
   enable_replica_load_balancer: false
+  # toggles service type load balancer pointing to the replica pooler pod of the cluster
+  enable_replica_pooler_load_balancer: false
   # define external traffic policy for the load balancer
   external_traffic_policy: "Cluster"
   # defines the DNS name string template for the master load balancer cluster

docs/administrator.md

@@ -750,6 +750,11 @@ lead to K8s removing this field from the manifest due to its
 Then the resultant manifest will not contain the necessary change, and the
 operator will respectively do nothing with the existing source ranges.
 
+Load balancer services can also be enabled for the [connection pooler](user.md#connection-pooler)
+pods with manifest flags `enableMasterPoolerLoadBalancer` and/or
+`enableReplicaPoolerLoadBalancer` or in the operator configuration with
+`enable_master_pooler_load_balancer` and/or `enable_replica_pooler_load_balancer`.
+
 ## Running periodic 'autorepair' scans of K8s objects
 
 The Postgres Operator periodically scans all K8s objects belonging to each

docs/reference/cluster_manifest.md

@@ -91,11 +91,23 @@ These parameters are grouped directly under  the `spec` key in the manifest.
   `enable_master_load_balancer` parameter) to define whether to enable the load
   balancer pointing to the Postgres primary. Optional.
 
+* **enableMasterPoolerLoadBalancer**
+  boolean flag to override the operator defaults (set by the
+  `enable_master_pooler_load_balancer` parameter) to define whether to enable
+  the load balancer for master pooler pods pointing to the Postgres primary.
+  Optional.
+
 * **enableReplicaLoadBalancer**
   boolean flag to override the operator defaults (set by the
   `enable_replica_load_balancer` parameter) to define whether to enable the
   load balancer pointing to the Postgres standby instances. Optional.
 
+* **enableReplicaPoolerLoadBalancer**
+  boolean flag to override the operator defaults (set by the
+  `enable_replica_pooler_load_balancer` parameter) to define whether to enable
+  the load balancer for replica pooler pods pointing to the Postgres standby
+  instances. Optional.
+
 * **allowedSourceRanges**
   when one or more load balancers are enabled for the cluster, this parameter
   defines the comma-separated range of IP networks (in CIDR-notation). The

docs/reference/operator_parameters.md

@@ -546,11 +546,21 @@ In the CRD-based configuration they are grouped under the `load_balancer` key.
   toggles service type load balancer pointing to the master pod of the cluster.
   Can be overridden by individual cluster settings. The default is `true`.
 
+* **enable_master_pooler_load_balancer**
+  toggles service type load balancer pointing to the master pooler pod of the
+  cluster. Can be overridden by individual cluster settings. The default is
+  `false`.
+
 * **enable_replica_load_balancer**
-  toggles service type load balancer pointing to the replica pod of the
-  cluster.  Can be overridden by individual cluster settings. The default is
+  toggles service type load balancer pointing to the replica pod(s) of the
+  cluster. Can be overridden by individual cluster settings. The default is
   `false`.
 
+* **enable_replica_pooler_load_balancer**
+  toggles service type load balancer pointing to the replica pooler pod(s) of
+  the cluster. Can be overridden by individual cluster settings. The default
+  is `false`.
+
 * **external_traffic_policy** defines external traffic policy for load
   balancers. Allowed values are `Cluster` (default) and `Local`.
 

e2e/tests/test_e2e.py

@@ -467,6 +467,9 @@ def test_enable_disable_connection_pooler(self):
         the end turn connection pooler off to not interfere with other tests.
         '''
         k8s = self.k8s
+        pooler_label = 'application=db-connection-pooler,cluster-name=acid-minimal-cluster'
+        master_pooler_label = 'connection-pooler=acid-minimal-cluster-pooler'
+        replica_pooler_label = master_pooler_label + '-repl'
         self.eventuallyEqual(lambda: k8s.get_operator_state(), {"0": "idle"}, "Operator does not get in sync")
 
         k8s.api.custom_objects_api.patch_namespaced_custom_object(
@@ -476,22 +479,23 @@ def test_enable_disable_connection_pooler(self):
                 'spec': {
                     'enableConnectionPooler': True,
                     'enableReplicaConnectionPooler': True,
+                    'enableMasterPoolerLoadBalancer': True,
+                    'enableReplicaPoolerLoadBalancer': True,
                 }
             })
+        self.eventuallyEqual(lambda: k8s.get_operator_state(), {"0": "idle"}, "Operator does not get in sync")
 
-        self.eventuallyEqual(lambda: k8s.get_deployment_replica_count(), 2,
-                             "Deployment replicas is 2 default")
-        self.eventuallyEqual(lambda: k8s.count_running_pods(
-                            "connection-pooler=acid-minimal-cluster-pooler"),
-                            2, "No pooler pods found")
-        self.eventuallyEqual(lambda: k8s.count_running_pods(
-                            "connection-pooler=acid-minimal-cluster-pooler-repl"),
-                            2, "No pooler replica pods found")
-        self.eventuallyEqual(lambda: k8s.count_services_with_label(
-                            'application=db-connection-pooler,cluster-name=acid-minimal-cluster'),
-                            2, "No pooler service found")
-        self.eventuallyEqual(lambda: k8s.count_secrets_with_label('application=db-connection-pooler,cluster-name=acid-minimal-cluster'),
-                             1, "Pooler secret not created")
+        self.eventuallyEqual(lambda: k8s.get_deployment_replica_count(), 2, "Deployment replicas is 2 default")
+        self.eventuallyEqual(lambda: k8s.count_running_pods(master_pooler_label), 2, "No pooler pods found")
+        self.eventuallyEqual(lambda: k8s.count_running_pods(replica_pooler_label), 2, "No pooler replica pods found")
+        self.eventuallyEqual(lambda: k8s.count_services_with_label(pooler_label), 2, "No pooler service found")
+        self.eventuallyEqual(lambda: k8s.count_secrets_with_label(pooler_label), 1, "Pooler secret not created")
+        self.eventuallyEqual(lambda: k8s.get_service_type(master_pooler_label+","+pooler_label),
+                             'LoadBalancer',
+                             "Expected LoadBalancer service type for master pooler pod, found {}")
+        self.eventuallyEqual(lambda: k8s.get_service_type(replica_pooler_label+","+pooler_label),
+                             'LoadBalancer',
+                             "Expected LoadBalancer service type for replica pooler pod, found {}")
 
         # Turn off only master connection pooler
         k8s.api.custom_objects_api.patch_namespaced_custom_object(
@@ -504,20 +508,17 @@ def test_enable_disable_connection_pooler(self):
                 }
             })
 
-        self.eventuallyEqual(lambda: k8s.get_operator_state(), {"0": "idle"},
-                             "Operator does not get in sync")
+        self.eventuallyEqual(lambda: k8s.get_operator_state(), {"0": "idle"}, "Operator does not get in sync")
+
         self.eventuallyEqual(lambda: k8s.get_deployment_replica_count(name="acid-minimal-cluster-pooler-repl"), 2,
                              "Deployment replicas is 2 default")
-        self.eventuallyEqual(lambda: k8s.count_running_pods(
-                             "connection-pooler=acid-minimal-cluster-pooler"),
+        self.eventuallyEqual(lambda: k8s.count_running_pods(master_pooler_label),
                              0, "Master pooler pods not deleted")
-        self.eventuallyEqual(lambda: k8s.count_running_pods(
-                             "connection-pooler=acid-minimal-cluster-pooler-repl"),
+        self.eventuallyEqual(lambda: k8s.count_running_pods(replica_pooler_label),
                              2, "Pooler replica pods not found")
-        self.eventuallyEqual(lambda: k8s.count_services_with_label(
-                             'application=db-connection-pooler,cluster-name=acid-minimal-cluster'),
+        self.eventuallyEqual(lambda: k8s.count_services_with_label(pooler_label),
                              1, "No pooler service found")
-        self.eventuallyEqual(lambda: k8s.count_secrets_with_label('application=db-connection-pooler,cluster-name=acid-minimal-cluster'),
+        self.eventuallyEqual(lambda: k8s.count_secrets_with_label(pooler_label),
                              1, "Secret not created")
 
         # Turn off only replica connection pooler
@@ -528,20 +529,24 @@ def test_enable_disable_connection_pooler(self):
                 'spec': {
                     'enableConnectionPooler': True,
                     'enableReplicaConnectionPooler': False,
+                    'enableMasterPoolerLoadBalancer': False,
                 }
             })
 
-        self.eventuallyEqual(lambda: k8s.get_operator_state(), {"0": "idle"},
-                             "Operator does not get in sync")
+        self.eventuallyEqual(lambda: k8s.get_operator_state(), {"0": "idle"}, "Operator does not get in sync")
+
         self.eventuallyEqual(lambda: k8s.get_deployment_replica_count(), 2,
                              "Deployment replicas is 2 default")
-        self.eventuallyEqual(lambda: k8s.count_running_pods("connection-pooler=acid-minimal-cluster-pooler"),
+        self.eventuallyEqual(lambda: k8s.count_running_pods(master_pooler_label),
                              2, "Master pooler pods not found")
-        self.eventuallyEqual(lambda: k8s.count_running_pods("connection-pooler=acid-minimal-cluster-pooler-repl"),
+        self.eventuallyEqual(lambda: k8s.count_running_pods(replica_pooler_label),
                              0, "Pooler replica pods not deleted")
-        self.eventuallyEqual(lambda: k8s.count_services_with_label('application=db-connection-pooler,cluster-name=acid-minimal-cluster'),
+        self.eventuallyEqual(lambda: k8s.count_services_with_label(pooler_label),
                              1, "No pooler service found")
-        self.eventuallyEqual(lambda: k8s.count_secrets_with_label('application=db-connection-pooler,cluster-name=acid-minimal-cluster'),
+        self.eventuallyEqual(lambda: k8s.get_service_type(master_pooler_label+","+pooler_label),
+                             'ClusterIP',
+                             "Expected LoadBalancer service type for master, found {}")
+        self.eventuallyEqual(lambda: k8s.count_secrets_with_label(pooler_label),
                              1, "Secret not created")
 
         # scale up connection pooler deployment
@@ -558,7 +563,7 @@ def test_enable_disable_connection_pooler(self):
 
         self.eventuallyEqual(lambda: k8s.get_deployment_replica_count(), 3,
                              "Deployment replicas is scaled to 3")
-        self.eventuallyEqual(lambda: k8s.count_running_pods("connection-pooler=acid-minimal-cluster-pooler"),
+        self.eventuallyEqual(lambda: k8s.count_running_pods(master_pooler_label),
                              3, "Scale up of pooler pods does not work")
 
         # turn it off, keeping config should be overwritten by false
@@ -569,12 +574,13 @@ def test_enable_disable_connection_pooler(self):
                 'spec': {
                     'enableConnectionPooler': False,
                     'enableReplicaConnectionPooler': False,
+                    'enableReplicaPoolerLoadBalancer': False,
                 }
             })
 
-        self.eventuallyEqual(lambda: k8s.count_running_pods("connection-pooler=acid-minimal-cluster-pooler"),
+        self.eventuallyEqual(lambda: k8s.count_running_pods(master_pooler_label),
                              0, "Pooler pods not scaled down")
-        self.eventuallyEqual(lambda: k8s.count_services_with_label('application=db-connection-pooler,cluster-name=acid-minimal-cluster'),
+        self.eventuallyEqual(lambda: k8s.count_services_with_label(pooler_label),
                              0, "Pooler service not removed")
         self.eventuallyEqual(lambda: k8s.count_secrets_with_label('application=spilo,cluster-name=acid-minimal-cluster'),
                              4, "Secrets not deleted")
@@ -1177,10 +1183,11 @@ def test_node_readiness_label(self):
 
     @timeout_decorator.timeout(TEST_TIMEOUT_SEC)
     def test_overwrite_pooler_deployment(self):
+        pooler_name = 'acid-minimal-cluster-pooler'
         k8s = self.k8s
         k8s.create_with_kubectl("manifests/minimal-fake-pooler-deployment.yaml")
         self.eventuallyEqual(lambda: k8s.get_operator_state(), {"0": "idle"}, "Operator does not get in sync")
-        self.eventuallyEqual(lambda: k8s.get_deployment_replica_count(name="acid-minimal-cluster-pooler"), 1,
+        self.eventuallyEqual(lambda: k8s.get_deployment_replica_count(name=pooler_name), 1,
                              "Initial broken deployment not rolled out")
 
         k8s.api.custom_objects_api.patch_namespaced_custom_object(
@@ -1193,7 +1200,7 @@ def test_overwrite_pooler_deployment(self):
         })
 
         self.eventuallyEqual(lambda: k8s.get_operator_state(), {"0": "idle"}, "Operator does not get in sync")
-        self.eventuallyEqual(lambda: k8s.get_deployment_replica_count(name="acid-minimal-cluster-pooler"), 2,
+        self.eventuallyEqual(lambda: k8s.get_deployment_replica_count(name=pooler_name), 2,
                              "Operator did not succeed in overwriting labels")
 
         k8s.api.custom_objects_api.patch_namespaced_custom_object(
@@ -1206,7 +1213,7 @@ def test_overwrite_pooler_deployment(self):
         })
 
         self.eventuallyEqual(lambda: k8s.get_operator_state(), {"0": "idle"}, "Operator does not get in sync")
-        self.eventuallyEqual(lambda: k8s.count_running_pods("connection-pooler=acid-minimal-cluster-pooler"),
+        self.eventuallyEqual(lambda: k8s.count_running_pods("connection-pooler="+pooler_name),
                              0, "Pooler pods not scaled down")
 
     @timeout_decorator.timeout(TEST_TIMEOUT_SEC)

manifests/configmap.yaml

@@ -44,13 +44,15 @@ data:
   # enable_init_containers: "true"
   # enable_lazy_spilo_upgrade: "false"
   enable_master_load_balancer: "false"
+  enable_master_pooler_load_balancer: "false"
   enable_password_rotation: "false"
   enable_pgversion_env_var: "true"
   # enable_pod_antiaffinity: "false"
   # enable_pod_disruption_budget: "true"
   # enable_postgres_team_crd: "false"
   # enable_postgres_team_crd_superusers: "false"
   enable_replica_load_balancer: "false"
+  enable_replica_pooler_load_balancer: "false"
   # enable_shm_volume: "true"
   # enable_sidecars: "true"
   enable_spilo_wal_path_compat: "true"

manifests/operatorconfiguration.crd.yaml

@@ -378,9 +378,15 @@ spec:
                   enable_master_load_balancer:
                     type: boolean
                     default: true
+                  enable_master_pooler_load_balancer:
+                    type: boolean
+                    default: false
                   enable_replica_load_balancer:
                     type: boolean
                     default: false
+                  enable_replica_pooler_load_balancer:
+                    type: boolean
+                    default: false
                   external_traffic_policy:
                     type: string
                     enum:

manifests/postgresql-operator-default-configuration.yaml

@@ -119,7 +119,9 @@ configuration:
     #   keyy: valuey
     # db_hosted_zone: ""
     enable_master_load_balancer: false
+    enable_master_pooler_load_balancer: false
     enable_replica_load_balancer: false
+    enable_replica_pooler_load_balancer: false
     external_traffic_policy: "Cluster"
     master_dns_name_format: "{cluster}.{team}.{hostedzone}"
     replica_dns_name_format: "{cluster}-repl.{team}.{hostedzone}"