Make Gateway Discovery work with GKE and kube-dns

[mlavacca]

Problem Statement

https://github.com/Kong/gateway-operator/pull/1261 enabled Gateway Discovery in ControlPlanes with Service DNSStrategy, setting the admin services spec.publishNotReadyAddresses field to true. This means that the admin service endpoints are created before the ready probes of the pod are successful. This implementation allows us to solve the chicken-egg problem (caused by the enablement of the dataplane /status/ready endpoint) that follows:

• the dataplane needs a configuration to get ready
• the controlplane needs a dataplane pod connection to get ready

This solution has been tested and has some problems in GKE

• when kube-dns is used, this solution does not work, because kube-dns needs the pod to be ready to resolve the service-based pod-ip-address.service-name.namespace.svc address
• when cloud-dns is enabled this solution works

Proposed Solution

To make KGO work with GKE and kube-dns, we may want to do the follows:

• introduce a new flag --controlplane-dns-strategy with default value "service" and allowed values {"service", "pod"}
• the controlplane DNS strategy is configured according to the flag's value
• in case the DNS strategy is pod, we need to create a looser certificate with the *.namespace.pod subject instead of *.service-name.namespace.svc
• in case a certificate already existed, we need to replace the old certificate with the new one and restart the dataplane pods to properly mount the new certificate.

Acceptance Criteria

• ControlPlane Gateway Discovery properly works in GKE with kube-dns

[aoktox]

I'm facing same issue with Kong/charts#1068, this workaround is working for me

kind: GatewayConfiguration
apiVersion: gateway-operator.konghq.com/v1beta1
metadata:
  name: kong
  namespace: kong-system
spec:
  <truncated>
  controlPlaneOptions:
    deployment:
      podTemplateSpec:
        spec:
          containers:
            - name: controller
              image: kong/kubernetes-ingress-controller:3.3.1
              env:
                - name: CONTROLLER_GATEWAY_DISCOVERY_DNS_STRATEGY
                  value: ip
                - name: CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY
                  value: "true"
<truncated>