Set cluster domain inside a tenant cluster

[rossbachp]

We want to generate a tenant cluster with clusterAPI and Kubevirt that can be used there own cluster domain.

We set the serviceDomain at the cluster definition:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: "${CLUSTER_NAME}"
  namespace: "${NAMESPACE}"
  labels:
    cluster.x-k8s.io/cluster-name: "${CLUSTER_NAME}"
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 10.243.0.0/16
    serviceDomain: ${CLUSTER_NAME}.local  
    services:
      cidrBlocks:
      - 10.96.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
    kind: KamajiControlPlane
    name: ${CLUSTER_NAME}
    namespace: ${NAMESPACE}
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: KubevirtCluster
    name: ${CLUSTER_NAME}
    namespace: ${NAMESPACE}

Setup the ControlPlane with this config:

apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  name: ${CLUSTER_NAME}
  namespace: ${NAMESPACE}
spec:
  dataStoreName: ${KAMAJI_DATASTORE}
  addons:
    coreDNS:
      dnsServiceIPs:
      - 10.96.0.10
  kubelet:
    cgroupfs: systemd
    preferredAddressTypes:
    - InternalIP
    - ExternalIP
  network:
    serviceType: LoadBalancer
    serviceAnnotations:
      lbipam.cilium.io/ips: "192.168.108.8"
  deployment: {}
  replicas: ${CONTROL_PLANE_MACHINE_COUNT}
  version: "${KUBERNETES_VERSION}"

Kamaji Version that we used:

helm search repo kamaji
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                       
clastix/kamaji          1.0.0           v1.0.0          Kamaji is the Hosted Control Plane Manager for ...
clastix/kamaji-console  0.0.5           v0.0.2          Kamaji deploys and operates Kubernetes at scale...
clastix/kamaji-etcd     0.7.0           3.5.6           Helm chart for deploying a multi-tenant `etcd` ...
clastix/console         0.0.1           0.0.1           Kamaji is a tool aimed to build and operate Kub...

Problem is that the resulting cluster use the default "cluster.local" domain

kubectl proxy &
k get nodes
curl -X GET http://127.0.0.1:8001/api/v1/nodes/tenant1-md-0-j8r46-qp5kc/proxy/configz | jq .

    "clusterDomain": "cluster.local",
    "clusterDNS": [
      "10.96.0.10"
    ],

Also included at the coreDNS core file:

kubectl get cm coredns -n kube-system -o jsonpath="{.data.Corefile}" \
  | grep ".local " \
  | awk -F ' ' '{print $2}'
cluster.local
k get cm -n kube-system coredns -o yaml
## output
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2024-09-13T08:31:44Z"
  name: coredns
  namespace: kube-system
  ownerReferences:
  - apiVersion: rbac.authorization.k8s.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: ClusterRoleBinding
    name: system:coredns
    uid: 3341239f-17aa-4a6c-8c53-3daf4fcc4e94
  resourceVersion: "3615354"
  uid: a519dba3-fd61-4b99-8ad8-aebc5f35eb8a

Is my configuration wrong?

[prometherion]

This is a feature request since we do not currently support this customisation.

[rossbachp]

OK, but if you want to setup a cilium multi cluster setup, this feature is needed to separate the services to multiple clusters. Hope you can add this feature at some of the next releases:)

[prometherion]

We'll work on this, even though we have different feature requests in pipelines so we can't guarantee when this lands.

[rossbachp]

I hope you can prioritize supporting this feature, as it is essential for two key scenarios. First, it allows the migration of workloads to newer major Kubernetes cluster releases, which is particularly important when some applications encounter compatibility issues with the latest versions. Second, it facilitates the connection of multi-rack or multi-region clusters using service meshes, enhancing the scalability and resilience of your infrastructure.

[prometherion]

Implemented with #594, still need to back ported to Cluster API.

[prometherion]

Implemented with the v0.12.0 Kamaji Cluster API Control Plane provider: https://github.com/clastix/cluster-api-control-plane-provider-kamaji/releases/tag/v0.12.0