Multi-node DNS/networking issue

[PaoloneM]

After starting multi-node cluster, DNS seems not to work.

Steps to reproduce the issue:

On macOS Catalina 10.15.5, Docker version 19.03.8, build afacb8b

1. minikube start --nodes 3
2. kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
3. kubectl exec -i -t dnsutils -- nslookup kubernetes.default

Full output of failed command:

;; connection timed out; no servers could be reached
command terminated with exit code 1

Full output of minikube start command used, if not already included:

minikube start --nodes 3 --alsologtostderr output

Optional: Full output of minikube logs command:

Logs output

Details
Inspecting the cluster seems that the DNS pod and the pod I started to troubleshoot DNS' issue has the same IP address:

╰─➤  kubectl get po --all-namespaces -o wide
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE     IP           NODE           NOMINATED NODE   READINESS GATES
default       dnsutils                           1/1     Running   0          72s     172.18.0.2   minikube-m02   <none>           <none>
kube-system   coredns-66bff467f8-f2r5w           1/1     Running   0          5m59s   172.18.0.2   minikube       <none>           <none>
kube-system   etcd-minikube                      1/1     Running   0          6m4s    172.17.0.3   minikube       <none>           <none>
kube-system   kindnet-2nqvk                      1/1     Running   0          5m14s   172.17.0.3   minikube       <none>           <none>
kube-system   kindnet-79bwt                      1/1     Running   0          4m37s   172.17.0.5   minikube-m03   <none>           <none>
kube-system   kindnet-fw9c7                      1/1     Running   0          5m14s   172.17.0.4   minikube-m02   <none>           <none>
kube-system   kube-apiserver-minikube            1/1     Running   0          6m4s    172.17.0.3   minikube       <none>           <none>
kube-system   kube-controller-manager-minikube   1/1     Running   0          6m4s    172.17.0.3   minikube       <none>           <none>
kube-system   kube-proxy-6lfcr                   1/1     Running   0          5m15s   172.17.0.4   minikube-m02   <none>           <none>
kube-system   kube-proxy-99q2r                   1/1     Running   0          4m37s   172.17.0.5   minikube-m03   <none>           <none>
kube-system   kube-proxy-szvc4                   1/1     Running   0          5m59s   172.17.0.3   minikube       <none>           <none>
kube-system   kube-scheduler-minikube            1/1     Running   0          6m4s    172.17.0.3   minikube       <none>           <none>
kube-system   storage-provisioner                1/1     Running   0          6m3s    172.17.0.3   minikube       <none>           <none> 

Might this be a network plugin setup issue?
Same happens with other providers, such as adding --driver=parallels to cluster start command.

[priyawadhwa]

Hey @PaoloneM thanks for reporting this issue. I was able to reproduce it with:

• minikube v1.12.0
• docker driver
• mac OS

multinode is currently experimental, so it looks like this is one of the bugs that still need to be fixed. To track progress on multi-node clusters, see #7538.

It seems like this bug is a dupe of #8055, which was closed as the issue was identified to be CNI related. @tstromberg is currently working on CNI in minikube, I think our next release should have a fix.

cc @sharifelgamal

[PaoloneM]

Hi @priyawadhwa , thanks for investigating the issue. I think too that it is more related to CNI: as a workaround I've been able to get the cluster working by removing the pre-installed net plugin, install Calico from scratch, recreating the DNS and any other pod I deployed to have them getting consistent networking config.

[priyawadhwa]

@PaoloneM glad you found a way to make this work!

[priyawadhwa]

Hey @PaoloneM as you are no longer seeing this issue, and I think CNI has recently been fixed, I'm going to go ahead and close this issue. Feel free to reopen at any time if you continue to have problems.