[BUG] DNS not resolving

[luisdavim]

What did you do?

• How was the cluster created?

 k3d create -n mycluster

• What did you do afterwards?

Start a pod and try a DNS query:

$ export KUBECONFIG="$(k3d get-kubeconfig --name='mycluster')"
$ kubectl run --restart=Never --rm -i --tty tmp --image=alpine -- sh
If you don't see a command prompt, try pressing enter.
/ # nslookup www.gmail.com
Server:         10.43.0.10
Address:        10.43.0.10:53

;; connection timed out; no servers could be reached

/ # cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.43.0.10
options ndots:5
/ # exit

Exec into the k3d container and do the same DNS query:

docker exec -it k3d-endpoint-server sh
/ # nslookup www.gmail.com
Server:         127.0.0.11
Address:        127.0.0.11:53

Non-authoritative answer:
www.gmail.com   canonical name = mail.google.com
mail.google.com canonical name = googlemail.l.google.com
Name:   googlemail.l.google.com
Address: 172.217.164.101

Non-authoritative answer:
www.gmail.com   canonical name = mail.google.com
mail.google.com canonical name = googlemail.l.google.com
Name:   googlemail.l.google.com
Address: 2607:f8b0:4005:80b::2005

/ # cat /etc/resolv.conf
nameserver 127.0.0.11
options ndots:0
/ # exit

What did you expect to happen?
I would expect the pods in the k3d cluster to be able to resolve DNS names

Which OS & Architecture?
MacOS 10.15.3

Which version of k3d?

• output of k3d --version

k3d version v1.7.0

Which version of docker?

• output of docker version

docker version
Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:22:34 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:29:19 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[iwilltry42]

Hi there, thanks for opening this issue.
As mentioned in the related issue (#101 (comment)), CoreDNS is doing the name resolution inside the cluster.
I couldn't reproduce this on Linux (with the same docker and k3d versions), so I guess that the difference in DNS settings is caused by Docker for Desktop.

[luisdavim]

To workaround this issue I'm patching the coredns configmap.

[iwilltry42]

@luisdavim what's the patch that you apply?

[consideRatio]

@luisdavim I'm also interested in what patch you applied

[irizzant]

I'm pretty sure that adapting the forward . /etc/resolv.conf part is enough.

This let k8s use the machine DNS when the name cannot be resolved internally.

I think this should be the default behaviour.

[irizzant]

After more investigation I found that this could be related to the way k3d creates the docker network.

Indeed, k3d creates a custom docker network for each cluster and when this happens resolving is done through the docker daemon. The requests are actually forwarded to the DNS servers configured in your host's resolv.conf. But through a single DNS server (the embedded one of docker).

This means that if your daemon.json is, like mine, not configured to provide extra DNS servers it defaults to 8.8.8.8 which does not resolve any company address for example.

It would be useful to have a custom options to provide to k3d when it starts the cluster and specify the DNS servers there, as proposed here #165

[iwilltry42]

Thanks for your additional input @irizzant , how would you add additional DNS servers here on k3d's side?
The network opts of docker don't seem to have such an option (after having a quick glance over what's available there) 🤔

[irizzant]

Personally I fixed this by injecting a custom ConfigMap for CoreDns, by changing

forward . /etc/resolv.conf

to:

forward . /etc/resolv.conf xxx.xxx.xxx.xxx

replacing the x with the IP of your DNS servers.

I had a quick look to the docker options and I confirm that I don't see an option to configure the custom DNS servers on the docker network.

Maybe a feasible option would be to add a custom flag to k3d command which adds the custom DNS servers to the CoreDns ConfigMap directly.

[iwilltry42]

Currently, k3d doesn't interact with any Kubernetes resources inside the cluster (i.e. in k3s) and I tend to avoid this because of the huge dependencies on Kubernetes libraries it could draw in. Upon cluster creation this could work however by modifying the Chart that's being auto-deployed by k3s. Not sure if this could go into k3s itself instead 🤔

[irizzant]

Maybe interacting with k8s itself it's not needed.
k3s deploys whatever is under /var/lib/rancher/k3s/server/manifests, so you could add a valid CoreDns configuration just customizing the DNS part according to the command line flag.