Multipass docker containers dns resolution not works on macos

[zopar]

Describe the bug
Containers dns resolution does not works

To Reproduce
How, and what happened?
multipass lauch docker

docker run busybox nslookup google.com
nslookup: read: connection refused

Expected behavior
container dns works correctly

Additional info
OS: macOS 13.4.1
multipass ubuntu version 22.04 LTS
multipass version 1.12.0+mac with qemu

Additional context
If I use --dns option in the docker run passing dns like 8.8.8.8, resolution works correctly
If I use --network host option also works
It seems a problem about bridged network at this point

[townsend2010]

Hi @zopar!

I cannot reproduce this issue on my M1 Macbook.

% docker run busybox nslookup google.com
Unable to find image 'busybox:latest' locally
latest: Pulling from library/busybox
ba1cab11642e: Pull complete 
Digest: sha256:2376a0c12759aa1214ba83e771ff252c7b1663216b192fbe5e0fb364e952f85c
Status: Downloaded newer image for busybox:latest
Server:		192.168.64.1
Address:	192.168.64.1:53

Non-authoritative answer:
Name:	google.com
Address: 2607:f8b0:4002:80a::200e

Non-authoritative answer:
Name:	google.com
Address: 64.233.176.101
Name:	google.com
Address: 64.233.176.138
Name:	google.com
Address: 64.233.176.100
Name:	google.com
Address: 64.233.176.139
Name:	google.com
Address: 64.233.176.102
Name:	google.com
Address: 64.233.176.113

Do you happen to be running the Apple Firewall or another firewall? If so, then you probably need to add an exception to allow the traffic.

[zopar]

Ok found the problem is due to:
dnscrypt and mdnsresponder

Just some explanation on why the ubuntu docker seems to work
On mac dnscrypt on localhost is already using port 53 on ipv4 so multipass only started his mdnsresponder on ipv6 port 53 via Internet Sharing.
The dhcp for the bridge network is 192.168.64.1 also used for dns, it has also an ipv6 ip, someone could think that
the ubuntu docker used ipv6 to resolve but the system does not have any route to the ipv6 ip, this type of route are setup only if you configure docker daemon to use ipv6 that is an experimental feature.
I verified that with a dig +trace indeed the network results unreachable for the ipv6.
I tried also to start a dig container and point to ipv4 and ipv6 via dig and nothing works obviously.
In reality it is systemd-resolved that bypass the problem.
For the containers it not works because the nameserver inside the containers are simply a 192.168.64.1 and obviously is possible to verify with a dig container that point to this ip and related ipv6 that the connection is refused.

How to bypass the problem:

Solution 1
Simple way is to create a docker network, in this case everything works also for containers because docker fall back on google dns as design. But that means that you do not use secured dnscrypt for docker.

Other solutions
This thread is also interesting for possible solution
kubernetes/minikube#3036

Solution 2
Using socat

sudo socat UDP4-RECVFROM:53,bind=192.168.64.1,fork UDP4-SENDTO:127.0.0.1:53

Solution 3
Using pfctl

rdr on bridge100 inet proto {tcp,udp} from any to bridge100 port 53 -> 127.0.0.1 port 53

Solution 4
It should be possible also to set multiple ip in dnscrypt if you can modify the configuration with something like --local-address=127.0.0.1:53,192.168.64.1
Here also info about dnscrypt and mdnsresponder
https://github.com/DNSCrypt/dnscrypt-proxy/wiki/Installation-macOS#error-12700153-bind-address-already-in-use

Proposal
@townsend2010 do you think is possible to implement something like this in multipass ?

kubernetes/minikube#3036 (comment)
This also will help with vpn problems
An user already created also a workaround for inject it https://github.com/amine7536/multipass-vpnkit
It seems related to this #1614

Anyway I think this issue can be closed.

[townsend2010]

Hi @zopar!

Thank for posting the detailed solutions for the issue you were running in to.

Regarding vpnkit, we did look into that in the past, but it's pretty specific to Hyperkit and that is now deprecated and will be removed in the next release. I do know the Docker folks are using vpnkit with Qemu, but I'm not sure how they are incorporating it since the sources are not available. That said, considering all of the issues we encounter with respect to macOS's virtualized network support, we may still consider it in the future.

I'll close this issue now and thanks again!