Local DNS resolution doeesn't work on macOS with MagicDNS enabled

[felixscheinost]

Bug description

When connected to a Headscale server where MagicDNS is enabled, local name resolution doesn't work anymore on macOS.

For example without Tailscale enabled my machine receives DNS configuration via DHCP from my home router.
The home router runs its own DNS server and resolves <router_name> to its own IP address.
Once connected to the Headscale Tailnet, <router_name> can no longer be resolved.

With the official Tailscale coordination server on the other hand there is an explicit option whether local DNS resolution should be overriden or not. If the toggle to override local DNS resolution is turned off, the router name can still be resolved.

So this seems like a bug in how Headscale configures the Tailscale client.

Context info

• Version of headscale 0.15.0
• Version of tailscale 1.16.1
• OS macOS 12.4
• config.yaml

dns_config:
  base_domain: 'example.com'
  domains: []
  magic_dns: true
  nameservers:
  - 1.1.1.1

Output of scutil --dns

The output of scutil --dns looks different when connected to Tailscale vs Headscale.

Tailscale

DNS configuration

resolver #1 
  search domain[0] : internal.example.com (note: probably because I configured scutil --set HostName to be <machine_name>.internal.example.com)
  search domain[1] : <my-email-address>.beta.tailscale.net
  search domain[2] : ts.net
  search domain[3] : <router DHCP domain_name>
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 13 (en7)
  flags    : Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2 - #65
  domain   : <64-127>.100.in-addr.arpa.
  nameserver[0] : 100.100.100.100
  if_index : 28 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103002 - 103065

resolver #66
  domain   : 0.e.1.a.c.5.1.1.a.7.d.f.ip6.arpa.
  nameserver[0] : 100.100.100.100
  if_index : 28 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103001

resolver #67
  domain   : <my email address>.beta.tailscale.net.
  nameserver[0] : 100.100.100.100
  if_index : 28 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103000

resolver #68
  domain   : ts.net.
  nameserver[0] : 100.100.100.100
  if_index : 28 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103066

resolver #69
  domain   : internal.example.com (note: this resolver comes from corporate IPSec VPN that I am connected to as well. this corporate VPN does split-tunneling. it takes over DNS for this subdomain)
  nameserver[0] : <corporate DNS 1>
  nameserver[1] : <corporate DNS 2>
  nameserver[2] : <corporate DNS 3>
  if_index : 27 (utun4)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 101200

resolver #70
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #71
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #72
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #73
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #74
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #75
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000


DNS configuration (for scoped queries)

resolver #1
  search domain[0] : <router DHCP domain_name>
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 13 (en7)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  search domain[0] : <router DHCP domain_name>
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 15 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #3
  search domain[0] : internal.example.com
  nameserver[0] : <corporate DNS 1>
  nameserver[1] : <corporate DNS 2>
  nameserver[2] : <corporate DNS 3>
  if_index : 27 (utun4)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

resolver #4
  search domain[0] : <my_email_address>.beta.tailscale.net
  search domain[1] : 0.e.1.a.c.5.1.1.a.7.d.f.ip6.arpa
  search domain[2] : 100.100.in-addr.arpa
  search domain[3] : 101.100.in-addr.arpa
  search domain[4] : 102.100.in-addr.arpa
  search domain[5] : 103.100.in-addr.arpa
  search domain[6] : 104.100.in-addr.arpa
  search domain[7] : 105.100.in-addr.arpa
  search domain[8] : 106.100.in-addr.arpa
  search domain[9] : 107.100.in-addr.arpa
  search domain[10] : 108.100.in-addr.arpa
  search domain[11] : 109.100.in-addr.arpa
  search domain[12] : 110.100.in-addr.arpa
  search domain[13] : 111.100.in-addr.arpa
  search domain[14] : 112.100.in-addr.arpa
  search domain[15] : 113.100.in-addr.arpa
  search domain[16] : 114.100.in-addr.arpa
  search domain[17] : 115.100.in-addr.arpa
  search domain[18] : 116.100.in-addr.arpa
  search domain[19] : 117.100.in-addr.arpa
  search domain[20] : 118.100.in-addr.arpa
  search domain[21] : 119.100.in-addr.arpa
  search domain[22] : 120.100.in-addr.arpa
  search domain[23] : 121.100.in-addr.arpa
  search domain[24] : 122.100.in-addr.arpa
  search domain[25] : 123.100.in-addr.arpa
  search domain[26] : 124.100.in-addr.arpa
  search domain[27] : 125.100.in-addr.arpa
  search domain[28] : 126.100.in-addr.arpa
  search domain[29] : 127.100.in-addr.arpa
  search domain[30] : 64.100.in-addr.arpa
  search domain[31] : 65.100.in-addr.arpa
  search domain[32] : 66.100.in-addr.arpa
  search domain[33] : 67.100.in-addr.arpa
  search domain[34] : 68.100.in-addr.arpa
  search domain[35] : 69.100.in-addr.arpa
  search domain[36] : 70.100.in-addr.arpa
  search domain[37] : 71.100.in-addr.arpa
  search domain[38] : 72.100.in-addr.arpa
  search domain[39] : 73.100.in-addr.arpa
  search domain[40] : 74.100.in-addr.arpa
  search domain[41] : 75.100.in-addr.arpa
  search domain[42] : 76.100.in-addr.arpa
  search domain[43] : 77.100.in-addr.arpa
  search domain[44] : 78.100.in-addr.arpa
  search domain[45] : 79.100.in-addr.arpa
  search domain[46] : 80.100.in-addr.arpa
  search domain[47] : 81.100.in-addr.arpa
  search domain[48] : 82.100.in-addr.arpa
  search domain[49] : 83.100.in-addr.arpa
  search domain[50] : 84.100.in-addr.arpa
  search domain[51] : 85.100.in-addr.arpa
  search domain[52] : 86.100.in-addr.arpa
  search domain[53] : 87.100.in-addr.arpa
  search domain[54] : 88.100.in-addr.arpa
  search domain[55] : 89.100.in-addr.arpa
  search domain[56] : 90.100.in-addr.arpa
  search domain[57] : 91.100.in-addr.arpa
  search domain[58] : 92.100.in-addr.arpa
  search domain[59] : 93.100.in-addr.arpa
  search domain[60] : 94.100.in-addr.arpa
  search domain[61] : 95.100.in-addr.arpa
  search domain[62] : 96.100.in-addr.arpa
  search domain[63] : 97.100.in-addr.arpa
  search domain[64] : 98.100.in-addr.arpa
  search domain[65] : 99.100.in-addr.arpa
  search domain[66] : ts.net
  nameserver[0] : 100.100.100.100
  if_index : 28 (utun3)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)

Headscale

DNS configuration

resolver #1
  search domain[0] : internal.example.com
  search domain[1] : <headscale namespace>.example.com
  search domain[2] : <router DHCP domain_name>
  nameserver[0] : 100.100.100.100
  if_index : 27 (utun4)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103400

resolver #2
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 13 (en7)
  flags    : Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)
  order    : 200000

resolver #3
  domain   : <headscale namespace>.example.com.
  nameserver[0] : 100.100.100.100
  if_index : 27 (utun4)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 103401

resolver #4
  domain   : internal.example.com
  nameserver[0] : <corporate DNS 1>
  nameserver[1] : <corporate DNS 2>
  nameserver[2] : <corporate DNS 3>
  if_index : 28 (utun3)
  flags    : Supplemental, Request A records, Request AAAA records
  reach    : 0x00000003 (Reachable,Transient Connection)
  order    : 101000

resolver #5
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #6
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #7
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #8
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #9
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #10
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records, Request AAAA records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  search domain[0] : <router DHCP domain_name>
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 13 (en7)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #2
  search domain[0] : <router DHCP domain_name>
  nameserver[0] : <router ipv4>
  nameserver[1] : <router ipv6>
  if_index : 15 (en0)
  flags    : Scoped, Request A records, Request AAAA records
  reach    : 0x00020002 (Reachable,Directly Reachable Address)

resolver #3
  search domain[0] : internal.example.com
  nameserver[0] : <corporate DNS 1>
  nameserver[1] : <corporate DNS 2>
  nameserver[2] : <corporate DNS 3>
  if_index : 28 (utun3)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

resolver #4
  search domain[0] : <headscale namespace>.example.com
  nameserver[0] : 100.100.100.100
  if_index : 27 (utun4)
  flags    : Scoped, Request A records
  reach    : 0x00000003 (Reachable,Transient Connection)

=> So it seems with Tailscale my resolver number 1 stays my router, even for the Tailscale namespaces, while with Headscale resolver number 1 is 100.100.100.100

[huskyii]

I encountered some dns issue in Linux and OpenBSD machine, have not dive into these issues, but I think it may be same as yours.

[huskyii]

After read some code, I think DefaultResolvers should be set to nil and a Route map item { '.' : '$resolver_set_in_config'} shoule be added into Routes.

I do not have a official tailscale account, maybe someone with official tailscale account could look into the log see how dns config looks like(keyword: dns: Set:)

If anyone confirm my guess is correct, I'm happy to implement the fix :)

[felixscheinost]

Oh cool, that you have a hunch so quickly! @huskyii

I just checked on a different Linux machine, that should work as well, right?

In there I have the following line in the log:

tailscaled[32149]: router: dns: Set: {Nameservers:[100.100.100.100] Domains:[<email>.beta.tailscale.net] PerDomain:false Proxied:true

Is that what you are looking for?

[huskyii]

Oh cool, that you have a hunch so quickly! @huskyii

I just checked on a different Linux machine, that should work as well, right?

In there I have the following line in the log:

tailscaled[32149]: router: dns: Set: {Nameservers:[100.100.100.100] Domains:[<email>.beta.tailscale.net] PerDomain:false Proxied:true

Is that what you are looking for?

Hi @felixscheinost which version of tailscale r u using?
The log is different from mine. I'm using tailscale 1.26.1

Here's my log, note: no router: before dns: Set:

tailscaled[51443]: dns: Set: {DefaultResolvers:[] Routes:{} SearchDomains:[] Hosts:0}

[vquicksilver]

Isn't this related to #280 ? My understanding is that an option for deciding if you want to override the DNS servers, or just use "split-dns" as in adding an extra DNS server to your current network configuration is missing on the headscale side vs tailscale.

In some cases you might want to force the DNS servers in the configuration to be the only ones (i.e if you are using another node as an exit node and the node has a DNS server configured and you don't want to leak DNS requests to the outside world), and in other cases you only want to add an extra DNS server to your configuration so that you can resolve the internal tailscale/headscale names.

I am not sure of which mode it is the default right now, for example in one of my Ubuntu systems I see:

# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

nameserver 100.100.100.100
nameserver 10.83.0.1
nameserver fd0c:989d:1024::1
search home.i.my-private.domain local

This system was joined with the following command:

tailscale up --exit-node=100.64.0.1 --exit-node-allow-lan-access --login-server=https://vpn.my-private.domain

I guess the parameters you use to join the overlay network do also effect the resulting DNS servers in your system, as someone new to tailscale/headscale it would be great to get more details on how this is supposed to work in the documentation.

I also have configured 100.64.0.1 as my default DNS server in the headscale config with:

dns_config:
  # List of DNS servers to expose to clients.
  nameservers:
    - 100.64.0.1

And I am running my own DNS server on that system. systemd-resolve --status is showing 100.100.100.100 as the default DNS server for ~* , but the queries are still being forwarded to 100.64.0.1 (I see them in the log), and as you see it is still keeping 10.83.0.1 as my local dns server for ".local" for my lan.

[felixscheinost]

@huskyii I think I was using 1.24.2 at the time.

@vquicksilver Yeah, its probably a duplicate of that issue.

I tried working around the issuee right now by using the Go tailscaled and not the version from the macOS AppStore. But then I need to configure 100.100.100.100 manually. I couldn't find a way to do this without also overriding the DHCP DNS servers.

[felixscheinost]

Okay, I just ran tailscaled manually on Linux against official Tailscale server.

I turned override local DNS on, then off again and watched the log:

# Turn override local DNS ON
wgengine: Reconfig: configuring router
wgengine: Reconfig: configuring DNS
dns: Set: {
  # Difference, empty when override is OFF 
  DefaultResolvers:[1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001] 
  # Same
  Routes:{beta.tailscale.net.:[] felix-scheinost.googlemail.com.beta.tailscale.net.:[] ts.net.:[]}+65arpa 
  # Same
  SearchDomains:[felix-scheinost.googlemail.com.beta.tailscale.net.] 
  # Same
  Hosts:7
}
dns: Resolvercfg: {
  # Difference, when override local DNS is OFF these contains the DHCP resolvers
  Routes:{.:[1.1.1.1 1.0.0.1 2606:4700:4700::1111 2606:4700:4700::1001]} 
  # Same
  Hosts:7 
  # Same
  LocalDomains:[felix-scheinost.googlemail.com.beta.tailscale.net. beta.tailscale.net. ts.net.]+65arpa
}
dns: OScfg: {
  # Same
  Nameservers:[100.100.100.100] 
  # Same
  SearchDomains:[felix-scheinost.googlemail.com.beta.tailscale.net.] 
  # Same
  MatchDomains:[]
}


# Turn override local DNS OFF
wgengine: Reconfig: configuring router
wgengine: Reconfig: configuring DNS
dns: Set: {
  DefaultResolvers:[] 
  Routes:{beta.tailscale.net.:[] felix-scheinost.googlemail.com.beta.tailscale.net.:[] ts.net.:[]}+65arpa 
  SearchDomains:[felix-scheinost.googlemail.com.beta.tailscale.net.] 
  Hosts:7
}
dns: Resolvercfg: {
  Routes:{.:[192.168.178.1 fd00::de39:6fff:fe31:7433]} 
  Hosts:7 
  LocalDomains:[felix-scheinost.googlemail.com.beta.tailscale.net. beta.tailscale.net. ts.net.]+65arpa
}
dns: OScfg: {
  Nameservers:[100.100.100.100]
  SearchDomains:[felix-scheinost.googlemail.com.beta.tailscale.net. fritz.box.] 
  MatchDomains:[]
}

[huskyii]

I don't think headscale has a override_local_DNS option now, maybe we should add one, what confused me is that offical Tailscale KB for maigic DNS stated that

Your network must have at least one DNS nameserver enabled in the admin console. These nameservers will receive all DNS queries not handled by MagicDNS. This restriction will be relaxed in the future.

I think it's basically says that to use magic DNS, you need override local DNS.

@felixscheinost does magic DNS work if you turn off override local DNS ?

And could you please also take a look at /etc/resolv.conf and /etc/resolv.pre-tailscale-backup.conf

[mlincett]

Your network must have at least one DNS nameserver enabled in the admin console. These nameservers will receive all DNS queries not handled by MagicDNS. This restriction will be relaxed in the future.

I think it's basically says that to use magic DNS, you need override local DNS.

This is incorrect, at least when using resolved.

MagicDNS sets in-addr.arpa records for all the IP address space used by tailscale.

Override local DNS enabled sets +DefaultRoute for the tailscale interface (meaning this should be the default DNS unless a more specific is found) and the global DNS domain ~.

The two settings are independent - however I have no idea how this works in systems pre-resolved.

[kradalby]

@mlincett @huskyii @felixscheinost

I think I have added support for this correctly in #905, can you please help me test it?