Cisco AnyConnect VPN with DNS

[dee-kryvenko]

AnyConnect will setup utunN interface as well as DNS resolvers I could see with scutil --dns.

This doesn't seem to be propagated into VMs. Is there a way to make it work, do I miss some configuration or it's just not supported?

I also tried to put some known DNS servers to my ~/.lima/default/lima.yaml:

dns:
- x.x.x.x
- y.y.y.y

And then limactl stop && limactl start - but my VM still doesn't have it in the /etc/resolv.conf neither am I able to use curl to get from a private host meaning it is not being injected to the VM in any other way. What am I missing?

[jandubois]

This doesn't seem to be propagated into VMs. Is there a way to make it work, do I miss some configuration or it's just not supported?

You have to do this manually via the dns settings in lima.yaml. The only name servers that should be propagated automatically are the ones from the host en0 interface.

my VM still doesn't have it in the /etc/resolv.conf

On a systemd setup they will not be added to /etc/resolv.conf, but should still be configured for the resolved service. You can check with resolvectl status; it should show additional servers beyond 192.168.19.53, e.g.

$ resolvectl status
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.19.53
       DNS Servers: 192.168.19.53 8.8.8.8 1.1.1.1

neither am I able to use curl to get from a private host

I suspect the issue is that the VM cannot route to the additional name servers because they too are on the VPN. I don't have an idea right now how to make that work, unfortunately.

[dee-kryvenko]

I figured the latter question out - modifying list of DNS servers on the existing VM is not having any effect (it only works for VM creation), I had to modify /etc/netplan/50-cloud-init.yaml and now it picked it up after a restart. My DNS server are indeed private but somehow VM can perfectly access it as well as other private hosts behind VPN. This seems to be a DNS only issue.

However, that doesn't solve it entirely. Being able to provide DNS servers is not enough - AnyConnect as well as other VPN clients are using conditional resolvers. There are different DNS servers for different search domains. For example my scutil --dns looking the following way (stripped out private details from the output):

resolver #1
  search domain[0] : enterprise.com
  ...
  search domain[19] : enterprise19.com
  nameserver[0] : 8.8.8.8
  nameserver[1] : 8.8.4.4
  if_index : 6 (en0)
  flags    : Request A records
  reach    : 0x00000002 (Reachable)

resolver #2
  domain   : enterprise2.tech
  nameserver[0] : x.x.x.x
  nameserver[1] : y.y.y.y
  flags    : Supplemental, Request A records
  reach    : 0x00000002 (Reachable)
  order    : 1

resolver #17
  domain   : local
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300000

resolver #23
  domain   : 254.169.in-addr.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300200

resolver #24
  domain   : 8.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300400

resolver #25
  domain   : 9.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300600

resolver #26
  domain   : a.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 300800

resolver #27
  domain   : b.e.f.ip6.arpa
  options  : mdns
  timeout  : 5
  flags    : Request A records
  reach    : 0x00000000 (Not Reachable)
  order    : 301000

DNS configuration (for scoped queries)

resolver #1
  nameserver[0] : 8.8.8.8
  nameserver[1] : 8.8.4.4
  if_index : 6 (en0)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)

resolver #2
  search domain[0] : enterprise.tech
  ...
  search domain[19] : enterprise19.tech
  nameserver[0] : x.x.x.x
  nameserver[1] : y.y.y.y
  if_index : 17 (utun2)
  flags    : Scoped, Request A records
  reach    : 0x00000002 (Reachable)
  order    : 1

These x.x.x.x and y.y.y.y DNS servers might not know about anything outside the VPN so I can't just use them as catch-all DNS. Adding both public and private DNS servers to /etc/netplan/50-cloud-init.yaml doesn't seem to work, probably because they are expected to be HA replicas instead of actually different authorities that could return different results.

[dee-kryvenko]

Upon further research - split DNS as well as systemd-resolved is not supported by underlying coredns anyway. I've found out that lima.yaml (and /etc/netplan/50-cloud-init.yaml) DNS ends up in the /run/systemd/resolve/resolv.conf so for K3s one could use --resolv-conf /run/systemd/resolve/resolv.conf to make it work with the corporate VPN.

It would be nice to fix Lima to refresh DNS from lima.yaml on restart so the user don't have to know about /etc/netplan/50-cloud-init.yaml but otherwise this ticket can be closed.

[jandubois]

It would be nice to fix Lima to refresh DNS from lima.yaml on restart so the user don't have to know about /etc/netplan/50-cloud-init.yaml but otherwise this ticket can be closed.

I was just about to say that Lima does refresh the DNS on each restart, but when I just tested it, the new servers were not configured.

Unfortunately this seems to be an issue with cloud-init, as we do pass in the new nameservers:

jan@lima-default:/Users/jan$ sudo cat /mnt/lima-cidata/network-config
version: 2
ethernets:
  eth0:
    match:
      macaddress: '52:55:55:e9:a1:b4'
    dhcp4: true
    set-name: eth0
    nameservers:
      addresses:
      - 1.1.1.1
      - 1.0.0.1

This seems to be processed only during first-boot.

[jandubois]

This seems to be processed only during first-boot.

Yes, this is intentional, as far as cloud-init is concerned, but I found a way to work around it, by changing the instance id on every reboot, so it always looks like a fresh install to cloud-init: #270

[jandubois]

Upon further research - split DNS as well as systemd-resolved is not supported by underlying coredns anyway. I've found out that lima.yaml (and /etc/netplan/50-cloud-init.yaml) DNS ends up in the /run/systemd/resolve/resolv.conf so for K3s one could use --resolv-conf /run/systemd/resolve/resolv.conf to make it work with the corporate VPN.

This is the part I still don't understand:

If coredns doesn't understand the split setup where each name server only knows about a subset of domains, how does it help to configure all with systemd. coredns would still forward to a randomly picked server from /run/systemd/resolve/resolv.conf, and you can either get lucky, or you could not.

I think leaving 192.168.5.3 in the list of name servers is a mistake because in case you explicitly supply VPN name servers, the qemu nameserver will not be able to resolve the name, so you have a 33% chance of failure.

But removing this address will still not make the split-dns scenario work. I'm actually not sure if it is even possible to make it work. This is turning into a rabbit hole...

I'm re-opening this issue, because I think we need to explore this further.

[jandubois]

I'm actually not sure if it is even possible to make it work.

On further thoughts, I think it should be possible: we could specify the search domains together with the nameserver addresses, and systemd-resolved should route queries appropriately. And k8s would use 127.0.0.53 from /etc/resolv.conf.

For systems without systemd-resolved we could run dnsmasq and configure it for conditional forwarding.

This is now getting quite a bit more complex, so it may take a while before I can look into it.