K3s single node cluster + MetalLB (ARP issues)

[smbambling]

TL;DR: It appears that the latest macOS 13.5 upgrade has broken what I believe is the ability for the VMs to ARP up additional addresses such as the metalLB addresses.

---

I've modified the current k3s example template (listed below) to deploy a single node k3s cluster on my MacBook Pro ( Intel )

The biggest change here is the use of shared network which was configured on my host MacBook Pro using the following

# Install socket_vmnet
brew install socket_vmnet

# Configure Lima Networks
socket_vmnet_path=$(readlink -f $(brew --prefix)/opt/socket_vmnet/bin/socket_vmnet | \
sed 's_/_\\/_g') && sed -i -re "s/^(  socketVMNet:).*/\1 \"$socket_vmnet_path\"/"  ${HOME}/.lima/_config/networks.yaml

# Set up the sudoers file for launching socket_vmnet from Lima
limactl sudoers | sudo tee /etc/sudoers.d/lima

--- Template ---

# Deploy kubernetes via k3s (which installs a bundled containerd).
# $ limactl start --name=localk3s --tty=false ./localk3s.yaml
# $ limactl shell localk3sk3s sudo kubectl

# This templete requires Lima v0.7.0 or later.

# set VM resources
cpus: 2
memory: "4GiB"
disk: "20GiB"

networks:
  # Interface "lima0": shared mode  (IP is assigned by macOS's bootpd)
  - lima: shared

caCerts:
  certs:
    - |
      -----BEGIN CERTIFICATE-----
      MIIDRDCCAiygAwIBAgIJALQYXmXdKxsBMA0GCSqGSIb3DQEBCwUAMBsxGTAXBgNV
      BAMTEEFSSU4gSW50ZXJuYWwgQ0EwHhcNMTcwMzE3MDE1NTU1WhcNMjcwMzE1MDE1
      NTU1WjAbMRkwFwYDVQQDExBBUklOIEludGVybmFsIENBMIIBIjANBgkqhkiG9w0B
      AQEFAAOCAQ8AMIIBCgKCAQEAo5Pmq1aKWI1IpN2PEUUfL50Pi5dhgOD0C4yFkw9m
      4DysK45onSBF7Mt9fLKKJ+rjFQc/SORExTiu/p24sYGq8vBec2imxDrcgqVxBYOZ
      OVWOigFaKH71VzNz1OHF1DHEfHL7hhMC6cc0AI1OPeNoodFGun/e5MPtle6xVA8O
      vinXC789bhD/8nB9NxdZyZx/2MakhRPAssK0Tq70JQt8UcIyNWbVimrdJLrRQdLm
      mnnuzKkd4PTznzWdMYFt7F0obqjRaMFZtu2UwBnKgtI/MeKzYfWhAc8fEvReVijE
      7gYFRMrM50D/4ViSvZlCPYcZgUBYywCdTCQaZu9XIuFjnwIDAQABo4GKMIGHMB0G
      A1UdDgQWBBTl5qMBQAry/aBb/umtkdmgkVKM6TBLBgNVHSMERDBCgBTl5qMBQAry
      /aBb/umtkdmgkVKM6aEfpB0wGzEZMBcGA1UEAxMQQVJJTiBJbnRlcm5hbCBDQYIJ
      ALQYXmXdKxsBMAwGA1UdEwQFMAMBAf8wCwYDVR0PBAQDAgEGMA0GCSqGSIb3DQEB
      CwUAA4IBAQCi33XkYBbdj6dI4g0RwsARn7NMVp0DtzNkm/krA503RxVqgairrXiU
      RuRTL27wWZrNzJ6biQWQi9lXL3go1VXeBrc5OPIW70vuto2C9AADIZh3jOymYNOn
      HybcOiNhVFVNX4MFAEOPrY4A85Gt5ZWe5sqAmNKAlqLrwDYnUNun+plClaSFE4vm
      LiirpnQtDJHrmTqnA7zPvddH5YnYKHKev/Mgp21zFPG507qxbLMDzH/WUeXBOqS7
      HbfiEfqThzyjLAAd9V3L7MCYS1DVxb0Kz2Kz2VXwqdLNKzX3pe0/mLVEzWLLmOfE
      xXvKWeKNqLA69PHcVOFMuUdl69WCHQEY
      -----END CERTIFICATE-----

# force ARM systems to run a VM with a foreign architecture
arch: "x86_64"

# enable fast mode 2
# reference: https://github.com/lima-vm/lima/blob/master/docs/multi-arch.md#fast-mode-2
vmType: "qemu"
rosetta:
  # Enable Rosetta for Linux.
  # Hint: try `softwareupdate --install-rosetta` if Lima gets stuck at `Installing rosetta...`
  enabled: true
  # Register rosetta to /proc/sys/fs/binfmt_misc
  binfmt: true

images:
# Try to use release-yyyyMMdd image if available. Note that release-yyyyMMdd will be removed after several months.
- location: "https://cloud-images.ubuntu.com/releases/22.04/release-20230518/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"
  digest: "sha256:afb820a9260217fd4c5c5aacfbca74aa7cd2418e830dc64ca2e0642b94aab161"
# Fallback to the latest release image.
# Hint: run `limactl prune` to invalidate the cache
- location: "https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.img"
  arch: "x86_64"

# Mounts are disabled in this example, but can be enabled optionally.
mounts: []

# containerd is managed by k3s, not by Lima, so the values are set to false here.
containerd:
  system: false
  user: false

provision:
- mode: system
  script: |
    #!/bin/sh

    # Install dependencies for Longhorn
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
    nfs-common open-iscsi

    # Enable/Start iscsid and nfs-common
    #
    # nfs-common is masked see: https://blog.ruanbekker.com/blog/2017/12/09/unmask-a-masked-service-in-systemd/
    rm /lib/systemd/system/nfs-common.service
    systemctl daemon-reload
    sudo sysctl --system
    systemctl start nfs-common
    systemctl enable nfs-common
    systemctl start iscsid
    systemctl enable iscsid

    cat << EOF > /etc/sysctl.d/90-k3s.conf
    net.ipv4.ip_forward=1
    net.ipv4.conf.all.forwarding=1
    net.ipv6.conf.all.forwarding=1
    EOF

    mkdir -p  /etc/rancher/k3s

    cat << EOF > /etc/rancher/k3s/config.yaml
    ---
    write-kubeconfig-mode: "0644"
    disable:
      - traefik
      - servicelb
    node-ip: $(ip -4 -o addr show lima0 | awk '{ print $4}' | awk -F/ '{print $1}')
    tls-san:
      - localk3s.ci.dev.example.net
      - lima-localk3s
      - lima-localk3s.ci.dev.example.net
    EOF

    cat << EOF > /etc/rancher/k3s/registries.yaml
    ---
    configs:
      harbor.example.net:
    mirrors:
      harbor.example.net:
        endpoint:
        - https://harbor.example.net
    EOF

    # install K3s
    curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.27.3+k3s1 sudo sh -

probes:
- script: |
    #!/bin/bash
    set -eux -o pipefail
    #if ! timeout 30s bash -c "until test -f /etc/rancher/k3s/k3s.yaml; do sleep 3; done"; then
    if ! timeout 30s bash -c "until kubectl wait node $(hostname) --for condition=ready; do sleep 3; done"; then
      echo >&2 "k3s is not running yet"
      exit 1
    fi
    if ! timeout 30s bash -c "until test -f /etc/rancher/k3s/k3s.yaml; do sleep 3; done"; then
      echo >&2 "k3s kubeconfig not generated yet"
      exit 1
    fi

Using a little wrapper script I trigger the creation / start of the VM with limactl start --name=localk3s --tty=false ./localk3s.yaml

This correctly creates the cluster and I'm able to access it via kubectl + helm after fetching the kubeconfig (k3s.yaml) off the VM.

Helm was used to install metalLB + ingress-nginx on the cluster. The controller and speaker for metalLB start successfully and an external IP address is correctly handed to the ingress-nginx controller

lima-localk3s:~$ kubectl get svc -n ingress-nginx
NAME                                 TYPE           CLUSTER-IP      EXTERNAL-IP       PORT(S)                      AGE
ingress-nginx-controller-admission   ClusterIP      10.43.216.101   <none>            443/TCP                      117s
ingress-nginx-controller             LoadBalancer   10.43.177.49    192.168.105.240   80:30968/TCP,443:30651/TCP   117s

Locally on the lima VM I'm able to curl that endpoint without issue

lima-localk3s:~$ curl -k https://192.168.105.240
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

If I attempt to curl from my host MacBook Pro I can see incomplete arp messages

❯ arp -a | grep bridge
? (192.168.105.1) at 16:7d:da:91:1d:64 on bridge100 ifscope permanent [bridge]
? (192.168.105.2) at 52:55:55:f1:a:94 on bridge100 ifscope [bridge]
? (192.168.105.240) at (incomplete) on bridge100 ifscope [bridge]    <----- here
? (224.0.0.251) at 1:0:5e:0:0:fb on bridge100 ifscope permanent [ethernet]

On the lima VM itself I see the following

lima-localk3s:~$ arp -a
? (10.42.0.5) at 26:bb:95:f6:7c:15 [ether] on cni0
host.lima.internal (192.168.5.2) at 52:55:c0:a8:05:02 [ether] on eth0
? (10.42.0.6) at <incomplete> on cni0
? (192.168.5.3) at 52:55:c0:a8:05:03 [ether] on eth0
? (10.42.0.7) at <incomplete> on cni0
? (10.42.0.8) at 82:2d:84:e1:e0:56 [ether] on cni0
? (10.42.0.2) at 9e:c5:ee:9e:f2:d1 [ether] on cni0
? (10.42.0.9) at <incomplete> on cni0
? (10.42.0.3) at fa:90:83:c5:8c:55 [ether] on cni0
_gateway (192.168.105.1) at 16:7d:da:91:1d:64 [ether] on lima0
? (10.42.0.4) at 9e:c9:ce:27:1c:e0 [ether] on cni0

Before updating macOS to 13.5 I was able to curl and access the ingress endpoints via a browser on my host macOS system.

My searching did come across some ARP issues with multipass on macOS that seems related, but I'm having trouble hunting down what in the upgrade might have triggered this issue.
https://multipass.run/docs/troubleshoot-networking#heading--arp-problems

Has anyone else come across similar issues ?

[smbambling]

This also happens when deploying applications that create an ingress object. As a test I've deployed the Karma project which created an ingress, which correctly received an external IP

lima-localk3s:~$ kubectl get ingress -A
NAMESPACE    NAME    CLASS    HOSTS                   ADDRESS           PORTS     AGE
monitoring   karma   <none>   karma.ci.dev.example.net   192.168.105.240   80, 443   33s

Again I'm not able to access this from my host, but I am from the VM itself.

[smbambling]

Even more information:

I've created a 3 node lima VM K3s cluster and deployed metalLB + ingress-nginx via the same Helm method. All the deployments are successful and the external IP address is granted to the ingress-nginx controller.

A good point of information, is that from each of the lima VMs I'm able to curl and hit the ingress endpoint.

lima-localk3sserver1:~$ curl -k https://192.168.105.240
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
lima-localk3sserver1:~$ exit
logout

❯ limactl shell localk3sserver2
lima-localk3sserver2:~$ curl -k https://192.168.105.240
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
lima-localk3sserver2:~$ exit
logout

❯ limactl shell localk3sserver3
lima-localk3sserver3:~$ curl -k https://192.168.105.240
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

However on the local VMS I'm not able to arping via the lima0 interface and get a reply

lima-localk3sserver3:~$ arping -I lima0 192.168.105.240
ARPING 192.168.105.240 from 192.168.105.13 lima0
^CSent 7 probes (7 broadcast(s))
Received 0 response(s)```

[smbambling]

After thinking a bit more I was able to verify that this wasn't an issue with the MacOS bridge. To verify this I did a TCPdump on one of the lima VM nodes

sudo tcpdump -n -i lima0 arp

and then from my host system performed an arping

arping -i bridge100 192.168.105.240

On the lima VM tcpdump output, I could see that the request was getting passed via the bridge to the nodes but there was no ARP reply

lima-localk3sserver3:~$ sudo tcpdump -n -i lima0 arp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on lima0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:47:12.926622 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:13.931447 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:14.936350 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:15.936444 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:16.936453 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:17.941090 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:18.945451 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:19.949370 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:20.953620 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:21.956646 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44

This pointed me back into the direction of metalLB and its configs, looking at the troubleshooting guide for metallb(https://metallb.universe.tf/troubleshooting/#troubleshooting-service-advertisements), there is a section for why the speaker wont advertise. Because I'm using L2 the below line stood out.

A given speaker won’t advertise the service if:
  there are no L2Advertisements / BGPAdvertisements matching the speaker node (if node selectors are specified)

Taking another look at my IPAddressPool and L2Advertisement there was a typo mismatch which prevented the speakers from advertising the addresses from their pool. Once this was corrected I was able to see arping replies being sent and I'm able to access the ingress-nginx engpoint from my MacBook Pro host

10:47:22.611594 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:22.961476 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:22.961639 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:23.712650 ARP, Request who-has 192.168.105.240 (ff:ff:ff:ff:ff:ff) tell 192.168.105.240, length 46
10:47:23.712867 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:23.966126 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:23.966540 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:24.812157 ARP, Request who-has 192.168.105.240 (ff:ff:ff:ff:ff:ff) tell 192.168.105.240, length 46
10:47:24.812537 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:24.969903 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:24.970002 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:25.912960 ARP, Request who-has 192.168.105.240 (ff:ff:ff:ff:ff:ff) tell 192.168.105.240, length 46
10:47:25.912990 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:25.974707 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:25.974875 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:26.979440 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:26.979551 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:27.011598 ARP, Request who-has 192.168.105.240 (ff:ff:ff:ff:ff:ff) tell 192.168.105.240, length 46
10:47:27.011636 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:27.980470 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:27.980569 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46
10:47:28.981836 ARP, Request who-has 192.168.105.240 tell 192.168.105.1, length 44
10:47:28.981926 ARP, Reply 192.168.105.240 is-at 52:55:55:e1:4d:34, length 46