k8s example fails on kubectl requirement step (`Temporary failure resolving 'apt.kubernetes.io'`)

[danielhelfand]

Description

limactl version:

$ limactl -v
limactl version 0.8.2

I ran limactl start https://raw.githubusercontent.com/lima-vm/lima/master/examples/k8s.yaml, chose to Proceed with the default configuration, and by the end of the process Kubernetes was running successfully.

After stopping and deleting the lima instance using limactl, I then tried to restart the Kubernetes cluster using the same command above. Logs from limactl were as follows:

$ limactl start https://raw.githubusercontent.com/lima-vm/lima/master/examples/k8s.yaml

? Creating an instance "k8s" Proceed with the default configuration
INFO[0001] Attempting to download the image from "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-arm64.img"  digest=
526.75 MiB / 526.75 MiB [----------------------------------] 100.00% 25.90 MiB/s
INFO[0023] Downloaded image from "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-arm64.img" 
INFO[0023] Attempting to download the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v0.16.1/nerdctl-full-0.16.1-linux-arm64.tar.gz"  digest="sha256:e47fd1a03545ce539005a7f8a29644cb2ab515012c1c82b8be0bbafbb1670699"
168.20 MiB / 168.20 MiB [----------------------------------] 100.00% 43.42 MiB/s
INFO[0028] Downloaded the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v0.16.1/nerdctl-full-0.16.1-linux-arm64.tar.gz" 
INFO[0029] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/dhelfand/.lima/k8s/serial.log") 
INFO[0029] SSH Local Port: 61175                        
INFO[0029] [hostagent] Waiting for the essential requirement 1 of 3: "ssh" 
INFO[0059] [hostagent] Waiting for the essential requirement 1 of 3: "ssh" 
INFO[0059] [hostagent] The essential requirement 1 of 3 is satisfied 
INFO[0059] [hostagent] Waiting for the essential requirement 2 of 3: "user session is ready for ssh" 
INFO[0099] [hostagent] Waiting for the essential requirement 2 of 3: "user session is ready for ssh" 
INFO[0100] [hostagent] The essential requirement 2 of 3 is satisfied 
INFO[0100] [hostagent] Waiting for the essential requirement 3 of 3: "the guest agent to be running" 
INFO[0100] [hostagent] The essential requirement 3 of 3 is satisfied 
INFO[0100] [hostagent] Waiting for the optional requirement 1 of 5: "systemd must be available" 
INFO[0100] [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/dhelfand/.lima/k8s/ga.sock" (host) 
INFO[0100] [hostagent] The optional requirement 1 of 5 is satisfied 
INFO[0100] [hostagent] Not forwarding TCP 127.0.0.53:53 
INFO[0100] [hostagent] Waiting for the optional requirement 2 of 5: "containerd binaries to be installed" 
INFO[0100] [hostagent] Not forwarding TCP 0.0.0.0:22    
INFO[0100] [hostagent] Not forwarding TCP [::]:22       
INFO[0112] [hostagent] The optional requirement 2 of 5 is satisfied 
INFO[0112] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0115] [hostagent] Forwarding TCP from 127.0.0.1:39313 to 127.0.0.1:39313 
INFO[0152] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0193] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0233] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0273] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0313] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0353] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0393] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0433] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0473] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0514] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0554] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
INFO[0594] [hostagent] Waiting for the optional requirement 3 of 5: "kubectl to be installed" 
FATA[0628] did not receive an event with the "running" status 

From log file:

+ apt-get install -y kubelet kubeadm kubectl
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
   conntrack ebtables socat
 Suggested packages:
   nftables
 The following NEW packages will be installed:
  conntrack ebtables kubeadm kubectl kubelet socat
 0 upgraded, 6 newly installed, 0 to remove and 9 not upgraded.
 Need to get 32.4 MB of archives.
 After this operation, 215 MB of additional disk space will be used.
 Get:1 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 conntrack arm64 1:1.4.5-2 [28.8 kB]
 Get:2 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 ebtables arm64 2.0.11-3build1 [77.7 kB]
 Get:3 http://ports.ubuntu.com/ubuntu-ports focal/main arm64 socat arm64 1.7.3.3-2 [315 kB]
 Err:4 https://apt.kubernetes.io kubernetes-xenial/main arm64 kubelet arm64 1.23.3-00
   Temporary failure resolving 'apt.kubernetes.io'
 Err:5 https://apt.kubernetes.io kubernetes-xenial/main arm64 kubectl arm64 1.23.3-00
   Temporary failure resolving 'apt.kubernetes.io'
 Err:6 https://packages.cloud.google.com/apt kubernetes-xenial/main arm64 kubeadm arm64 1.23.3-00
   Temporary failure resolving 'packages.cloud.google.com'
 Fetched 421 kB in 1min 17s (5447 B/s)
 E: Failed to fetch https://apt.kubernetes.io/pool/kubelet_1.23.3-00_arm64_ee64fe02a5efe13ab328a61ff099e276c0e1c0c86900270f9b5843ccf0a9d1e9.deb  Temporary failure resolving 'apt.kubernetes.io'
E: Failed to fetch https://apt.kubernetes.io/pool/kubectl_1.23.3-00_arm64_e6c6e72f957b2576e2db007136466f1c42353447e760d49685cbb46a2c612aad.deb  Temporary failure resolving 'apt.kubernetes.io'
E: Failed to fetch https://packages.cloud.google.com/apt/pool/kubeadm_1.23.3-00_arm64_d79f598ceb8e64fcbcb2326b993dd7c1aaf642a446721676e65e5212320b1742.deb  Temporary failure resolving 'packages.cloud.google.com'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
 + systemctl enable --now kubelet
 Failed to enable unit: Unit file kubelet.service does not exist.
 LIMA| WARNING: Failed to execute /mnt/lima-cidata/provision.system/00000000
 LIMA| Executing /mnt/lima-cidata/provision.system/00000001
 + test -e /etc/kubernetes/admin.conf
 + export KUBECONFIG=/etc/kubernetes/admin.conf
 + KUBECONFIG=/etc/kubernetes/admin.conf
 + kubeadm config images list
/mnt/lima-cidata/provision.system/00000001: line 5: kubeadm: command not found
LIMA| WARNING: Failed to execute /mnt/lima-cidata/provision.system/00000001
LIMA| Exiting with code 1

[afbjorklund]

Unfortunately we haven't found a good solution for caching yet, so you would have to fix your internet (and dns)

   Temporary failure resolving 'apt.kubernetes.io'

Weird that two domains worked, and two domains failed ?

You can check what systemd is up to, with resolvectl status

[afbjorklund]

Is this still a problem, or did get the DNS issues get fixed in later versions of lima (and friends) ?

Not sure what else do to otherwise, except from maybe caching packages and images on the host.

• Caching packages and images on the lima host #397

It needs to be able to resolve apt.kubernetes.io and k8s.gcr.io

[piky]

For the first issuing of limactl start k8s, while cloud-init script is running apt-get update command.
I got the error message that "...connect (101: Network is unreachable) Could not connect to:"

• security.ubuntu.com:80
• archive.ubuntu.com:80 and
• pkgs.k8s.io:443

Lima v0.20.1
QEMU v8.2.1
MacOS 10.15.7 (Catalina)

WORKAROUND : Once limactl returned prompt, just simply stop and start the instance.