Arch Linux template fails to start (connection refused)

[NateyB]

Description

Arch Linux template fails to start. The error that seems to switch the VM state from running to stopped is this one:

{
  "level": "debug",
  "msg": "stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 63388: Connection refused\r\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 63388: Connection refused\r\n\": exit status 255",
  "time": "2024-11-07T08:45:48-06:00"
}

Host: Macbook Air M2, macOS Sequoia 15.1

Lima Info:

~ % limactl -v
limactl version 1.0.0

Starting the VM:

~ % limactl create --name=arch template://archlinux

? Creating an instance "arch" Proceed with the current configuration
WARN[0001] vmType vz: ignoring [User]
INFO[0001] Attempting to download the image              arch=aarch64 digest="sha512:27524910bf41cb9b3223c8749c6e67fd2f2fdb8b70d40648708e64d6b03c0b4a01b3c5e72d51fefd3e0c3f58487dbb400a79ca378cde2da341a3a19873612be8" location="https://github.com/mcginty/arch-boxes-arm/releases/download/v20220323/Arch-Linux-aarch64-cloudimg-20220323.0.qcow2"
Downloading the image (Arch-Linux-aarch64-cloudimg-20220323.0.qcow2)
588.38 MiB / 588.38 MiB [----------------------------------] 100.00% 12.11 MiB/s^[[C
INFO[0050] Downloaded the image from "https://github.com/mcginty/arch-boxes-arm/releases/download/v20220323/Arch-Linux-aarch64-cloudimg-20220323.0.qcow2"
INFO[0050] Converting "/Users/nateyb/.lima/arch/basedisk" (qcow2) to a raw disk "/Users/nateyb/.lima/arch/diffdisk"
2.00 GiB / 2.00 GiB [---------------------------------------] 100.00% 1.11 GiB/s
INFO[0052] Expanding to 100GiB
INFO[0052] Attempting to download the nerdctl archive    arch=aarch64 digest="sha256:fe085381a09aa240ae5d1e0bbef1beccfb7c1d6dbb98bdc55bd416581d46ebc8" location="https://github.com/containerd/nerdctl/releases/download/v2.0.0/nerdctl-full-2.0.0-linux-arm64.tar.gz"
Downloading the nerdctl archive (nerdctl-full-2.0.0-linux-arm64.tar.gz)
195.48 MiB / 195.48 MiB [----------------------------------] 100.00% 11.73 MiB/s
INFO[0070] Downloaded the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v2.0.0/nerdctl-full-2.0.0-linux-arm64.tar.gz"
INFO[0070] Run `limactl start arch` to start the instance.
~ % 
~ % limactl start arch
INFO[0000] Using the existing instance "arch"
INFO[0000] Starting the instance "arch" with VM driver "vz"
WARN[0000] vmType vz: ignoring [User]
INFO[0000] [hostagent] hostagent socket created at /Users/nateyb/.lima/arch/ha.sock
INFO[0000] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/nateyb/.lima/arch/serial*.log")
INFO[0000] SSH Local Port: 63388
INFO[0000] [hostagent] [VZ] - vm state change: running
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 2: "ssh"
INFO[0000] [hostagent] [VZ] - vm state change: stopped
INFO[0000] [hostagent] Driver stopped due to error: "vz driver state stopped"
INFO[0000] [hostagent] Shutting down the host agent
ERRO[0000] [hostagent] accept tcp 127.0.0.1:63388: use of closed network connection
WARN[0000] [hostagent] failed to exit SSH master         error="failed to execute `ssh -O exit -p 63388 127.0.0.1`, out=\"Control socket connect(/Users/nateyb/.lima/arch/ssh.sock): No such file or directory\\r\\n\": exit status 255"
INFO[0000] [hostagent] Shutting down VZ
FATA[0001] exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/nateyb/.lima/arch/ha.stderr.log")

Empty serial logs:

~ % cat ~/.lima/arch/serial*.log
~ %

Stderr logs:

~ % cat ~/.lima/arch/ha.stderr.log
{"level":"debug","msg":"ResolveVMType: resolved VMType \"vz\" (existing instance, with \"/Users/nateyb/.lima/arch/vz-identifier\")","time":"2024-11-07T08:45:47-06:00"}
{"level":"debug","msg":"Creating iso file /Users/nateyb/.lima/arch/cidata.iso","time":"2024-11-07T08:45:47-06:00"}
{"level":"debug","msg":"Using /var/folders/hm/3bjlsptx67x0_thjl0rnn47c0000gn/T/diskfs_iso3225452499 as workspace","time":"2024-11-07T08:45:47-06:00"}
{"level":"debug","msg":"Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon.","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"OpenSSH version 9.8.1 detected","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing aes128-gcm@openssh.com and aes256-gcm@openssh.com","time":"2024-11-07T08:45:48-06:00"}
{"level":"info","msg":"hostagent socket created at /Users/nateyb/.lima/arch/ha.sock","time":"2024-11-07T08:45:48-06:00"}
{"level":"info","msg":"Starting VZ (hint: to watch the boot progress, see \"/Users/nateyb/.lima/arch/serial*.log\")","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"Start udp DNS listening on: 127.0.0.1:63344","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"Start tcp DNS listening on: 127.0.0.1:63389","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"Kernel file \"/Users/nateyb/.lima/arch/kernel\" not found","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"Using EFI Boot Loader","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"new connection from  to ","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"[VZ] - vm state change: \"VirtualMachineStateStarting\"","time":"2024-11-07T08:45:48-06:00"}
{"level":"info","msg":"[VZ] - vm state change: running","time":"2024-11-07T08:45:48-06:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 2: \"ssh\"","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/nateyb/.lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=nateyb -o ControlMaster=auto -o ControlPath=\"/Users/nateyb/.lima/arch/ssh.sock\" -o ControlPersist=yes -p 63388 127.0.0.1 -- /bin/bash -c $'while read -r line; do [ -n \"$line\" ] \u0026\u0026 export \"$line\"; done\u003c\u003cEOF\\n$(sudo cat /mnt/lima-cidata/param.env)\\nEOF\\n/bin/bash']","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 63388: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 63388: Connection refused\\r\\n\": exit status 255","time":"2024-11-07T08:45:48-06:00"}
{"level":"info","msg":"[VZ] - vm state change: stopped","time":"2024-11-07T08:45:48-06:00"}
{"level":"info","msg":"Driver stopped due to error: \"vz driver state stopped\"","time":"2024-11-07T08:45:48-06:00"}
{"level":"info","msg":"Shutting down the host agent","time":"2024-11-07T08:45:48-06:00"}
{"level":"error","msg":"accept tcp 127.0.0.1:63388: use of closed network connection","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"shutting down the SSH master","time":"2024-11-07T08:45:48-06:00"}
{"level":"debug","msg":"executing ssh for exiting the master: /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/nateyb/.lima/_config/user\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=nateyb -o ControlMaster=auto -o ControlPath=\"/Users/nateyb/.lima/arch/ssh.sock\" -o ControlPersist=yes -O exit -p 63388 127.0.0.1]","time":"2024-11-07T08:45:48-06:00"}
{"error":"failed to execute `ssh -O exit -p 63388 127.0.0.1`, out=\"Control socket connect(/Users/nateyb/.lima/arch/ssh.sock): No such file or directory\\r\\n\": exit status 255","level":"warning","msg":"failed to exit SSH master","time":"2024-11-07T08:45:48-06:00"}
{"level":"info","msg":"Shutting down VZ","time":"2024-11-07T08:45:48-06:00"}
{"level":"fatal","msg":"vz: CanRequestStop is not supported","time":"2024-11-07T08:45:48-06:00"}

[Da-Juan]

I ran into the same issue.
The aarch64 is from 2022 and does not seem maintained anymore, I didn't find any aarch64 cloud image for arch linux unfortunately.

[jandubois]

I can repro this issue; it doesn't work with VZ, but I could run it with QEMU:

$ limactl start --vm-type qemu template://archlinux
? Creating an instance "archlinux" Proceed with the current configuration
INFO[0001] Starting the instance "archlinux" with VM driver "qemu"
INFO[0001] QEMU binary "/opt/homebrew/bin/qemu-system-aarch64" seems properly signed with the "com.apple.security.hypervisor" entitlement
[...]
INFO[0268] [hostagent] Waiting for the guest agent to be running
INFO[0268] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished"
INFO[0271] [hostagent] Forwarding TCP from 127.0.0.1:42383 to 127.0.0.1:42383
INFO[0280] [hostagent] The final requirement 1 of 1 is satisfied
INFO[0281] READY. Run `limactl shell archlinux` to open the shell.

[mschirrmeister]

qemu works, but it takes 4-5 minutes to finally boot, because it tries for whatever reason PXE and HTTP boots over IPv4 and v6 before it finally boots.
Second issue with qemu is, that it is emulation and you have the qemu process constantly using 100% CPU.

Even if it boots, you will have issues with the upgrade, since the /boot partition is only 200MB which is to small for a recent 6.14 kernel.