Help debugging a 'stuck' VM

[EdwardMoyse]

Hi all,

I'm trying to test Lima for use in my organisation, but repeatedly when I try to compile a small part of our codebase as a test, the VM seems to become 'stuck' ... as in, the existing terminals become unresponsive, the qemu process stops using significant CPU on the host, and limactl shell alma9 no longer works (after >30mins, nothing has happened).

Initially I thought it was due to my using vz, but it happens also with qemu. I also worried that the VM was running out of memory / disk, but I reduced the cores (currently 4, but I also tried with only 1) and increased the memory (currently it is 16 GiB) and added a 64GiB swap ... none of this seems to be anywhere close to being exhausted by the time it freezes (but since it can take an hour to freeze, it's hard to be absolutely sure). I also added a very large (>500GiB) Volume to work in to make sure the disk wasn't exhausted, and when the VM freezes it is less than 1% full. So it's not that either. I also tried increasing the working disk to 200 GiB:

disk: 200GiB

(I'm not completely sure what this command does and what this disk space refers to ... but it seemed worth a try).

When I try to stop the VM:

emoyse@Edwards-MacBook-Pro-5 ~ % limactl stop alma9 
INFO[0000] Sending SIGINT to hostagent process 81989    
INFO[0000] Waiting for the host agent and the driver processes to shut down 
INFO[0000] [hostagent] Received SIGINT, shutting down the host agent 
INFO[0000] [hostagent] Shutting down the host agent     
INFO[0000] [hostagent] Stopping forwarding "/run/lima-guestagent.sock" (guest) to "/Users/emoyse/.lima/alma9/ga.sock" (host) 
FATA[0180] did not receive an event with the "exiting" status 

We do use our own filesystem (cvmfs) which I am running within the VM, so perhaps something is going wrong there ... but I'm not sure how to debug this? Or what to try next in general?

Thanks in advance for any tips!

[afbjorklund]

You can connect to the QMP socket, and use either QMP or HMP to help diagnose and troubleshoot.

• qmp-shell ~/.lima/alma9/qmp.sock

• qmp-shell ~/.lima/alma9/qmp.sock -H

HMP (Human) is recommended, see: https://qemu.readthedocs.io/en/latest/system/monitor.html

[EdwardMoyse]

I just tried again now and it seems to have died:

emoyse@Edwards-MBP-6 LocalDocuments % limactl stop alma9                                  
FATA[0000] expected status "Running", got "Broken" (maybe use `limactl stop -f`?) 

serial.log does not seem to have been updated recently (!) but in ha.stderr.log I see:

{"error":"dial tcp 137.138.16.5:53: connect: network is unreachable","level":"debug","msg":"handleDefault failed to perform a synchronous query with upstream [137.138.16.5:
53]","time":"2023-07-25T13:13:15+02:00"}

But at this time I was still able to connect with QMP, so maybe this doesn't mean what I think.

But the instance seems to have problems now. I can start it again, but then I cannot stop it normally:

emoyse@Edwards-MacBook-Pro-5 Lima % limactl stop alma9
INFO[0000] Sending SIGINT to hostagent process 61933    
INFO[0000] Waiting for the host agent and the driver processes to shut down 
FATA[0180] did not receive an event with the "exiting" status 

I'll try with display. Thanks!

[afbjorklund]

The serial.log would be the first place to look

[EdwardMoyse]

I got kicked out again from limactl shell alma9 and cannot reconnect but the display stayed active. I tried doing ls in the display shell, but got a timeout error (which is at the end of serial.log).

Now if I do ls I get ls: cannot open directory '.': Transport endpoint is not connected

I'm not really sure what else I can see in serial.log ... there are some strange errors at the beginning, but I'm not sure if they are related. I have attached it below.

serial.log

[EdwardMoyse]

Coming back to this again.

I'm (as usual) dealing with a stuck/frozen VM:

• display is unresponsive
• cannot shell into the VM
• limactl reports it is running

I did notice however that ha.stderr.log is still updating with e.g.

{"error":null,"level":"warning","msg":"Failed to clean up \"/Users/emoyse/.lima/alma_92_1c/ga.sock\" (host) after forwarding failed","time":"2023-11-08T16:19:08+01:00"}
{"error":"stat /Users/emoyse/.lima/alma_92_1c/ga.sock: no such file or directory","level":"warning","msg":"connection to the guest agent was closed unexpectedly","time":"2023-11-08T16:19:08+01:00"}

I checked and indeed the sock file is missing.

If I do limactl stop alma_92_1c this doesn't work cleanly (obviously --force would work):

{"level":"info","msg":"Shutting down VZ","time":"2023-11-08T16:21:33+01:00"}
{"level":"error","msg":"vz timeout while waiting for stop status","time":"2023-11-08T16:21:38+01:00"}

And limactl reports it is still running:

alma_92_1c    Running    127.0.0.1:64368    vz        aarch64    1       6GiB      100GiB    ~/.lima/alma_92_1c

But the .sock messages stop appearing.

[jerryguowei]

Hi @EdwardMoyse, do you have any progress on debug the issue. I am having exactly the same issue. I am deployed some new pods in k3s, which is installed inside lima. but the lima got stuck after some pod installation failure. I can confirm it didn't run out of memory, cpu and there is enough disk.