fix(start): recover from the broken states left by an unclean shutdown

[resker]

Problem

Filed as #5087. A KeepAlive-enabled LaunchDaemon crash-loops after an unclean shutdown, because the host agent can exit before the VM is stopped and limactl start then refuses to start from the state that leaves behind.

#5436 has since fixed the first of the two bugs of #5087 — the vz driver now asks the guest to stop over SSH when CanRequestStop is false, and removes vz.pid when it stops. This PR is rebased onto that and now covers the rest: recovering an instance that is already in one of the two broken states.

Changes

One commit per topic:

fix(store) — name the recoverable broken states, and only when provable
Sentinel errors (ErrDriverRunningHostAgentStopped, ErrHostAgentUnreachable) so callers match with errors.Is rather than on message strings. The second is reported only on ECONNREFUSED, the one outcome proving nothing is listening on ha.sock; a timeout, a protocol error, or a socket not yet bound can all come from a live host agent and must not be auto-recovered, or the caller would delete a live agent's files and start a second one. (On Windows WSAECONNREFUSED is not mapped to ECONNREFUSED, so the sentinel never fires there — reported but not recovered, which is the safe direction.)

fix(start) — recover from the broken states left by an unclean shutdown
An orphaned driver is force-stopped; an unreachable host agent has its files removed without signaling any process, since the recorded PID may since belong to an unrelated one. Only ha.pid/ha.sock are removed — a driver running as its own process may still be alive, and hiding it from the next inspection would allow a second driver for the same instance; a driver that runs in the host agent process (vz) records the same PID and is removed with them. The two run in sequence, not as alternatives, because clearing a departed host agent can reveal a driver that still needs stopping — and doing that after ha.pid is gone means the force stop cannot signal a reused PID.

docs(autostart) — documents both states, when they occur, and what limactl start does about each.

What is left after e529ce0 and #5436

Both narrowed how these states arise; neither closes them.

• e529ce0 makes ReadPIDFile ignore PID files written during a previous boot, covering PID reuse across a reboot.
• fix(vz): stop the VM even when CanRequestStop is false #5436 removes vz.pid whenever the vz driver stops, covering the launchd SIGTERM path for vz.

Still reachable, and what this PR recovers from:

• a driver that runs as its own process (qemu, and the other out-of-process drivers) outliving a host agent that died within the same boot — no reboot and no PID reuse needed, and nothing stops that VM or clears its PID file;
• PID reuse within a single boot;
• instances whose directory carries no boot session marker, because they were started by an older version of Lima — i.e. every existing autostart instance, until it is started once by a build that has e529ce0;
• platforms that expose no boot session ID (osutil.ErrBootSessionNotSupported), where PID files are handled as they were before the marker.

Testing

Unit tests for both halves: the socket cases drive the real host agent client against a unix socket rather than synthetic errors, and the cleanup cases cover in-process vs. separate driver.

Validated on Apple Silicon (macOS 26, arm64) with a --condition=boot LaunchDaemon running k3s — hard reboot auto-starts cleanly, both broken states recover, k3s and its workloads healthy afterwards. @CFBancroft independently validated the orphaned-driver recovery on a 2018 Intel Mac mini.

Closes #5087

Also fixes the problem reported in #5075, which is the stale-ha.sock state above; leaving that issue open for @lonlundgren to confirm against their setup rather than closing it from here.

Assisted-by: Claude Code

[lonlundgren]

Nice job.

This needs additional testing on Intel. HVF has reliability issues that don't show up on VZ that are especially exacerbated by launching a Lima VM closer to boot time.

See: #84 , #2252 , #5074

We have custom launchdaemon scripts running across stably Intel and Apple Silicon that encompass these fixes and the #5075 fix you included.

[resker]

Thanks @loncharles — noted wrt Intel systems... alas I don't have any accessible to validate w/. I'd had a look at some of your comments and wonder if you'd considered vzNAT as alternative to socket_vmnet - might sidestep the XPC timing issue and hew closer to what macOS is layering in.

[lonlundgren]

We absolutely need bridged network in our setup and it is the reason why we chose Lima instead of an alternative and so that we'd have one deployment surface area over Intel and Apple Silicon.

The XPC timing issue is less of a problem in practice, but socket_vmnet has several other repeatable ways it fails when you try to start a VM at boot. And much more reproducible on a machine that has a two-step boot when FileVault is enabled, than one that doesn't. So net net, the better ordering when wanting to boot a VM with a luanchscript is to have the script own start/retry on socket_vmnet creation and then only start Lima after.

The APIC timer kernel panic is independent of networking choice and is highly repeatable on Intel. Many people will hit this. And Lima made a change in their CI to fix it but never shipped any fixes in any of their VM template files.

[resker]

fyi, just tested this with socket_vmnet bridged networking on Apple Silicon (most of my use is k3s w/ NAT) — both the LaunchDaemon boot and keep-alive restart work w/o any observed issues.

[lonlundgren]

Good stuff. I should have been clear that these are primarily Intel issues and both are intermittent, but real and regular.

This should get some Intel coverage, if not include the other fixes, before merge.

[resker]

FYI, #5091 should improve the silent hang described

[resker]

loncharles — since you've Intel + FileVault setups running, could you give this a spin? I don't have Intel hw accessible.

EDIT: tagged the wrong person on these originally... removing the "@" to spare them further noise.

[resker]

loncharles — thanks again for the detailed notes. I believe this PR should be scoped to the LaunchDaemon restart behavior, with the two issues you raised handled separately, since they're really independent of the self-healing change here:

• socket_vmnet silent hang — addressed in fix(networks): fail fast when socket_vmnet daemon does not start #5091 (fail fast + retry with backoff for the XPC timing race).
• APIC timer kernel panic — this is a guest-kernel issue on x86 under HVF, independent of networking or how the VM is launched. I looked at shipping the no_timer_check fix in the templates, but it can't be done cleanly: the cmdline can only be set by pinning an external kernel + initrd by digest, which goes stale on every Ubuntu kernel update and would pin a kernel for all users to fix an Intel-only panic. That's presumably why it's only in CI today (via inject-cmdline-to-template.sh). Probably better tracked under Ubuntu and Debian intermittently fail to boot: Kernel panic - not syncing: IO-APIC + timer doesn't work #84 where it can get a proper fix.

EDIT: tagged the wrong person on these originally... removing the "@" to spare them further noise.

[AkihiroSuda]

Irrelevant change

[AkihiroSuda]

Use errors.Is or errors.As

[AkihiroSuda]

In the case of sudo, this should be explicitly printed

[AkihiroSuda]

Note: this PR stacks on #5086 (keep-alive flag) which stacks on #4984 (LaunchDaemon support). The diff includes those commits as context; the new changes are the four commits on top. Target base will move to master once those merge.

Let me mark this PR as a draft

[resker]

@AkihiroSuda — addressed:

• dropped the .gitignore change
• recovery detection now matches sentinel errors in pkg/store (ErrDriverRunningHostAgentStopped, ErrHostAgentUnreachable) via errors.Is instead of string-matching
• runSudo now logs at info level and notes it may prompt for a password

Also re-stacked on the updated #5086.

[resker]

@AkihiroSuda — I tested this directly. Two clarifications:

1. launchd isn't restarting a separate host-agent process of its own — the LaunchDaemon just runs limactl start <instance> --foreground, and KeepAlive re-runs that same command. So "restarting the host agent" really just means "re-running limactl start", the normal start path.
2. With vz the VM runs in-process in the host agent, so an unclean exit doesn't leave a separate VM process behind — the broken states this PR handles come from stale PID files after a reboot (a persisted PID that's since been reused by another process).

I reproduced both broken states on a test instance and confirmed limactl start recovers each one:

• orphaned driver ("driver is running but host agent is not") → force-stops the orphan, then starts cleanly → Running.
• stale host agent (ha.pid live but socket unreachable) → removes the stale files and starts cleanly → Running, without signaling the possibly-unrelated PID.

[CFBancroft]

Intel Mac test result for Pull Request #5088

I am a regular Mac mini user, not a Lima developer. I performed this testing step-by-step with help from ChatGPT.

Test hardware

2018 Intel Mac mini (Macmini8,1)
Intel Core i7, 6-core 3.2 GHz
64 GB RAM
macOS Sequoia 15.7.9 (24G830)
Lima/Colima using Apple VZ
Lima VM architecture: x86_64
Separate Mac Studio used as the remote test/control machine

How the headless test was performed

This was intended to be a real unattended server test, not a GUI-login test.

I enabled Remote Login on the Intel Mac mini and then rebooted it while the Lima/Colima VM was running.

After reboot, I did not log into any macOS GUI account on the Intel Mac mini. The Mac mini remained at the macOS login screen.

From a separate Mac Studio, I attempted to access my Docker-hosted DokuWiki service on host port 5555. The connection was refused, showing that the container stack had not started.

I then connected from the Mac Studio to the Intel Mac mini using SSH, while the Intel Mac mini was still sitting at its macOS login screen.

All subsequent launchctl, Unified Log, filesystem, and Lima checks were performed through that SSH session. This allowed me to inspect the failed boot without first logging into the Mac mini GUI and changing the headless test conditions.

Build result: PASS

I built the current #5088 branch locally on the Intel Mac. The resulting limactl was a native x86_64 Mach-O executable.

Manual #5088 recovery test: PASS

Before the headless reboot test, I reproduced the broken/orphaned Lima state:

vz driver is running but host agent is not

Running the #5088 limactl start colima manually as my dedicated containerserver account successfully detected the orphaned state, cleaned stale state, restarted VZ, restored the Docker socket, and reached:

READY.

My DokuWiki container on port 5555 then became reachable again.

This confirms that the #5088 recovery logic itself worked on this Intel Mac.

Normal Intel VZ shutdown: PASS

For additional Intel testing, I added a temporary diagnostic to vz_driver_darwin.go to report CanRequestStop().

During a fresh normal start/stop cycle on this Intel Mac, the result was:

CFB Intel test: CanRequestStop=true

The VZ VM shut down normally.

Therefore, the normal Intel shutdown path does not reproduce the CanRequestStop=false condition involved in #5087/#5088.

Headless LaunchDaemon reboot test: BLOCKED before Lima executes

After the real reboot, while the Intel Mac mini remained at the login screen, SSH inspection from the Mac Studio showed that the system LaunchDaemon had attempted to run once and exited with:

last exit code = 78: EX_CONFIG

The macOS Unified Log showed:

Service could not initialize: posix_spawn(.../limactl), error 0xd - Permission denied

Importantly, this occurs before limactl itself starts, so the #5088 recovery code never gets an opportunity to execute during this boot test.

I reproduced the posix_spawn EACCES behavior using multiple executable locations, including:

the Homebrew Lima 2.2.0 binary
my locally built #5088/CFB test binary
a copy of that binary placed under /Users/Shared/Server

The same locally built binary executes successfully when invoked manually as the dedicated containerserver account.

I also unloaded and freshly reloaded the LaunchDaemon after changing the executable location. launchd correctly loaded the new /Users/Shared/Server executable path, but still failed with:

posix_spawn(/Users/Shared/Server/lima-cfb/limactl), error 0xd - Permission denied

Therefore, simply moving the executable does not resolve the LaunchDaemon problem.

Result

On this 2018 Intel Mac mini:

Building #5088 for x86_64: PASS
Running the #5088 build manually: PASS
#5088 recovery from the broken/orphaned VZ state: PASS
Normal Intel VZ shutdown (CanRequestStop=true): PASS
Real reboot with no GUI login: TESTED
Remote verification from a separate Mac Studio over SSH while the Intel Mac remained at the login screen: CONFIRMED
Automatic LaunchDaemon execution after reboot: BLOCKED by macOS posix_spawn EACCES

Therefore, I cannot honestly mark the complete unattended reboot test for #5088 as either PASS or FAIL. The Intel #5088 code works when executed, but my true headless boot test is currently blocked by a separate macOS LaunchDaemon execution problem that occurs before Lima starts.

For Akihiro Suda

I understand that you are in the Tokyo area and mentioned that you do not currently have access to an Intel Mac for testing.

Accessibility note: I am Deaf. I communicate through written English. I also use some Japanese Sign Language (JSL), but I do not use Japanese fingerspelling. If we meet in person, please communicate with me using written English.

I have a train pass between Fujisawa and Ueno on the JR Tokaido Line (Ueno-Tokyo Line), so I can meet you somewhere convenient along that line.

I can bring my 2018 Intel Mac mini, keyboard, mouse, and a very small monitor. We can meet at a café or another place with a power outlet. That way, you can test Lima directly on an actual Intel Mac.

Alternatively, if meeting is unnecessary, please give me the exact commands or test procedure you want me to run. I can run them on the Intel Mac mini and post the complete terminal output here.

[CFBancroft]

Additional Technical Details — Exact Intel Test Code and Commands

Follow-up to my previous Intel Mac test report above.

I realized that my previous post described the hardware, headless reboot/SSH test, results, and the LaunchDaemon failure, but I did not include the actual source-code modification and exact commands I used during the Intel testing.

I am adding those details here so the test can be reproduced and so it is clear exactly what I changed and what I observed.

Important: The source modification below is my local CFB Intel experiment on top of Pull Request #5088. It is not code from Pull Request #5088 itself. I made this modification only to observe the Intel VZ shutdown behavior and experiment with the SSH-failure path.

Local Intel diagnostic modification

In:

pkg/driver/vz/vz_driver_darwin.go

immediately after:

logrus.Info("Shutting down VZ")

I added:

logrus.Infof("CFB Intel test: CanRequestStop=%v", l.machine.CanRequestStop())

I also changed:

logrus.WithError(err).Warn("vz: SSH shutdown failed; VM process may require manual cleanup")
return nil

to:

logrus.WithError(err).Warn("vz: SSH shutdown failed; continuing to observe VM state")

The purpose of this second experimental change was to allow execution to continue into the existing 30-second VM-state polling loop rather than immediately returning when the SSH shutdown request failed.

My resulting local diff was:

@@ -501,6 +501,7 @@
func (l *LimaVzDriver) Stop(ctx context.Context) error {
logrus.Info("Shutting down VZ")

•    logrus.Infof("CFB Intel test: CanRequestStop=%v", l.machine.CanRequestStop())
     if l.machine.CanRequestStop() {
  

...

•                    logrus.WithError(err).Warn("vz: SSH shutdown failed; VM process may require manual cleanup")
  

•                    return nil
  

•                    logrus.WithError(err).Warn("vz: SSH shutdown failed; continuing to observe VM state")
  

Building on the Intel Mac

I built limactl directly on the 2018 Intel Mac mini:

make limactl

The build completed for:

GOARCH=amd64

and produced a native x86_64 limactl.

The locally built version reported:

limactl version 2.1.2-beta.0-633-g0ba123d4.m

I also built the x86_64 guest agent and templates:

make x86_64-guestagent templates

The guest-agent target produced a gzip-related Error 1, but the resulting compressed x86_64 guest-agent file existed:

_output/share/lima/lima-guestagent.Linux-x86_64.gz

I then ran:

make templates

which completed successfully and populated the required runtime templates.

Starting the existing VM with my modified Intel build

I started my existing Colima VM with the locally compiled binary:

sudo -u containerserver -H env
LIMA_HOME=/Users/containerserver/.colima/_lima
/Users/macmini-clainkawada/Developer/lima-5088/_output/bin/limactl start colima

The VM successfully started and ultimately reported:

READY.

Docker became available again and my DokuWiki container on host port 5555 was reachable.

Intel CanRequestStop() result

I then performed a fresh normal start/stop cycle using the modified build.

My diagnostic produced:

CFB Intel test: CanRequestStop=true

The VZ VM shut down normally.

This shows that on my 2018 Intel Mac mini, a normal VZ shutdown uses the CanRequestStop=true path. I therefore could not reproduce CanRequestStop=false simply by performing a normal Intel start/stop cycle.

Pull Request #5088 recovery test

Separately from my local diagnostic modification, I tested the #5088 recovery behavior.

The instance had reported:

vz driver is running but host agent is not

I started it using the #5088 test build:

sudo -u containerserver -H env
LIMA_HOME=/Users/containerserver/.colima/_lima
/usr/local/bin/limactl-5088-test start colima

The #5088 code detected the orphaned state and attempted recovery.

During that process, the attempt to send SIGKILL to the old VZ driver reported:

operation not permitted

However, recovery continued. Stale state including vz.pid was removed, VZ restarted, the Docker socket was restored, and Lima ultimately reported:

READY.

My DokuWiki service on port 5555 then became reachable again.

Therefore, manual #5088 broken/orphaned-state recovery worked on this Intel Mac despite the SIGKILL operation-not-permitted message.

Attempt to reproduce the abnormal state manually

I also experimented with killing only the host agent after obtaining its PID from ha.pid.

After sending SIGKILL to that host-agent process, this did not reproduce the original orphaned-VZ condition. The VZ process disappeared as well, and limactl list showed the instance as stopped.

Therefore, I do not count that experiment as evidence either for or against #5088.

Relationship to my previous report

These are the source-code and command details that I accidentally omitted from my previous comment.

My previous comment contains the separate real headless reboot test: the Intel Mac mini was rebooted and left at the macOS login screen, I tested the service from a separate Mac Studio, and then connected to the Intel Mac mini from the Mac Studio using SSH without logging into the Intel Mac GUI.

That test remains blocked by the separate macOS LaunchDaemon failure:

Service could not initialize: posix_spawn(.../limactl), error 0xd - Permission denied

Because that error occurs before limactl executes, I do not consider the headless LaunchDaemon result a failure of #5088 itself.

Summary of this follow-up:

#5088 builds natively on my 2018 Intel Mac: PASS
#5088 manual orphaned-state recovery on Intel: PASS
My diagnostic build runs on Intel: PASS
Normal Intel VZ shutdown: CanRequestStop=true
Deliberately killing the host agent did not reproduce the original orphaned state
Full headless reboot testing remains blocked before Lima executes by the separate macOS launchd posix_spawn EACCES issue

I apologize for not including these exact technical details in my first report. I wanted to add them rather than edit the original comment so that the testing history remains clear.

[resker]

@CFBancroft - A few questions:

1. How did you create your LaunchDaemon? Was it w/ limactl autostart enable --condition=boot <instance>, or something put together manually? Lima's generated daemon plist sets UserName, so the job runs as that user rather than root — a hand-written one without it will behave quite differently.

2. If it is running as user other than root, does this work?

   sudo -u containerserver /Users/Shared/Server/lima-cfb/limactl --version

3. Could you post the output of the following (making sure it doesn't reveal anything sensitive / private beforehand)?

   sudo plutil -p /Library/LaunchDaemons/<the plist for your Lima job>.plist

[resker]

Rebased onto latest master (clean, no conflicts) and pushed four commits addressing the Copilot review:

• vz Stop — a failed SSH shutdown request returned nil, so the host agent exited 0 while the VM was likely still running, and KeepAlive{SuccessfulExit=false} would not restart it. It now falls through to the existing stop poll, as the CanRequestStop() branch already does.
• ErrHostAgentUnreachable — was attached to every failure to reach the host agent, including a timeout and the startup race where ha.pid is written before the socket is bound. Now only on ECONNREFUSED, the one result proving nothing is listening. Other failures are still recorded and still leave the instance Broken; they're just not auto-recovered. (On Windows WSAECONNREFUSED isn't mapped to ECONNREFUSED by syscall.Errno.Is, so the sentinel never fires there — reported but not recovered, which is the safe direction.)
• Stale-file cleanup — removed every .pid/.sock/.tmp in the instance dir, which could delete a live driver's PID file and let a second driver start. Now only ha.pid/ha.sock, plus the driver PID file when the driver runs in-process (vz) and records that same PID.
• The two recovery paths now run in sequence rather than as alternatives, so a driver revealed by clearing the host agent is force-stopped on its own — and since that happens after ha.pid is gone, the force stop can't signal a PID that may have been reused.

Added unit tests for both: the socket cases drive the real host agent client against a unix socket rather than synthetic errors, and the cleanup cases cover in-process vs. separate driver.

[CFBancroft]

@resker Thank you. Here are the answers to your questions.

I powered on the 2018 Intel Mac mini again tonight (September 6). I left the Mac mini at the macOS login screen and did not log into a GUI account. I am currently accessing it remotely from my separate Mac Studio using SSH. This is the same headless arrangement I was trying to test previously.

1. How was the LaunchDaemon created?

The LaunchDaemon I tested was created manually. I did not create it using:

limactl autostart enable --condition=boot

This may be an important distinction. My previous report should not be interpreted as a test of the LaunchDaemon generated by Lima's own autostart enable --condition=boot command.

I was trying to run the existing Colima/Lima VM under a dedicated macOS service account named containerserver, with no GUI login required.

2. Can containerserver execute the test binary manually?

Yes.

I previously ran:

sudo -u containerserver /Users/Shared/Server/lima-cfb/limactl --version

and received:

limactl version 2.1.2-beta.0-633-g0ba123d4.m

The same locally built binary also successfully starts the existing Lima/Colima VM when I invoke it manually as containerserver with the appropriate LIMA_HOME.

Therefore, the containerserver account itself can execute this binary.

3. Current LaunchDaemon plist

I ran the command you requested:

sudo plutil -p /Library/LaunchDaemons/io.lima-vm.daemon.colima.plist

Output:

{
"EnvironmentVariables" => {
"HOME" => "/Users/containerserver"
"LIMA_HOME" => "/Users/containerserver/.colima/_lima"
}
"Label" => "io.lima-vm.daemon.colima"
"ProcessType" => "Background"
"ProgramArguments" => [
0 => "/Users/Shared/Server/lima-cfb/limactl"
1 => "start"
2 => "colima"
3 => "--foreground"
]
"RunAtLoad" => 1
"StandardErrorPath" => "/var/log/colima-boot-error.log"
"StandardOutPath" => "/var/log/colima-boot.log"
"UserName" => "containerserver"
"WorkingDirectory" => "/Users/containerserver/.colima/_lima/colima"
}

This is the manually created plist that produced the posix_spawn(.../limactl), error 0xd - Permission denied behavior I reported earlier.

I also want to correct/clarify my previous report: because this plist was manually created, my LaunchDaemon failure does not demonstrate that Lima's own limactl autostart enable --condition=boot implementation has the same problem.

If you would like me to test the official Lima-generated boot autostart configuration instead, please give me the exact commands/procedure you want me to use. I will run them on this Intel Mac mini and report the complete output.

Likewise, I see that #5088 has changed since the 0ba123d build I tested. If you want me to rebuild and test the current #5088 head on this Intel Mac, please tell me which test sequence you would like me to run, and I will follow that sequence exactly.

[resker]

@CFBancroft — thanks, that clarifies things.

It also turned up a real gap, so thank you for that: limactl autostart enable generates a plist with no EnvironmentVariables, so LIMA_HOME isn't propagated and the job looks in ~/.lima at boot — which would never find a Colima instance under ~/.colima/_lima. Presumably why you hand-rolled the plist to begin with. Filed as #5488 with a draft fix in #5489 (credited you as reporter).

Independent of this PR, so nothing here changes.

If you're still curious about the EACCES on the manual plist, the one thing I didn't get was:

ls -lde / /Users /Users/Shared /Users/Shared/Server /Users/Shared/Server/lima-cfb

Since sudo -u containerserver ... --version works the mode bits look fine, so an ACL or a mount option is the likelier culprit.

[CFBancroft]

@resker Thanks. I ran the exact command you requested on the 2018 Intel Mac mini.

The Mac mini is still being accessed remotely from my Mac Studio over SSH.

$ ls -lde / /Users /Users/Shared /Users/Shared/Server /Users/Shared/Server/lima-cfb

drwxr-xr-x 22 root wheel 704 Aug 2 01:31 /
drwxr-xr-x 11 root admin 352 Aug 31 20:26 /Users
drwxrwxrwt 27 root wheel 864 Aug 31 23:06 /Users/Shared
drwxr-xr-x 4 root wheel 128 Sep 1 18:40 /Users/Shared/Server
drwxr-xr-x 3 root wheel 96 Sep 1 18:41 /Users/Shared/Server/lima-cfb

There were no additional ACL entries printed by ls -lde.

I have not changed any permissions, ACLs, mount settings, the LaunchDaemon, Lima, or Colima after running this command.

If you would like another check on this Intel Mac, please give me the exact command(s) and I will run them and post the output.

[resker]

@CFBancroft — that rules out the path then. I ought to point out that this thread isn't ideal for troubleshooting and isn't really topical to the PR at this point — it's not anything Lima is generating.

I'd suggest keeping an eye on #5488 / #5489 and adopting them if they merge. #5489 makes limactl autostart enable --condition=boot colima work with Colima's LIMA_HOME, which is what your hand-written plist is trying to do today — Lima generates the plist itself (with UserName and KeepAlive). #5088 here is the other half: recovery after an unclean shutdown.

[resker]

@AkihiroSuda — bumping this... since your review: a few Copilot findings were addressed & it's been rebased. CI indicates green — the one red is a cancelled release job. Most of the thread above is unrelated, but @CFBancroft also validated the recovery on an Intel Mac (I've only access to Apple Silicon based systems).

Anything further needed?

[AkihiroSuda]

Please tidy up the commits (one commit for one topic)

[AkihiroSuda]

The file should be named *_test_unix.go

[resker]

@AkihiroSuda - thanks and I've collapsed everything to four commits, one per topic (each builds on its own):

• fix(vz) (SSH shutdown fallback)
• fix(store) (the recoverable-state sentinels)
• fix(start) (the recovery itself)
• docs(autostart).

But the contributing guide says commits are usually squashed to one before merge — please advise if that's what you meant.

Also renamed the test file to staleagent_unix_test.go, matching pkg/limayaml/defaults_unix_test.go. I read your *_test_unix.go as a typo for that — let me know if you meant something else.

Rebased on master and updated the description to match the new commits.

[resker]

@AkihiroSuda — rebased. #5436 landed yesterday and supersedes my fix(vz) commit, so I dropped it (thanks @ekalinin — yours also removes vz.pid on stop, mine didn't). Three left, still one per topic: fix(store), fix(start), docs(autostart). The docs commit lost its VZ shutdown section for the same reason.

This PR is still applicable even though #5436 and e529ce0 both narrow how these states arise. The description has been rewritten to cover what's left rather than lean on the launchd SIGTERM story those two now cover — mainly an out-of-process driver (qemu) outliving a host agent that died in the same boot, which needs neither a reboot nor PID reuse, and instances with no boot session marker because an older Lima started them. #5075 is that same state, reported independently.

This is on the v2.3.0 milestone and #5436 went in after beta.0, so please advise if anything further is needed to proceed.

• These are one commit per topic, as requested. The contributing guide does say squash to one before merge — so happy to do that if preferred.

[resker]

Hi @AkihiroSuda — bumping this again. Still green & still clean on master.

[AkihiroSuda]

thanks