Skip to content

RSDK-14344: Don't reboot while an OS package upgrade is installing - #279

Open
Cheuk (cheukt) wants to merge 1 commit into
viamrobotics:mainfrom
cheukt:agent-restart-after-installs-finish
Open

RSDK-14344: Don't reboot while an OS package upgrade is installing#279
Cheuk (cheukt) wants to merge 1 commit into
viamrobotics:mainfrom
cheukt:agent-restart-after-installs-finish

Conversation

@cheukt

@cheukt Cheuk (cheukt) commented Jul 29, 2026

Copy link
Copy Markdown
Member

Fixes RSDK-14344.

Problem

In managed upgrade modes the agent both installs OS updates and owns the reboot, but the two loops share no state. The syscfg upgrade worker runs apt-get/dnf/Get-WindowsUpdate -Install for minutes, while the reboot poller (manager.go:730-745) checks every 60s and reboots if one is pending and the maintenance window is open. Nothing tells the poller an install is in flight, so a reboot can land mid-transaction — a dpkg database needing dpkg --configure -a on Linux, or a rollback loop on Windows.

This isn't a narrow interleaving: manager.go:156 passes viamServer.RestartAllowed as syscfg's maintenanceAllowed, so "upgrade may run" and "reboot may fire" are the same predicate. Two overlap paths need no timing luck — a config change re-runs Stop/Update/Start with needsOSReboot still latched, and after a process restart the agent re-derives "reboot pending" from the on-disk indicator while the startup install runs. On Linux that indicator is set mid-transaction anyway: /var/run/reboot-required is written by package postinst scripts (libc6, libpam0g, libssl3 on stock Ubuntu 22.04) during dpkg --configure.

Fix

Gate the reboot on two checks, since OS state outlives agent state:

  • In-process flag (upgradeState) around our own install. Carries its own mutex rather than reusing s.mu, which Stop holds while waiting for the upgrade worker to exit.
  • OS-level probe for transactions the agent didn't start — F_GETLK against the dpkg/rpm lock files on Linux (non-destructive: it queries the holder, acquires nothing), and IUpdateInstaller.IsBusy on Windows, falling back to the TrustedInstaller service state. Unreadable state counts as busy.

Both NeedsOSReboot implementations now separate is a reboot pending (latched) from is it safe to act now (re-evaluated each poll), with the gate applied to the cached-true path too. A held-back reboot logs once via blockNotice and rearms when the condition clears.

Testing

  • Cross-compiles for linux/windows/darwin; mise run lint clean for GOOS=linux and GOOS=windows.
  • Full go test -race passes on Linux. (utils.TestInitPaths/failure_cannot_create_directory fails when the suite runs as root in a container — pre-existing, passes as non-root.)
  • TestPackageLockHeld re-execs the test binary as a child holding a real F_SETLK write lock (F_GETLK ignores the caller's own locks), asserts the probe reports held with that PID, and re-probes to confirm it didn't take the lock. TestNeedsOSRebootWaitsForUpgradeToFinish covers the end-to-end suppress/release.

Reviewer notes

  • Windows paths are compile- and lint-verified only — no Windows host to exercise IsBusy against a live update session. Worth a manual check before shipping.
  • Open question tracked as RSDK-14345, not a blocker: whether the WUA writes RebootRequired per-update or at session end. The IsBusy gate holds either way.
  • Not addressed: cancelling the upgrade context SIGKILLs the package manager (pkgCmd uses exec.CommandContext), so agent shutdown can kill apt-get mid-transaction with no reboot involved. Same corruption class, different trigger; noted on RSDK-14344.

🤖 Generated with Claude Code

The syscfg upgrade worker and the manager's reboot poller ran with no
shared state between them, so a pending-reboot indicator could trigger
'shutdown /r /t 0 /f' or 'systemctl reboot' while a package transaction
was still in flight. On Linux that leaves a broken dpkg database; on
Windows it tends to produce a rollback loop.

Gate the reboot on both an in-process flag covering our own install and
an OS-level probe that also catches transactions the agent did not
start: F_GETLK against the dpkg/rpm lock files on Linux, and
IUpdateInstaller.IsBusy (falling back to the TrustedInstaller service
state) on Windows.

Also separate 'a reboot is pending' from 'it is safe to reboot now', so
that a reboot latched by an earlier cycle cannot fire during a later
install.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant