RSDK-14344: Don't reboot while an OS package upgrade is installing - #279
Open
Cheuk (cheukt) wants to merge 1 commit into
Open
RSDK-14344: Don't reboot while an OS package upgrade is installing#279Cheuk (cheukt) wants to merge 1 commit into
Cheuk (cheukt) wants to merge 1 commit into
Conversation
The syscfg upgrade worker and the manager's reboot poller ran with no shared state between them, so a pending-reboot indicator could trigger 'shutdown /r /t 0 /f' or 'systemctl reboot' while a package transaction was still in flight. On Linux that leaves a broken dpkg database; on Windows it tends to produce a rollback loop. Gate the reboot on both an in-process flag covering our own install and an OS-level probe that also catches transactions the agent did not start: F_GETLK against the dpkg/rpm lock files on Linux, and IUpdateInstaller.IsBusy (falling back to the TrustedInstaller service state) on Windows. Also separate 'a reboot is pending' from 'it is safe to reboot now', so that a reboot latched by an earlier cycle cannot fire during a later install. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes RSDK-14344.
Problem
In managed upgrade modes the agent both installs OS updates and owns the reboot, but the two loops share no state. The syscfg upgrade worker runs
apt-get/dnf/Get-WindowsUpdate -Installfor minutes, while the reboot poller (manager.go:730-745) checks every 60s and reboots if one is pending and the maintenance window is open. Nothing tells the poller an install is in flight, so a reboot can land mid-transaction — a dpkg database needingdpkg --configure -aon Linux, or a rollback loop on Windows.This isn't a narrow interleaving:
manager.go:156passesviamServer.RestartAllowedas syscfg'smaintenanceAllowed, so "upgrade may run" and "reboot may fire" are the same predicate. Two overlap paths need no timing luck — a config change re-runsStop/Update/StartwithneedsOSRebootstill latched, and after a process restart the agent re-derives "reboot pending" from the on-disk indicator while the startup install runs. On Linux that indicator is set mid-transaction anyway:/var/run/reboot-requiredis written by package postinst scripts (libc6,libpam0g,libssl3on stock Ubuntu 22.04) duringdpkg --configure.Fix
Gate the reboot on two checks, since OS state outlives agent state:
upgradeState) around our own install. Carries its own mutex rather than reusings.mu, whichStopholds while waiting for the upgrade worker to exit.F_GETLKagainst the dpkg/rpm lock files on Linux (non-destructive: it queries the holder, acquires nothing), andIUpdateInstaller.IsBusyon Windows, falling back to the TrustedInstaller service state. Unreadable state counts as busy.Both
NeedsOSRebootimplementations now separate is a reboot pending (latched) from is it safe to act now (re-evaluated each poll), with the gate applied to the cached-true path too. A held-back reboot logs once viablockNoticeand rearms when the condition clears.Testing
mise run lintclean forGOOS=linuxandGOOS=windows.go test -racepasses on Linux. (utils.TestInitPaths/failure_cannot_create_directoryfails when the suite runs as root in a container — pre-existing, passes as non-root.)TestPackageLockHeldre-execs the test binary as a child holding a realF_SETLKwrite lock (F_GETLKignores the caller's own locks), asserts the probe reports held with that PID, and re-probes to confirm it didn't take the lock.TestNeedsOSRebootWaitsForUpgradeToFinishcovers the end-to-end suppress/release.Reviewer notes
IsBusyagainst a live update session. Worth a manual check before shipping.RebootRequiredper-update or at session end. TheIsBusygate holds either way.pkgCmdusesexec.CommandContext), so agent shutdown can killapt-getmid-transaction with no reboot involved. Same corruption class, different trigger; noted on RSDK-14344.🤖 Generated with Claude Code