Repository navigation
Upgrade spored on a live instance in place (show latest available + spawn upgrade-spored), preserving state #234
Copy link
Copy link
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Summary
Builds on #232 (status now shows the spored version running on an instance). Two gaps remain:
- Show what's available, not just what's running.
spawn statusshows the on-instance spored version, but not whether it's behind the latest release — so you can't tell at a glance that an upgrade exists. - Upgrade spored in place on a live host, without terminating/relaunching the instance and without losing spored's lifecycle state (TTL deadline, idle timer, compute-seconds, completion/pre-stop config, FSx-pending leases).
Today the only way to get a newer spored onto a box is to terminate and relaunch, which is disruptive for long-lived/expensive instances (GPU, Capacity Blocks) and loses anything not yet checkpointed.
Part 1 — surface "upgrade available"
- Resolve the latest released spored version (GitHub Releases API for
spore-host/spawn, or the same artifact source the installer uses), cached briefly. - In
spawn status, annotate the spored line when behind, e.g.spored: v0.61.0 (latest v0.63.1 — run 'spawn upgrade-spored <instance>'). - Keep it best-effort/offline-tolerant: no network → just show the running version (don't fail status).
Part 2 — spawn upgrade-spored <instance> (in-place, state-preserving)
Proposed flow (SSM/SSH exec, mirroring how status already drives the instance):
- Resolve target version (default: latest;
--versionto pin; refuse a downgrade unless--force). - Fetch the matching spored binary for the instance's OS/arch onto the host (or push it).
- Preserve state across the swap. spored's durable state lives in EC2 tags (
spawn:ttl-deadline,spawn:idle-timeout,spawn:command,spawn:fsx-pending, compute-seconds, …) and on-disk files (/tmp/SPAWN_COMPLETE, any config). The upgrade must: stop the spored service, replace/usr/local/bin/spored, restart it, and have the new spored re-read its state from the tags/files (it already does this on boot) so the TTL/idle/compute clocks continue rather than reset. Verify compute-seconds and the absolute TTL deadline survive (the deadline is tag-stored and never reset by design — confirm the restart path honors that). - Health-check: confirm the new spored came back up (
spored statusreturns, version matches target) and the monitor loop is running; roll back / loud-fail if not. - Update the
spawn:spored-versiontag (introduce one) so the version is queryable without an exec.
Design questions to settle
- Service manager: systemd on AL2023 (
systemctl restart spored); Windows is a native service (Windows spored — cross-platform lifecycle agent as a Windows Service #77) — scope this to Linux first, file Windows follow-up. - State re-read vs handoff: simplest is "new spored boots and reconstructs from tags" (no IPC handoff). Confirm there's no in-memory-only state that would reset (e.g. idle timer baseline) — if there is, persist it to a tag/file as part of this work.
- Atomicity: download to a temp path,
mvinto place, restart — so a failed download never leaves a broken binary. - Authenticity: when Implement spore.host authenticity key: signed spore tag that all spore tools verify before touching an instance #76 (signed spore tag / authenticity key) lands, the upgrade should verify the new binary's signature before installing.
Acceptance
spawn statusshows running vs latest and flags when an upgrade is available (offline-tolerant).spawn upgrade-spored <instance>replaces spored on a running instance, the agent comes back healthy on the target version, and TTL deadline / compute-seconds / idle / completion config are preserved (verified: an instance mid-life keeps its original termination deadline, doesn't reset its clock).- Refuses downgrades without
--force; rolls back or fails loudly on a bad upgrade.
Refs
#232 (show running version), #76 (authenticity key — gate the upgrade), #77 (spored-on-Windows — Windows upgrade follow-up), lifecycle invariant (TTL always terminates; an upgrade must not extend or reset it).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
feat: spawn upgrade-spored — in-place, state-preserving spored upgrade (
Verify spored on the Windows launch path, and fail closed (#879) (#881)