Skip to content

Upgrade spored on a live instance in place (show latest available + spawn upgrade-spored), preserving state #234

Closed

Description

@scttfrdmn

Summary

Builds on #232 (status now shows the spored version running on an instance). Two gaps remain:

  1. Show what's available, not just what's running. spawn status shows the on-instance spored version, but not whether it's behind the latest release — so you can't tell at a glance that an upgrade exists.
  2. Upgrade spored in place on a live host, without terminating/relaunching the instance and without losing spored's lifecycle state (TTL deadline, idle timer, compute-seconds, completion/pre-stop config, FSx-pending leases).

Today the only way to get a newer spored onto a box is to terminate and relaunch, which is disruptive for long-lived/expensive instances (GPU, Capacity Blocks) and loses anything not yet checkpointed.

Part 1 — surface "upgrade available"

  • Resolve the latest released spored version (GitHub Releases API for spore-host/spawn, or the same artifact source the installer uses), cached briefly.
  • In spawn status, annotate the spored line when behind, e.g. spored: v0.61.0 (latest v0.63.1 — run 'spawn upgrade-spored <instance>').
  • Keep it best-effort/offline-tolerant: no network → just show the running version (don't fail status).

Part 2 — spawn upgrade-spored <instance> (in-place, state-preserving)

Proposed flow (SSM/SSH exec, mirroring how status already drives the instance):

  1. Resolve target version (default: latest; --version to pin; refuse a downgrade unless --force).
  2. Fetch the matching spored binary for the instance's OS/arch onto the host (or push it).
  3. Preserve state across the swap. spored's durable state lives in EC2 tags (spawn:ttl-deadline, spawn:idle-timeout, spawn:command, spawn:fsx-pending, compute-seconds, …) and on-disk files (/tmp/SPAWN_COMPLETE, any config). The upgrade must: stop the spored service, replace /usr/local/bin/spored, restart it, and have the new spored re-read its state from the tags/files (it already does this on boot) so the TTL/idle/compute clocks continue rather than reset. Verify compute-seconds and the absolute TTL deadline survive (the deadline is tag-stored and never reset by design — confirm the restart path honors that).
  4. Health-check: confirm the new spored came back up (spored status returns, version matches target) and the monitor loop is running; roll back / loud-fail if not.
  5. Update the spawn:spored-version tag (introduce one) so the version is queryable without an exec.

Design questions to settle

Acceptance

  • spawn status shows running vs latest and flags when an upgrade is available (offline-tolerant).
  • spawn upgrade-spored <instance> replaces spored on a running instance, the agent comes back healthy on the target version, and TTL deadline / compute-seconds / idle / completion config are preserved (verified: an instance mid-life keeps its original termination deadline, doesn't reset its clock).
  • Refuses downgrades without --force; rolls back or fails loudly on a bad upgrade.

Refs

#232 (show running version), #76 (authenticity key — gate the upgrade), #77 (spored-on-Windows — Windows upgrade follow-up), lifecycle invariant (TTL always terminates; an upgrade must not extend or reset it).

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions