Repository navigation
spored and the lambdas never upgrade with the CLI, and nothing warns about the skew #654
Description
Raised 2026-10-03. Answering "when a user updates spawn, do the lambdas also upgrade?" turned up that the answer is no for every component except a fresh launch — and that nothing surfaces the resulting skew.
What actually happens
| component | upgrades when? |
|---|---|
spawn / truffle / lagotto CLI |
when the user updates the CLI |
spored on a new instance |
automatically — the bootstrap downloads it at launch |
spored on a running instance |
only via spawn upgrade-spored <instance-id>, operator-initiated, one instance at a time |
| spore.host-operated lambdas (ttl-reaper, autoscale-orchestrator, pipeline-orchestrator) | only when a human runs sam deploy |
| self-hosted reaper (#625) | only when the operator re-runs spawn reaper deploy, which does fetch ArtifactURL(version) for the matching release |
There is no autonomous self-upgrade in pkg/agent — spawn upgrade-spored (#232/#234) does the clever part (in-place swap over SSM, no relaunch, lifecycle state preserved in EC2 tags, absolute TTL deadline not reset by the restart) but it has to be invoked.
Why it matters
Concretely: the spawn-ttl-reaper-production stack sat untouched from 2026-07-31 to 2026-10-04 while the CLI went from ~v0.9x to v0.116.0. In that window the deployed reaper was missing #504's not-invoked alarm, #625's fsx:/ssm: grants, and #645's packaging fix — and nothing anywhere said so.
So version skew between a user's CLI and the control plane is the normal state, not an exception. That is defensible as a design (they are independently deployed), but it is currently undocumented and invisible, which is the actual problem.
Shapes
- Detect and report. A
spawn doctorcheck comparing the CLI version against the deployed lambda versions (and the running spored versions it can see). Doesn't change the architecture, makes the skew visible. Cheapest useful step. - Document the contract. State plainly that the CLI and the control plane are independently versioned, and what that implies — rather than leaving users to assume an upgrade is end-to-end.
- Stamp versions where they can be read. The lambdas would need to expose their build version somewhere queryable (a tag, an env var, a log line) before (1) is possible. Worth checking whether they already do;
sporedalready publishes its version to an EC2 tag for exactly this reason (spawn status should show the spored version running on the instance #232/Upgrade spored on a live instance in place (show latest available +spawn upgrade-spored), preserving state #234). - A fleet-wide
upgrade-spored(all instances in a region, or by tag) rather than one id at a time — separate from the lambda question but the same "upgrade is manual and per-resource" shape.
(3) then (1) is the natural order, and (2) is worth doing immediately regardless.
Related: #652, and the control-plane reaping question filed alongside this.
Activity
scttfrdmn commentedon Oct 4, 2026
Partially addressed — PR #670 covers the self-hosted reaper
Worth recording what turned out to be already done, since the issue's suggestion (3) read as outstanding:
spawn reaper deployalready stamps the version.reaperdeploy.Tagssetsspawn:versionon the function and the execution role, andInspectalready reads it back intoInfo.Version. So for the self-hosted path, "stamp versions where they can be read" was complete — including the property that matters for a check, which is that it's readable in one API call without invoking the function.What was missing was anything comparing it to the running CLI. That's suggestion (1), and #670 adds it to
spawn reaper status, plus (2) — the help text now states the independent-versioning contract outright.Still open: the spore.host-operated reaper has no version at all
The SAM-deployed one exposes nothing. Verified against the live function:
ACCOUNTS_TABLE,EC2_EXTERNAL_ID,REAPER_DNS_*,REAPER_DRY_RUN,REAPER_MAX_AGE,REAPER_NOTIFY_URL,REAPER_REGIONS,REAPER_ROLE_ARNS,REAPER_SCAN_SELF— no versionaws:cloudformation:*,lambda:createdBy)So there is nothing for a check to read there, and that's the half this doesn't close.
A note on how to close it
The obvious move is another CFN parameter plumbed to an env var. I'd argue against doing it that way, given today's experience: two reaper changes landed this session that were correct-looking CFN and wrong in practice —
ec2:ResourceTagon anssm:action (#665), and a missing--regionthat reset every stack parameter and disarmed the production reaper (#666). Both were YAML that no compiler and no test could reach.pkg/reaperdeployalready does the equivalent work through the Lambda and IAM APIs from Go, with the version stamped as data rather than a template parameter — which is why the self-hosted half was already correct and testable. Extending that to the operated reaper is also the standing direction for the project (removing CFN in favour of the API), so it seems better to close this half as part of that rather than adding one more parameter to a template we intend to retire.Leaving this issue open for that half.
scttfrdmn commentedon Oct 4, 2026
Reopening — I closed this by accident.
PR #670 used
Fixes #654when it should have saidRefs: it covers the self-hosted reaper (version skew now reported byspawn reaper status, plus the contract documented in the help text), but the comment above says plainly that the spore.host-operated reaper still exposes no version at all — no env var, no tag beyond SAM's own, nothing in its Go source. There is still nothing for a check to read there.So the keyword contradicted the comment I posted a minute earlier. My own standing rule is to grep a PR body for closing keywords before merging, and the reason it exists is to catch exactly this.
Remaining scope, unchanged: give the operated reaper a readable version. Preferably as part of moving it off CloudFormation to the Go/API path
pkg/reaperdeployalready uses, for the reasons in the comment above.chore(release): v0.124.0
test(sweep): the detached-sweep harness passes; correct its header
test(sweep): the detached-sweep harness passes; correct its header (#766
feat(lambda): stamp the deployed version, and gate that every deploy …
scttfrdmn commentedon Oct 9, 2026
Closed by #768, which stamps
spawn:versionon all six operated deploy mechanisms and gates in CI that every deploy path does.To be transparent about how it closed: my PR body opened with
Closes #654's remaining scope.I believed the trailing'smade that inert —closingIssuesReferenceson the PR did report[]— but GitHub's merge-time parser read it asCloses #654and closed the issue. So the close was a regex decision rather than a deliberate one. The end state is the one I would have proposed anyway, so I have not churned it back and forth.The part this issue preferred and #768 did not do — moving the operated reaper off CloudFormation onto
pkg/reaperdeploy— is now #769, so it is not lost.What #768 found while implementing, beyond the stated scope:
scripts/deploy-custom-dns.shdeploysspawn-dns-updater; its own name says custom-dns, so enumerating by lambda directory never sees it.scheduler-handlerwith nospawn-prefix at all. A hardcoded list reports a clean bill of health for everything it forgot — the same lessonlambda-runtime-census.shrecords aboutgithub-oauth-bridge. It now enumerates.alert-evaluator, both autoscale orchestrators,cost-history-collector, and both dashboard-websocket functions. That is a much larger skew than the two-month ttl-reaper case this issue was filed about, and it was not visible at a glance before.And a correction worth recording: both halves of the new gate initially passed a deliberate revert.
grep -q spawn:versionalso matches the string in the parameter's ownDescriptionand in the script's ownecho, so the gate could never have fired. Tightened to require the assignment (spawn:version: !Ref Version) and a--tagsvalue. The ERE also had to move off[^\n]*, which excludes a literalnand therefore could never match "spawn:version".test(lambda): gate that every template parameter is actually passed (#…
build: make check runs the nested-module loop, via the script CI uses (…