Repository navigation
Leave no trace: nothing reaps the control plane — the lambdas, their roles/rules/alarms, or the buckets spawn auto-creates #653
Description
Activity
Measured the footprint before acting, and two of this issue's priorities are wrong
Shape (1) — "make the footprint visible" — turns out to be worth more than it
looks, because doing the measurement by hand immediately contradicted the table
above. All figures from966362334030 / us-east-1today.(3) is NOT "the biggest real cost today"
The issue says an S3 lifecycle policy on
spawn-results-*is "arguably the
biggest real cost today since nothing ever prunes it." Measured:spawn-results-435415984226-us-east-1 182 objects, 83,302 bytes total oldest 2026-07-19, newest 2026-09-0583 KB after three months of use — a fraction of a cent per month, and only
one such bucket exists across both accounts. A retention policy there is still
right under "leave no trace" (bounded by design rather than by luck), but it is
hygiene, not cost control, and prioritising it over the items below would be
backwards. It also removes any pressure to pick an aggressive retention, which
matters given the issue's own "do not delete a user's data without asking".The actual unbounded thing is not in the table: log groups with no retention
10 log groups with retentionInDays = null (never expire) 51,236,505 bytesIncluding
/aws/lambda/spawn-dashboard-apiat 28.7 MB and
/aws/lambda/spawn-dns-updaterat 14.5 MB. Only 4 of 14 have retention set, and
those are the ones a SAM template declares (LogGroupwithRetentionInDays: 30)
— every log group auto-created by Lambda on first invocation has none, by
default, forever. That is the one artifact here that genuinely grows without
limit, and it is absent from the inventory above.The bucket count is 21, not 2
The table lists two bucket patterns. The account holds 21
spawn*/spore*
buckets, including a family nobody has mentioned:- 11 regional
spawn-binaries-<region>buckets (ap-northeast-1,
ap-southeast-1/2, ca-central-1, eu-central-1, eu-west-1/2, us-east-1/2,
us-west-1/2) — created Apr–May 2026 spawn-certs-us-east-1,spawn-cloudformation-artifacts-us-east-1,
spawn-pipelines-us-east-1,spawn-sweeps-us-east-1,
spawn-schedules-{us-east-1,us-west-2,eu-west-1}spawn-queue-results-test— a test bucket from 2026-05-01
Revised priority
- Set retention on the 10 log groups with none, and make every Lambda deploy
path declare it so a new function cannot create an immortal log group. This is
the only measured unbounded growth. It deletes log data older than the
window, so it needs a decision on the window rather than my picking one. - Shape (1), the inventory — now clearly the highest-value step, since doing
it by hand found a bucket family and a log-group class the issue did not know
about. Aspawn footprintthat enumerates rather than matching a remembered
list, for the same reasonlambda-version-census.shhad to stop hardcoding
function names (spored and the lambdas never upgrade with the CLI, and nothing warns about the skew #654: it listed 5, the account had 16). spawn-queue-results-testand the 11 regional binaries buckets want a
decision on whether they are still load-bearing, which is a question for a
human, not a reaper.spawn-results-*retention — do it, but as hygiene, last.
Nothing changed in AWS; this is measurement only.
- 11 regional
Retention applied, and both "cleanup candidates" turned out to be load-bearing
First, a correction to my own measurement above
I reported "10 log groups, 51 MB". It is 14 groups and 238.7 MB. My query
filtered onspawn*/spore*prefixes — which is the "match a remembered list"
mistake I had warned about one paragraph earlier, made immediately afterwards.
Enumerating without a prefix filter changes the picture substantially:group size note /aws/lambda/lagotto-capacity-poller189.79 MB 80% of the total, and not spawn's /aws/lambda/spawn-dashboard-api27.36 MB /aws/lambda/spawn-dns-updater13.88 MB /aws/lambda/github-oauth-bridge0.01 MB the function was deleted; its log group outlived it /aws/lambda/portal-phone-home,prism-bot, 9 others~7 MB github-oauth-bridgeis a clean "leave no trace" finding in its own right:
deleting a Lambda does not delete its log group. Anything that removes a
function needs to remove the group too, or the footprint outlives the thing that
made it.Applied: 30-day retention, everywhere spore.host owns
account / region result 966362334030 / us-east-1 14 groups set → 0 of 18 without retention 966362334030 / us-west-2 already 0 435415984226 / us-east-1 6 groups set (4 imagebuilder, capacity-poller, portal phone-home) → 0 remaining 942542972736 / us-west-2 only the 2 that are ours — lagotto-capacity-poller,spawn-ttl-reaper-selfscanThat last row was deliberately scoped. The account has 57 groups without
retention, but most belong to unrelated projects (geos-chem-*,
inside-the-lines-web-tool,/aws/batch/job,/aws/sagemaker/ProcessingJobs).
Applying a retention there would delete someone's project data on spore.host's
behalf, which is not spore.host's call. Worth flagging anyway:
geos-chem-cost-tracking-RealTimeCostTracker…holds 1,169 MB with no retention
— far larger than anything in the control plane, and entirely your decision.Neither bucket is deletable — both are load-bearing
I listed both as cleanup candidates. Checked, and both are wrong:
spawn-queue-results-testis empty, but referenced by integration
fixtures —testdata/integration/queue/timeout-queue.json,
dependency-queue.json(result_s3_bucket), and
scheduler/schedule-with-queue.yaml. Deleting it breaks those tests.The 11
spawn-binaries-<region>buckets are core infrastructure, not cruft:cmd/launch_config.go:541-543— the Windows bootstrap pulls
spored-windows-amd64.exefromspawn-binaries-$regioncmd/upgrade_spored.go:174—spawn upgrade-sporedfetches
spawn-binaries-${REGION}/spawn/versions/${TARGET_VERSION}- 803 objects in us-east-1, 538 in ap-northeast-1, including lambda artifacts
- newest object:
spored-windows-amd64.exe.sig, 2026-10-09T01:23:38Z — written
by the v0.126.1 release a few hours ago
They are per-region by design: an instance in ap-northeast-1 should not pull its
agent across the planet from us-east-1. Regional duplication is the feature.Where that leaves this issue
The one measured unbounded-growth item is now bounded, in code (#770) and in the
live accounts. What remains is the inventory command, and its case is stronger
than when this issue was written: three separate attempts at a footprint list
here — the issue's own table, my prefix-filtered query, and my bucket
"candidates" — were each incomplete or wrong. That is the argument for a command
that enumerates instead of a list anyone maintains by hand.- added 7 commits that reference this issue
on Oct 9, 2026
Project philosophy, stated 2026-10-03: spore.host should "leave no trace." This extends the existing lifecycle invariant ("everything dies eventually") from launched instances to the control plane itself.
The TTL reaper is the backstop for instances. Nothing is the backstop for the reaper, or for anything else the tools create.
Long-lived artifacts with no reaper
make deployspawn-results-<account>-<region>bucketstask run, auto-createdcompletion.jsonandcommand.logaccumulate indefinitelyspawn-reaper-artifacts-<account>-<region>bucketspawn reaper deploy(#625)reaper teardown, which removes rule/permission/function/role and reports what it keptCreateOrGetInstanceProfilelagotto deployThe artifact-bucket retention was a conscious choice at the time. Under "leave no trace" it is worth revisiting.
The hard part
MaxAge(168h) bounds instances. There is no equivalent notion for the control plane's own footprint, and a self-deleting reaper is chicken-and-egg: it has to outlive what it reaps. So this is a design question, not a one-line fix. Plausible shapes:spawn footprint/spawn orphans --control-plane: enumerate everything spore.host has created in an account, so "leave no trace" is at least auditable before it is automatic. Cheapest, and useful immediately.reaper teardown(and a new equivalent for the other stacks) remove everything including buckets, with an explicit--keep-artifactsopt-out rather than silent retention.spawn-results-*at creation time. Bounded, independent of the harder reaping question, and arguably the biggest real cost today since nothing ever prunes it.(1) and (3) are separable and worth doing regardless of how the rest lands.
Not in scope here
Deleting a user's data without asking. Several of these hold results someone may still want — which is exactly why the honest first step is making the footprint visible rather than automatically removing it.
Related: #650 (deploy hazard in the same area), #645 (packaging), #625 (
reaper deploy/teardown).