Repository navigation
Move the operated ttl-reaper off CloudFormation onto pkg/reaperdeploy (split from #654) #769
Description
Split out of #654, which closed when its remaining scope landed in #768.
What
The spore.host-operated ttl-reaper is deployed by CloudFormation/SAM. The
self-hosted one (spawn reaper deploy) is deployed by direct API calls through
pkg/reaperdeploy. #654 preferred consolidating the former onto the latter, and
#768 deliberately did not do it — it stamped the version instead, because that is
what made the skew visible, which was the actual complaint.
Why it is still worth doing
- One deployment path instead of two. Today a reaper change has to be
verified twice, in two mechanisms with different failure modes. - The CFN path has sharp edges that keep drawing blood. Its Makefile now
carries a read-live-parameters-first dance specifically because an omitted
parameter silently reverted to a template default and disarmed the live
production reaper (lambda/ttl-reaper: 'make deploy' defaults would DISARM the live production reaper and silently drop its DNS/alarm config #650, then reintroduced by its own fix).pkg/reaperdeploy
has no equivalent hazard: it passes what it is given. - ttl-reaper: cross-account trust names a CFN-generated role ARN suffix — recreating the stack breaks EVERY account at once (#457 trap 1) #476 is a CFN-specific trap — the cross-account trust names a
CFN-generated role ARN suffix, so recreating the stack breaks every onboarded
account at once. A Go path can use a stable role name. pkg/reaperdeployalready has the pieces:Tags,AssumeRolePolicy,
Inspect, arm/teardown, and unit tests for the behaviours that matter.
Why it is not urgent
The thing that hurt was not having two paths — it was that one of them was
unreadable. That is fixed: make lambda-versions-deployed now reports every
operated function's spawn:version, and CI gates that each deploy mechanism
stamps one (#768).
Related
- ttl-reaper: cross-account trust names a CFN-generated role ARN suffix — recreating the stack breaks EVERY account at once (#457 trap 1) #476 — the CFN-generated role ARN trap, which this would remove
- ttl-reaper: production deploy parameters are recorded nowhere in the repo — a bare
make deploysilently regresses prod #474 — production deploy parameters recorded nowhere in the repo, same class - spawn-pipeline-orchestrator's CloudFormation template is not in the repo #754 — the pipeline orchestrator's template, recently converged back to SAM
form; worth deciding whether that one should move too, or stay
Activity
scttfrdmn commentedon Oct 9, 2026
Recommendation: do not do this as specified. My premise above was wrong.
I wrote "pkg/reaperdeploy already has the pieces". It has them for the
self-hosted reaper, which is architecturally the opposite of the operated
one. Measured against the live functions today:
pkg/reaperdeploy)REAPER_SCAN_SELF=false— cross-account hubREAPER_SCAN_SELF: "true", hardcodedREAPER_ROLE_ARNS→ 435415984226sts:AssumeRolegrant at allSCAN_SELF,DRY_RUN,REGIONS)OptionsThe blocker is the IAM shape, not the line count
pkg/reaperdeploy's onlysts:AssumeRoleis in the trust policy — who mayassume the Lambda's role. There is no grant, so it cannot reach another account
at all.
That absence is not an oversight. It is the property that made the in-account
reaper acceptable in 942542972736, whose organization forbids trusting an
external principal, and #613's audit recorded it explicitly: "The inline policy
contains no
sts:AssumeRoleat all."So migrating the operated reaper onto this path means either:
property that audit turned on; or
defeats the one stated benefit.
Both cited motivations have cheaper fixes
the live reaper) is already mitigated by the read-live-parameters-first dance,
and now additionally by test(lambda): gate that every template parameter is actually passed #779, which fails CI when any template parameter is not
passed by its Makefile.
by setting an explicit
RoleName. No migration required.The trap is real — confirmed today, the dev-account role trusts
arn:aws:iam::966362334030:role/spawn-ttl-reaper-production-TTLReaperFunctionRole-ZJ84YZ2dCPeiwith
sts:ExternalId=spawn-ttl-reaper. But note what that means: changing therole name replaces the role and breaks every onboarded account until each one's
trust policy is updated. It is a coordinated migration in its own right, not a
one-line template edit, and it is the thing #476 already describes.
What the migration would actually cost
Reimplementing 6 CloudWatch alarms and 4 metric filters as imperative Go API
calls, on the one piece of infrastructure whose failure mode is instances not
being terminated. That trades declarative diffs, drift detection and automatic
rollback for hand-rolled reconciliation — and CloudFormation is genuinely good at
exactly this part.
Suggested re-scope
RoleNamefix — which addresses ttl-reaper: cross-account trust names a CFN-generated role ARN suffix — recreating the stack breaks EVERY account at once (#457 trap 1) #476, theconcrete trap, and needs a coordinated trust-policy update across onboarded
accounts. That is worth doing and does not require leaving CFN.
its deployed state was unreadable (spored and the lambdas never upgrade with the CLI, and nothing warns about the skew #654), and that is fixed:
make lambda-versions-deployedreports itsspawn:version, and test(lambda): gate that every template parameter is actually passed #779 gates thatevery parameter is passed.
pkg/reaperdeployas the in-account path, with no AssumeRole grant.Two deployment shapes exist because there are genuinely two situations: a
cross-account hub, and an account that may not trust one.
Happy to do (1) if wanted — but it touches every onboarded account's trust
policy, so it needs sequencing rather than a deploy.
scttfrdmn commentedon Oct 9, 2026
Reframed by #780 — the premise matters more than the mechanics
The project's actual architecture has three deployment shapes, and I was assessing this against two:
spawn reaper deploy(No supported way to run the TTL reaper in your own account, so governed orgs can't have reaper coverage at all #625), with nosts:AssumeRolegrant by design, for orgs that block exactly thatShape 3 is unsupported and was untracked. Now #780.
That changes what this issue is. Assessed as a cleanup — "one path instead of two" — it is a bad trade, for the reasons in my comment above: it means either adding an AssumeRole grant to the shape whose security story is the absence of one, or forking the policy and keeping two paths regardless.
Assessed as enabling work for shape 3, it is a different proposition. Shape 3 requires a self-deployable cross-account hub, so
pkg/reaperdeployneeds a deliberate hub mode whether or not spore.host's own reaper ever migrates. The migration then becomes a consequence — spore.host dogfooding the path it ships to institutions — rather than the goal.And it changes what gets built: a mode, chosen explicitly, not a widening of the default.
The constraint that has to hold
An in-account deploy must remain auditably free of
sts:AssumeRole. That absence is what made shape 2 acceptable in a governed org, and #613's audit is quotable because the answer was "zero". If hub mode lands, it needs a test asserting the in-account path emits no AssumeRole grant — otherwise the property degrades from a fact to a claim, and the next audit cannot repeat it.Suggested disposition
Keep this open but blocked on #780's design, rather than doing it as a refactor. The two things worth doing independently of either:
…TTLReaperFunctionRole-ZJ84YZ2dCPei), and fixable inside CFN with an explicitRoleName— but renaming replaces the role and breaks every onboarded account until each trust policy is updated, so it needs sequencing.make deploysilently regresses prod #474 — the operated stack's parameters are recorded nowhere, which is part of why shape 1's template is not something a third party could reuse.