Skip to content

No supported way to run the TTL reaper in your own account, so governed orgs can't have reaper coverage at all #625

Closed

Description

@scttfrdmn

The structural half of the #613 coverage gap. #624 is "stop being silent about it"; this is "make it fixable".

The gap

The ttl-reaper Lambda is designed to run in spore.host's infra account and reach into the accounts where spores launch, by assuming a role in each (REAPER_ROLE_ARNS → spawn-ttl-reaper-ec2, lambda/ttl-reaper/README.md:85-101). Coverage therefore requires the launching account to trust an external principal.

That works when the launching account is also spore.host-owned. It cannot work when the launching account belongs to an organization with a policy against external-account trust — which is the normal state of affairs for any enterprise, university or government org, i.e. much of the intended audience.

There is no alternative path. The reaper has no CLI deploy command (ls cmd/ | grep reaper → nothing); lambda/ttl-reaper/ ships a template.yaml and is deployed by hand. resolveAccounts does already support a self-contained mode (REAPER_SCAN_SELF=true with no REAPER_ROLE_ARNS, zero cross-account trust), so the runtime is capable of this today — what's missing is any supported way to stand it up.

So for a governed org the current answer is: grant a trust their policy forbids, hand-deploy a Lambda from a template in this repo, or have no backstop. In practice they get no backstop, and per #624 nothing tells them.

This was demonstrated, not hypothesised

Attempting the documented cross-account setup into an org-governed account produced an automated security finding within 12 minutes, and the org's tooling auto-mitigated by flipping every statement in the role's trust policy from Allow to Deny. The setup is not merely discouraged there; it is actively reverted. Least-privilege didn't help — the objection was the external principal itself, not its permissions, so sts:ExternalId and tag-gated TerminateInstances conditions are irrelevant to it.

Also worth noting the existing path is awkward even where it's permitted: the README documents a chicken-and-egg first deploy (deploy the Lambda, read its role ARN, deploy the roles, wait for the next tick), a CFN stack per launching account, and CAPABILITY_NAMED_IAM — the same brittleness that led to CFN being removed from lagotto's deploy path in lagotto#154.

Ask: spawn reaper deploy, following lagotto's precedent

lagotto already solved this exact shape. lagotto deploy — "Deploy the hosted capacity poller into your own AWS account" — downloads the published Lambda artifact for a given --version, uploads it to a bucket in the user's account, and converges SNS + Lambda + EventBridge Scheduler with direct AWS API calls, no CloudFormation (lagotto#154), with --teardown, --dry-run-style gating and a doctor that verifies the result.

The reaper should get the same treatment:

  • spawn reaper deploy stands up the reaper in the caller's own account — REAPER_SCAN_SELF=true, no REAPER_ROLE_ARNS, no cross-account trust anywhere. Intra-org and policy-compatible by construction.
  • Direct API calls rather than CFN, matching lagotto#154 and avoiding the per-account stack and CAPABILITY_NAMED_IAM.
  • --dry-run first, and REAPER_DRY_RUN=true as the default on a fresh deploy — a tool whose job is terminating instances should have to be armed deliberately, not arrive enforcing.
  • --teardown, and a doctor check that reports what got deployed and whether it is armed or in dry-run (feeds Nothing warns when an account has no TTL-reaper coverage, so the 'everything dies eventually' invariant silently doesn't hold #624 directly).
  • Keep the existing cross-account mode for the multi-account case where it is permitted; this adds a mode, it doesn't remove one.

Why this rather than a workaround

Two cheaper things were considered and rejected as answers to this:

  • Deploy the reaper by hand into the affected account. Works once, for one person, and leaves the product gap in place for everyone else.
  • Relocate workloads to an account the existing reaper already covers. Sound advice, and the right call for a personal account, but it's not available to an org that must launch inside its own boundary — which is the whole point of this issue.

The reaper enforces the project's core design invariant ("everything dies eventually"). Right now that invariant is only available to users who can grant spore.host cross-account access — and it's the users who can't who most need cost enforcement they don't have to remember.

Related: this belongs with the enterprise deployment work (self-hosted infra, account 812107987990). Refs #613, #624.

Activity

  1. scttfrdmn commented on Oct 3, 2026

    @scttfrdmn
    ContributorAuthor

    Done, in two PRs, and shipped in v0.115.0.

    Verified on the account this was written for

    The complete cycle, against the account that genuinely had no backstop:

    doctor    ⚠ not covered
    deploy    + role, + bucket, + artifact, + function, + schedule
    doctor    ✓ in-account reaper spawn-ttl-reaper-selfscan
    invoke    dry-run=true → FSxScanned:3 Errors:0, Reaped:0
    arm       (confirmation) → ARMED
    launch    t3.micro --ttl 1m
    invoke    REAPED i-0befb34f37c81fbe4 (reapertest) — ttl-deadline   [dry-run=false, Reaped:1]
    teardown  - 7 resources removed
    doctor    ⚠ not covered
    

    The reap is attributed to the reaper by its own log line rather than inferred, since spored also enforces TTL.

    And after the release, the path that could not exist until the asset did:

    $ spawn reaper deploy --version 0.115.0      # no --artifact
      + uploaded artifact s3://…/ttl-reaper/v0.115.0.zip (25689052 bytes)
      + created function spawn-ttl-reaper-selfscan
    

    The released artifact ran correctly in Lambda (fsx_scanned=3, errors=0). Account left exactly as found both times, leak-checked.

    Two things that differ from the issue as filed

    It deploys unarmed. REAPER_DRY_RUN=true, with spawn reaper arm as a separate confirmed step. The issue proposed this and it was the right call: the schedule logs what it would reclaim against your own account's tagging before you grant it the power to terminate, which is how spawn's own reaper was rolled out. Re-deploying an armed reaper does not silently disarm it.

    EventBridge Rule, not Scheduler. A rule needs no separate IAM invoke role — one fewer principal for a lagotto#170-style authorization bug to hide in. It also matches what the existing CFN template creates.

    A gap found on the way, worth knowing about

    The reaper's execution policy granted no fsx: or ssm: actions, while its code calls fsx:DescribeFileSystems/DeleteFileSystem (the #210 ephemeral-orphan net) and ssm:SendCommand/GetCommandInvocation (REAPER_GRACEFUL). In cross-account mode those run under the assumed spawn-ttl-reaper-ec2 role, which does grant FSx — verified against the live role — so the gap only ever affected scan-self, which is exactly what this feature uses, and it failed silently: instances terminated correctly while the orphaned 1200 GiB filesystem #613 was reported for was left behind.

    pkg/reaperiam now holds the set as data, with a test that reads the reaper's own *API interfaces and fails if a call has no grant — the structural end of the class behind #622 and lagotto #149/#151/#153. The CFN template's own ScanSelf policy is fixed too, so the hand-deployed path stops differing.

    The reverse half of that test (grant nothing uncalled) also established that ec2:DescribeTags, which both the template and the cross-account role grant, is never called — the reaper reads tags from the DescribeInstances response.

    Not covered here

    • Cross-account deployment. Deliberately the scan-self shape; the existing cross-account path is unchanged for operators who can use it.
    • REAPER_GRACEFUL's missing SSM grants on the cross-account role. Fixed on the Lambda's own policy here; the cross-account role still lacks them, so graceful mode silently degrades to an immediate terminate there. Worth its own issue.
    • The download path needs a released version. --artifact covers mirrored and air-gapped setups, and is how this was tested pre-release.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions