Skip to content

spawn autoscale advertises --region and ignores it (ambient region only) #774

Closed

Description

@scttfrdmn

Found incidentally while building #772's coverage check, which it made report a
confident false negative.

--region is advertised and ignored

spawn autoscale status --help offers it, inherited from the root:

--region string    Default AWS region (overrides SPORE_REGION/AWS_REGION and the shared config)

It overrides nothing. Demonstrated against an account whose orchestrator is in
us-east-1:

$ AWS_PROFILE=spore-host-infra spawn autoscale status --region us-east-1
⚠️  no autoscale orchestrator found in us-west-2 …

$ AWS_PROFILE=spore-host-infra AWS_REGION=us-east-1 spawn autoscale status --region us-west-2
autoscale orchestrator: spawn-autoscale-orchestrator-staging invoked by … (rate(1 minute))

The flag is ignored in both directions, and AWS_REGION wins even when the flag
contradicts it — the exact inversion of what the help text promises.

Cause

cmd/autoscale_helpers.go builds its config from the ambient chain with no
region applied, in two places:

  • getAutoscaler (line ~19) — the EC2/DynamoDB/SQS/CloudWatch clients
  • triggerLambda (line ~43) — the immediate-reconciliation invoke
cfg, err := config.LoadDefaultConfig(ctx)   // no config.WithRegion(...)

Other commands do this correctly, so the idiom already exists:

  • cmd/cancel.go:160,223 — config.LoadDefaultConfig(ctx, config.WithRegion(region))
  • cmd/doctor.go:42 — aws.NewClientWithRegion(ctx, doctorRegion)

Why it matters beyond tidiness

  • The DynamoDB scan fails outright in a region with no groups table:
    ResourceNotFoundException: Requested resource not found. That is the visible
    symptom users would hit first.
  • triggerLambda invokes spawn-autoscale-orchestrator-<env> by name in the
    ambient region.
    In a region where the function does not exist, the immediate
    reconciliation silently does not happen — and autoscale launch currently
    treats a trigger failure as a warning, not an error.
  • It made Two autoscale orchestrators have polled every minute since February against empty tables #772's new coverage check lie. The probe reported "no autoscale
    orchestrator runs in this account" against an account that has one. I worked
    around it by naming the probed region in the warning so the false negative is
    self-diagnosing, but that is a mitigation, not a fix — a cry-wolf coverage
    warning is the Nothing warns when an account has no TTL-reaper coverage, so the 'everything dies eventually' invariant silently doesn't hold #624 failure, and the workaround only makes it legible.
  • An account with per-region deployments (this one has both -production and
    -staging) cannot be addressed deliberately at all.

Fix

Thread the resolved region into both call sites, the way cancel and doctor
already do. Worth checking whether the right move is aws.NewClientWithRegion
for consistency with the rest of the codebase rather than a bare
config.WithRegion.

A test that --region reaches the client would catch the class; today nothing
asserts that any command's --region is honoured, which is why this survived.

Related: #772 (the coverage check this broke), #624 (cry-wolf coverage warnings).

Activity

  1. scttfrdmn commented on Oct 9, 2026

    @scttfrdmn
    ContributorAuthor

    Scope correction before the fix lands: this is four call sites, not the two I reported. The gate I wrote for it found the other two, and they are the more serious pair — both in cmd/ with cfg, _ := config.LoadDefaultConfig(ctx), discarding the error as well:

    • autoscale_status.go runAutoscaleHealth — reports a healthy group as having no instances
    • autoscale_lifecycle.go runAutoscaleTerminate — finds nothing to terminate and reports success while the instances keep billing

    That last one reframes the severity. I led with the DynamoDB ResourceNotFoundException because it is the most visible symptom; a terminate that silently no-ops is the one with a cost consequence, and it is the inverse of the leak the TTL reaper exists to prevent.

  2. added 3 commits that reference this issue on Oct 9, 2026
    d373ace
    c6b8240
    c8a699a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions