Repository navigation
spawn autoscale advertises --region and ignores it (ambient region only) #774
Description
Found incidentally while building #772's coverage check, which it made report a
confident false negative.
--region is advertised and ignored
spawn autoscale status --help offers it, inherited from the root:
--region string Default AWS region (overrides SPORE_REGION/AWS_REGION and the shared config)
It overrides nothing. Demonstrated against an account whose orchestrator is in
us-east-1:
$ AWS_PROFILE=spore-host-infra spawn autoscale status --region us-east-1
⚠️ no autoscale orchestrator found in us-west-2 …
$ AWS_PROFILE=spore-host-infra AWS_REGION=us-east-1 spawn autoscale status --region us-west-2
autoscale orchestrator: spawn-autoscale-orchestrator-staging invoked by … (rate(1 minute))The flag is ignored in both directions, and AWS_REGION wins even when the flag
contradicts it — the exact inversion of what the help text promises.
Cause
cmd/autoscale_helpers.go builds its config from the ambient chain with no
region applied, in two places:
getAutoscaler(line ~19) — the EC2/DynamoDB/SQS/CloudWatch clientstriggerLambda(line ~43) — the immediate-reconciliation invoke
cfg, err := config.LoadDefaultConfig(ctx) // no config.WithRegion(...)Other commands do this correctly, so the idiom already exists:
cmd/cancel.go:160,223—config.LoadDefaultConfig(ctx, config.WithRegion(region))cmd/doctor.go:42—aws.NewClientWithRegion(ctx, doctorRegion)
Why it matters beyond tidiness
- The DynamoDB scan fails outright in a region with no groups table:
ResourceNotFoundException: Requested resource not found. That is the visible
symptom users would hit first. triggerLambdainvokesspawn-autoscale-orchestrator-<env>by name in the
ambient region. In a region where the function does not exist, the immediate
reconciliation silently does not happen — andautoscale launchcurrently
treats a trigger failure as a warning, not an error.- It made Two autoscale orchestrators have polled every minute since February against empty tables #772's new coverage check lie. The probe reported "no autoscale
orchestrator runs in this account" against an account that has one. I worked
around it by naming the probed region in the warning so the false negative is
self-diagnosing, but that is a mitigation, not a fix — a cry-wolf coverage
warning is the Nothing warns when an account has no TTL-reaper coverage, so the 'everything dies eventually' invariant silently doesn't hold #624 failure, and the workaround only makes it legible. - An account with per-region deployments (this one has both
-productionand
-staging) cannot be addressed deliberately at all.
Fix
Thread the resolved region into both call sites, the way cancel and doctor
already do. Worth checking whether the right move is aws.NewClientWithRegion
for consistency with the rest of the codebase rather than a bare
config.WithRegion.
A test that --region reaches the client would catch the class; today nothing
asserts that any command's --region is honoured, which is why this survived.
Related: #772 (the coverage check this broke), #624 (cry-wolf coverage warnings).
Activity
scttfrdmn commentedon Oct 9, 2026
Scope correction before the fix lands: this is four call sites, not the two I reported. The gate I wrote for it found the other two, and they are the more serious pair — both in
cmd/withcfg, _ := config.LoadDefaultConfig(ctx), discarding the error as well:autoscale_status.gorunAutoscaleHealth— reports a healthy group as having no instancesautoscale_lifecycle.gorunAutoscaleTerminate— finds nothing to terminate and reports success while the instances keep billingThat last one reframes the severity. I led with the DynamoDB
ResourceNotFoundExceptionbecause it is the most visible symptom; a terminate that silently no-ops is the one with a cost consequence, and it is the inverse of the leak the TTL reaper exists to prevent.fix(autoscale): honour --region instead of silently ignoring it (#775)
test(lambda): gate that every template parameter is actually passed (#…
remove: spawn autoscale, its orchestrators, and the escalation role t…