Repository navigation
fix(autoscale): honour --region instead of silently ignoring it - #775
Merged
Merged
Conversation
Fixes #774. The root --region flag is advertised in `spawn autoscale --help` as overriding SPORE_REGION/AWS_REGION and the shared config. It overrode nothing: every call site built its config with a bare config.LoadDefaultConfig(ctx), so the region came from the ambient chain. Ignored in BOTH directions, with the env beating the flag that claims to beat it: $ spawn autoscale status --region us-east-1 # probed us-west-2 $ AWS_REGION=us-east-1 … --region us-west-2 # used us-east-1 ## Four sites, not the two I reported The gate found two I had missed, and they are the worse two: autoscale_helpers.go getAutoscaler (EC2/DynamoDB/SQS/CloudWatch) autoscale_helpers.go triggerLambda (invokes by name) autoscale_status.go runAutoscaleHealth (cfg, _ := …, error discarded) autoscale_lifecycle.go runAutoscaleTerminate (same) **autoscale terminate is the one that matters.** With the wrong region it finds NOTHING to terminate and reports success while the instances keep billing — a cost-safety failure, and the inverse of the leak the reaper exists to prevent. `health` reported a healthy group as having no instances. The DynamoDB ResourceNotFoundException was merely the visible symptom. All four now route through autoscaleConfig, which goes through aws.NewClientWithRegion(ctx, spawnRegion) — the same resolution every other command uses, and which fixes the shared PROFILE these sites also ignored. This also replaces the mitigation in #773 with a real fix: the coverage probe used the same bare call and so reported "no autoscale orchestrator runs in this account" against an account that has one. Naming the probed region made that legible; sharing the resolver makes it correct. ## The gate TestAutoscaleHonoursTheRegionFlag forbids an unparameterised LoadDefaultConfig in cmd/autoscale*.go. A source assertion on purpose: proving a region reached the SDK needs a live account or an injected client, neither of which exists on this path, and forbidding the construct that caused it is cheap and sufficient. `config.LoadDefaultConfig(ctx, config.WithRegion(r))` does not match, so it bans the bug rather than the function. It skips comment lines, because the explanation above names the broken call and three gates this session passed against their own prose. Confirmed to fail on a COMPILING revert. The first attempt reverted the call without restoring the import, so the package did not build and the gate could not run — a build failure is not a passing gate, which is the second time that trap came up today. Verified live in both directions: --region us-east-1 now lists groups and finds the orchestrator; --region us-west-2 genuinely reports its absence there.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
scttfrdmn
added a commit
that referenced
this pull request
Oct 10, 2026
`spawn` is not only a CLI. 966362334030/us-east-1 holds 18 Lambda functions, 23 DynamoDB tables, 21 S3 buckets, 16 IAM roles, 18 log groups and 5 EventBridge rules — 103 control-plane resources — and nothing documented what they were or which code owned them. Every figure is MEASURED from the live account (2026-10-09) via `spawn footprint` and the AWS API, with the re-measure commands in the page. That matters because the answer was previously reconstructed from memory and three attempts at it were each incomplete: #653's inventory omitted log groups entirely, a prefix-filtered query found 10 of 14, and a bucket list missed an 11-bucket family. What the page records that the source does not make obvious: - **How each function is invoked** — EventBridge schedule, API Gateway, or direct CLI invoke. The last explains why four functions have no resource policy at all, which otherwise reads like a misconfiguration. - **Which live functions come from OTHER repos** (lagotto, spore-host umbrella, spawn-ts, prism). An unfamiliar function name in that account is not necessarily an orphan, and treating it as one would be a mistake. - **lambda/alert-handler/ has no live function anywhere.** It builds and vets in CI via the nested-module loop, but nothing runs it. Flagged for a decision rather than left ambiguous. - Both `spawn-autoscale-groups-*` tables are EMPTY — no autoscale group has ever been configured, which is the context for #772. - `spawn-ts-portal` is the only non-Go runtime (nodejs20.x). - The deploy traps, each of which cost a real failure: the space in `--parameter-overrides`, a parameter omitted there resetting to the template default (#650), SAM's `Enabled: !Ref` always resolving to ENABLED, the CFN-generated role ARN in the cross-account trust (#476), and autoscale ignoring --region until #775. One suspicion checked and NOT reported as a defect: the `spawn-binaries-<region>` buckets hold 538–803 objects, which looked like uneven replication of the agent. It is historical backfill — us-east-1 and us-west-2 carry 130 released versions, ap-northeast-1 carries 80 — and 0.126.1 plus spored-windows-amd64.exe are present in every region checked, so bootstrap is unaffected. A missing agent in one region would have been a silent regional failure, so it was worth verifying rather than asserting either way. Referenced from CLAUDE.md, since the thing most likely to go wrong is changing something under lambda/ without knowing the control plane has no reaper and spans several repos.
scttfrdmn
added a commit
that referenced
this pull request
Oct 11, 2026
…hey used (#772) (#822) BREAKING: 13 `spawn autoscale` subcommands removed, plus 13 exported members of pkg/aws. Next release bumps MINOR. The production orchestrator reconciled on rate(1 minute) from 2026-02-14 — roughly 350,000 invocations — against a table that has NEVER held a record. Both groups tables measured Count=0 at deletion. ## Why removal rather than tightening the role cloudformation/autoscale-ec2-role.yaml held, and the live role confirmed: ec2:RunInstances, ec2:TerminateInstances, ec2:CreateTags -> * iam:PassRole -> * Condition: iam:PassedToService = ec2.amazonaws.com iam:PassedToService constrains the SERVICE, not the role. With RunInstances on * in the same document, the holder launches an instance carrying any EC2-assumable role in the account and reads its credentials from IMDS. Trust was arn:aws:iam::966362334030:root — the whole infra account — gated by an sts:ExternalId that is published in the orchestrator template. Bounded today: no Administrator/PowerUser role in that account trusts ec2.amazonaws.com, so there is nothing privileged to reach. It becomes serious the moment someone creates one — which is the argument for deleting an unused primitive rather than trusting the role inventory to stay benign. The TerminateInstances on * was the more immediate problem: it could kill any instance in the compute account. ## Live removal Three CloudFormation stacks deleted, all DELETE_COMPLETE: spawn-autoscale-orchestrator-production function, role, rule, spawn-autoscale-orchestrator-staging permission, log group, table spawn-autoscale-ec2-role (compute acct) the role EVERYTHING was CloudFormation-managed, including that role. My first instinct was to delete the role directly; describe-stack-resources showed it belonged to a stack, so direct deletion would have caused drift and it would have come back. I verified that stack owned exactly one resource before deleting it. Control plane re-measured with `spawn footprint`: 100 -> 90. Not decremented by hand — an earlier edit of that page did the arithmetic manually and was wrong. ## A live route nearly went with it handleGetCostSummary and calculateInstanceCosts lived in lambda/dashboard-api/autoscale.go and have nothing to do with autoscale. Deleting the file wholesale silently removed /api/cost-summary; the build caught it. Moved to cost_summary.go VERBATIM — changing cost arithmetic inside a removal commit would hide a behaviour change in a mechanical one. That move surfaced a finding, documented in place rather than fixed: the handler runs on a hardcoded, UNDATED, us-east-1-only price table that defaults an unknown instance type to a fabricated $0.10/hr, then multiplies by 730 and shows it as a monthly estimate. truffle#114's rule is that a fabricated price is worse than no price. Tracked in truffle#184. ## The API-surface gate earned its place TestPublicAPISurface failed and named the 13 removed exported members and the three consumer repos. Verified zero consumers two ways — local checkouts of spore-host-mcp, lagotto and calque, and an org-wide code search — before accepting the break and running `make api-snapshot`. Those symbols were added THIS AFTERNOON in #774/#775 to warn when an autoscale orchestrator ran without coverage. That question no longer exists, so they go too. ## Also resolved spawn-registry-production grants removed from two IAM scripts. That table was already dead: dynamoRegistryTable was declared in autoscale.go and referenced nowhere, so the dashboard held access to a registry it never read. Found by the data-chain audit; removed here as a side effect. Four stale figures fixed, all mine: CLAUDE.md's function count (stale twice over), the Makefile's hand-written module count (now a discovery statement — the count had drifted 12 -> 10), and two wrong claims in docs/infra-account.md. nested-modules.sh reports 9 modules now, down from 10, noticed by discovery with no code change. make check green, check-docs, census, changelog, semgrep 0 findings, Tier 0 e2e with -count=1 against a fresh binary (68s).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #774.
The root
--regionflag is advertised inspawn autoscale --helpas overridingSPORE_REGION/AWS_REGIONand the shared config. It overrode nothing — everycall site built its config with a bare
config.LoadDefaultConfig(ctx), so theregion came from the ambient chain. Ignored in both directions, with the env
beating the flag that claims to beat it:
Four sites, not the two I reported on the issue
The new gate found two I had missed, and they are the worse two:
autoscale_helpers.gogetAutoscalerautoscale_helpers.gotriggerLambdaautoscale_status.gorunAutoscaleHealthcfg, _ :=, error discardedautoscale_lifecycle.gorunAutoscaleTerminateautoscale terminateis the one that matters. With the wrong region it findsnothing to terminate and reports success while the instances keep billing — a
cost-safety failure, and the inverse of the leak the reaper exists to prevent.
healthreported a healthy group as having no instances. The DynamoDBResourceNotFoundExceptionI originally led with was merely the most visiblesymptom, not the most serious one.
All four now route through one
autoscaleConfighelper that callsaws.NewClientWithRegion(ctx, spawnRegion)— the resolution every other commanduses, which also fixes the shared profile these sites ignored.
This replaces #773's mitigation with a real fix: the coverage probe used the same
bare call, which is why it reported "no autoscale orchestrator runs in this
account" against an account that has one. Naming the probed region made that
legible; sharing the resolver makes it correct.
The gate
TestAutoscaleHonoursTheRegionFlagforbids an unparameterisedLoadDefaultConfigincmd/autoscale*.go.A source assertion deliberately: proving a region reached the SDK needs a live
account or an injected client, neither of which exists on this path today, and
forbidding the construct that caused it is cheap and sufficient.
config.LoadDefaultConfig(ctx, config.WithRegion(r))does not match — itbans the bug, not the function.
It skips comment lines, because the explanation above names the broken call and
three gates this session passed against their own prose.
Confirmed to fail on a compiling revert. My first attempt reverted the call
without restoring the import, so the package didn't build and the gate couldn't
run — a build failure is not a passing gate, which is the second time that trap
came up today.
Verified live, both directions
The
ResourceNotFoundExceptionis gone, and the absence report is now genuinelyabout the region asked for.