Repository navigation
Tracking: full backlog in batches (A: spawn fixes, B: adapter cost caps, C: changelog gate, D: lagotto, E: deferred) #655
Description
Organizing the full backlog into batches. Scope: all of it, agreed 2026-10-04. Batches are independent and can land in any order; only Batch A gates the spawn release.
Batch A — spawn core fixes (gates the spawn release)
- retryIAM doesn't retry ConcurrentModification, so concurrent
task runlaunches fail on CreateInstanceProfile #648 —retryIAMdoesn't retryConcurrentModification;terminateidempotent for names — PR fix(iam): retry ConcurrentModification, and make terminate idempotent #651 (green) - lambda/ttl-reaper: 'make deploy' defaults would DISARM the live production reaper and silently drop its DNS/alarm config #650 —
lambda/ttl-reapermake deploydefaults would disarm the live production reaper and blank its DNS/alarm config. Highest consequence item in the whole backlog. - REAPER_GRACEFUL cannot work cross-account: the spawn-ttl-reaper-ec2 role grants no ssm: actions #652 —
REAPER_GRACEFULcan't work cross-account (spawn-ttl-reaper-ec2grants nossm:). Precondition already checked:tryGracefulPreStopis best-effort andterminatealways runs, so Guardrail: TTL must always terminate — never add a TTL→stop mode #72's hard-deadline guarantee holds and the missing grant is not masking a worse bug. Also reconcile thefsx:DeleteFileSystemtag-condition discrepancy between this role andpkg/reaperiam— one of the two is wrong. - task run: the completion record's location is not settable, so a consumer must read two places #646 —
task run: the completion record's location isn't settable, so a consumer reads two places
Batch B — cost-cap the workflow adapters (cost safety, cross-repo)
No adapter emits lifecycle.cost_limit. TTL is the only ceiling in every integration, and the expensive failure is a job that hangs rather than fails — nothing retries or aborts it, and nothing else stops the spend. Worst case per fan-out is N × TTL × rate.
- snakemake-executor-plugin-spawn#12 —
container+cost_limit(in progress) - snakemake-executor-plugin-spawn#13 — not on PyPI; unpinned Snakemake/storage specs resolved per job
- nf-spawn#100 —
cost_limit - miniwdl-spawn#12 —
cost_limit - cwl-spawn#12 —
cost_limit - airflow-spawn#12 —
cost_limit - airflow-spawn#13 —
container(the only adapter missing both) - pegasus-spawn#5 —
cost_limit(prototype: the example builds its own spec, so likely a docs/example fix)
Batch C — changelog gate propagation (6 Go repos + 3 missing CHANGELOGs)
Only spawn has the gate; 18 of 19 repos lack it. In spawn it has caught a duplicate group three times in one session, so it earns its keep. Shared Go implementation (changelog_test.go + a CI step), copied.
- lagotto, libs, truffle, spore-host-mcp
- spore-host-plugin-globus, spore-host-plugin-tailscale
- CHANGELOG.md missing entirely —
spore-plugins,spore-host-plugin-rstudio,spore-host-plugin-spore-sync. Suite policy says every repo keeps one, so these are violations today.
Out of scope here, noted for follow-up: the Python adapters, nf-spawn (Groovy), pegasus-spawn (Shell), calque, cohort — each needs a separate non-Go implementation.
Batch D — lagotto
- lagotto#172 — GoReleaser builds the capacity-poller from
main.go
Batch E — deferred by agreement (do NOT block the release)
- spawn service --image: run a supervised headless container on a long-running instance #620 —
spawn service --image. A real feature (supervised container on a long-running instance, host networking), best sized on its own. - Leave no trace: nothing reaps the control plane — the lambdas, their roles/rules/alarms, or the buckets spawn auto-creates #653 — who reaps the control plane ("leave no trace"). Design question; first useful steps are a footprint/inventory command and an S3 lifecycle policy on
spawn-results-*. - spored and the lambdas never upgrade with the CLI, and nothing warns about the skew #654 — spored/lambda version skew. Blocked on the lambdas exposing a readable version before a
doctorcheck is possible. - --fsx-lifecycle ephemeral did not reap the filesystem when the instance terminated (1.2 TiB left AVAILABLE and billing) #613 — needs
spawn reaper deployrun in 942542972736, then a deliberate ephemeral-FSx orphan confirmed reaped. Root cause established: zero reaper coverage of that account, not an FSx logic bug.
Landed this session
#637, #645, #564, #565, #620-docs, #616, #639, #632, #642 — and the live production reaper redeployed from 75.7 MiB to 25.2 MiB with zero config drift and all 16 stack parameters preserved (DryRun=false intact, i.e. still armed).
Activity
[-]Tracking: pre-release cleanup across the tools touched this session[/-][+]Tracking: full backlog in batches (A: spawn fixes, B: adapter cost caps, C: changelog gate, D: lagotto, E: deferred)[/+]