feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools

[omeryahud]

Summary

WorkerPool replicas are static today: a human sets spec.replicas and the controller applies it verbatim. There is no mechanism that grows or shrinks a pool in response to actor demand. This issue opens a discussion on an autoscaling story for WorkerPools — what it should optimize for, why "just attach an HPA" is not sufficient for substrate's workload shape, and a concrete approach we've been exploring (a warm-buffer model with a fast scale-up trigger and a slow scale-down). The goal here is to align on direction and on the in-tree vs. out-of-tree boundary.

Autoscaling is already named as future work in docs/roadmap.md ("Worker horizontal autoscaling: Ability to rapidly scale up nodes and warm Pods to meet actor demand") and docs/architecture.md ("Autoscaling: We will need to be able to automatically scale the number of workers up and down based on demand…"), but there's no design or tracking issue for it yet.

Current state

• WorkerPoolSpec carries only replicas + ateomImage. No min/max bounds, no target utilization, no metric, no demand input.
• The scale subresource is exposed: the CRD declares
  +kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas,
  but HPA integration is incomplete; no pool-occupancy/capacity metrics exist
• A full pool fails fast: when no idle worker exists, AssignWorkerStep returns FailedPrecondition "no free workers available", which the router maps to HTTP 503. There is no queue/wait (tracked separately in Router needs to park requests and wait for capacity #27) and no capacity creation in response.

Net: the substrate partially provides the lever (the scale subresource) but no control loop and partial signals to drive one.

Why autoscaling matters

1. The density premise requires elasticity. Substrate's whole value is multiplexing many actors onto few warm workers. Without autoscaling, operators must statically over-provision for peak (wasting money on idle workers) or under-provision (and serve 504 timeouts under load). Elastic pools are what let density translate into actual cost savings.
2. Cost — scale-to-zero / scale-to-floor for idle pools. Idle actors are just snapshots in object storage; a pool serving no active actors is pure waste. Autoscaling (down to a small floor, or to zero for cold-start-tolerant pools) directly reduces spend.
3. Activation-latency SLO under bursts. Agent workloads tend to wake in correlated storms (sub-agent tasks/CI fan-out). Maintaining warm headroom and refilling it reactively is what helps the storm pass.
4. Eliminating avoidable 503s. Today an empty pool returns 503 with no recourse. Pairing capacity creation with the request-parking work in Router needs to park requests and wait for capacity #27 turns "no capacity → error" into "no capacity → brief wait → served."
5. Operability. Operators shouldn't hand-tune replicas per pool per time-of-day. A control loop (with sane bounds) mandatory for running this in production.

The core constraint - why a naive metric→HPA isn't enough

The binding latency is pod start (schedule + image pull + ateom boot + readiness — seconds to minutes; plus node provisioning if the cluster is full).

Pure metric-loop reactivity is bounded by detection lag. Prometheus scrape (~15–30s) + KEDA/HPA poll & stabilization (tens of seconds) is perfectly fine for smooth/diurnal demand, but too slow as the sole up-path for spiky wake-storms with a lean buffer — the buffer empties and stays empty through the lag window.

So "reactive enough" should be read as "fast enough to refill the buffer before the next comparable burst," not "fast enough to serve this burst."

Proposed approach (for discussion)

A warm-buffer invariant, replenished on consumption

Each pool targets a small number of idle (warm) workers. When a resume consumes a slot and the buffer dips below target, emit a refill — sized net of in-flight provisioning (anti-windup), so the loop doesn't pile on scale-ups while pods are still booting:

deficit = max(0, target - (free + in_flight)) // buffer refill
scale_to = clamp(current + deficit, reservation_floor, max_replicas).  // rate-limited per step

When preemption becomes available, preemptible actors will be deducted from the deficit.

Where should this live?

Substrate is intentionally low-opinion, so the policy could reasonably live outside the core. Much of the above is buildable today as an external controller — it can drive the existing scale subresource and read pool occupancy via a new WatchWorkers API.

Scale-to-zero and node headroom

• Scale-to-zero is attractive for idle/cold-start-tolerant pools, but the first post-zero request pays full cold start; keep a per-pool floor for latency-sensitive pools - Will handle in a follow-up issue
• Pod scale-up only helps if a node has room. A pod request that lands Pending behind cluster-autoscaler/Karpenter is not fast. Warm node headroom is a complementary lever worth calling out.

[omeryahud]

Update 2: The below approach has been dropped, the main issue was updated with the new approach
Update 1: dropping the stock-HPA down-path (re: open question 1)

When I opened this I assumed scale-down would be a stock metric loop (HPA/KEDA on the scale subresource), with a separate fast trigger only for scale-up. Building it out, I dropped HPA and folded both directions into a single in-tree autoscaler. Why:

1. Single-writer integrity — the decisive one. The original issue specifies spec.replicas has exactly one writer, for the fast up-path and the slow down-path. Creating a component that synchronizes pressure events from ateapi, and scale down events from an extern autoscaler is awkward (e.g. the trigger bumps the HPA's minReplicas instead of /scale) and still splits one control law across two systems. One component owning both directions makes single-writer structural instead of a coordination problem.

2. Fewer dependencies, no detection lag. No HPA/KEDA/Prometheus to install or operate. The loop reads occupancy straight from ateapi (ListWorkers) — no scrape lag.
   What it is now: a single WorkerPoolAutoscaler in ate-controller, the sole writer of spec.replicas. Up-path = event-driven capacity-pressure from ateapi (new WatchCapacityPressure stream, emitted where the resume misses). Down-path = the same loop's hysteretic poll on occupancy (which doubles as the safety net). minReady / targetBuffer / maxReplicas are first-class WorkerPoolSpec fields (re: open question 2).

flowchart TD
  subgraph dp["Data plane · ateapi"]
    MISS["Resume finds no free worker<br/>(AssignWorkerStep — t=0)"]
    OCC["Per-pool occupancy<br/>free / in-flight (ListWorkers)"]
  end
  MISS -->|"capacity-pressure event<br/>WatchCapacityPressure · fast"| AS
  OCC -->|"reconcile poll ~10s<br/>hysteretic · safety net"| AS
  subgraph ctl["ate-controller · single writer of spec.replicas"]
    AS["WorkerPoolAutoscaler 
UP immediate
 DOWN after stabilization"]
  end
  AS -->|"patch /scale"| WP["WorkerPool.spec.replicas"]
  WP --> WPR["WorkerPoolReconciler<br/>materializes the Deployment"]
  WPR --> PODS["Warm worker pods<br/>join / drain the buffer"]
  PODS -.->|"register / release"| OCC

Loading