EKS has no autoscaler installed

[negz]

What happened?

A GPU node pool's minNodeCount/maxNodeCount have no effect on EKS, and the scheduler trusts maxNodeCount as if the pool will scale to it. Nothing on the workload cluster actually scales the node group, so a pool with nodeCount: 1, maxNodeCount: 4 only ever has one node, but the scheduler treats it as a four-node pool.

compose-eks-cluster maps the pool's counts straight onto the EKS managed node group's scaling config at fn.py:399:

scalingConfig=ngv1beta1.ScalingConfig(
    desiredSize=pool.nodeCount,
    minSize=pool.minNodeCount,
    maxSize=pool.maxNodeCount,
),

EKS treats minSize/maxSize as bounds for an autoscaler to move desiredSize between. But Modelplane installs no cluster-autoscaler or Karpenter on the workload cluster, so desiredSize (i.e. nodeCount) is the only count that ever takes effect. minNodeCount/maxNodeCount are inert.

This becomes a scheduling correctness problem because the InferenceCluster status reports the pool size as maxNodeCount, not the real node count. The XRD says so directly at inferenceclusters/definition.yaml:257:

nodes:
  type: integer
  description: >-
    Number of nodes in this pool. Derived from
    maxNodeCount (if autoscaling) or nodeCount.

So the ModelDeployment scheduler, which matches a gang against status.gpuPools[].nodes, will place a multi-node deployment on a pool that advertises enough nodes but can never produce them. The gang's extra pods stay Pending forever with no signal that the pool won't grow.

I hit the symptom directly while validating a multi-node (2-node gang) deployment: with nodeCount: 1 the worker pod could not schedule, and the node group never scaled up. Setting nodeCount: 2 fixed it; maxNodeCount: 4 did nothing.

0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 cannot allocate all claims. still not schedulable.

How can we reproduce it?

1. Create an InferenceCluster on EKS with a GPU pool sized below a gang, relying on autoscaling to cover it:

nodePools:
- name: gpu-l4
  className: eks-l4-1x-g6
  nodeCount: 1
  minNodeCount: 0
  maxNodeCount: 4

2. Deploy a multi-node ModelDeployment whose gang needs 2 nodes (e.g. examples/deployment/model-deployment-multinode.yaml).
3. The scheduler places the replica on gpu-l4 (its status.gpuPools[].nodes reports 4). The leader runs, but the worker pod stays Pending because the node group never scales past nodeCount: 1.

Workaround: set nodeCount to at least the gang size. The min/maxNodeCount fields don't substitute for it without an autoscaler.

What environment did it happen in?

Modelplane version: combined branch of #150/#154/#155/#160/#161/#162/#163 off main 48700e6d
Crossplane version: v2.3.2
Inference backend: vLLM 0.11.0 (LeaderWorkerSet, 2-node gang)
Cloud / cluster: AWS EKS, Kubernetes v1.36, provider-aws-eks v2.5.0

[haarchri]

I think we should look at EKS Auto Mode https://docs.aws.amazon.com/eks/latest/best-practices/automode.html
Auto Mode runs Karpenter as an AWS-managed part of the control plane. It also handles the nodes for us, creating them, picking AMIs, patching, and consolidating.

We describe each GPU pool as a Karpenter NodePool with a limits ceiling. Karpenter creates nodes just in time when pods are Pending. So when the scheduler places a 2-node gang, the nodes really show up — up to the cap. The mismatch between status and scheduler goes away, because the number we advertise is the number Karpenter scales toward.
GPU nodes are the right size. A NodePool lets us say what GPU instances we want (families, capacity type), set taints (nvidia.com/gpu), and turn on consolidation. We no longer pin one instanceType per managed group. Karpenter bin-packs the nodes and scales to zero when nothing runs, which matters for expensive GPU capacity.
Less for us to run. No autoscaler add-on to install and upgrade. No node-group AMI lifecycle to manage.

Some things to flag before we commit:

• GPU details for Pool Config, base AMIs etc.
• Cost, what additional cost on top for the AutoMode

[dennis-upbound]

Laying out the options for how EKS should handle node capacity, with trade-offs — no preference, want the team's call.

The bug: on EKS we set the nodegroup min/max, but we run no autoscaler, so only desiredSize (nodeCount) materializes. The scheduler trusts status.gpuPools[].nodes = maxNodeCount and places gangs onto nodes that never appear → pods hang Pending.

Option 1 — Don't autoscale EKS in v0.1 (PR #173). Advertise the realized nodeCount for sources where we don't control an autoscaler (EKS, Existing); keep maxNodeCount for GKE.

• Pro: smallest change, correct now, unopinionated, no new infra.
• Con: no elasticity on EKS — users size nodeCount themselves and min/maxNodeCount stay inert; bring-your-own autoscaler if they want one.

Option 2 — We install Karpenter / cluster-autoscaler as an add-on (the way we model coredns).

• Pro: real autoscaling; stays infra we control and can tune.
• Con: it's ours to run and upgrade (version lifecycle, IAM, ASG/subnet tagging); more moving parts; cluster-autoscaler still pins one instanceType per managed nodegroup.

Option 3 — EKS Auto Mode (@haarchri's suggestion). AWS-managed Karpenter via the cluster's computeConfig; GPU sizing as a Karpenter NodePool/NodeClass on the workload cluster.

• Pro: least for us to operate (no add-on lifecycle); JIT nodes so advertised capacity actually materializes (fixes this bug at the root); scale-to-zero (GPU cost win); symmetric with GKE's built-in autoscaler.
• Con: opinionated (Auto Mode owns the cluster's compute); a management surcharge on top of EC2; GPU support is still young (Bottlerocket AMIs + getting the DRA driver onto JIT nodes).
• Provider support: confirmed — clusters.eks.aws.m.upbound.io exposes computeConfig (enabled, nodePools, nodeRoleArn) + storageConfig/kubernetesNetworkConfig; the GPU NodePool is a workload-cluster CR we already know how to compose.

Whichever of 2/3 we land, the per-compose-path autoscaled flag in #173 flips to True for EKS and maxNodeCount headroom becomes honest again — so #173 is a clean step regardless.

@negz @haarchri — which direction for v0.1?

[negz]

@dennis-upbound Given we require DRA, sadly I think manually installing the cluster autoscaler (i.e. via a Helm chart) is our only option. Karpenter (and thus auto mode) don't support DRA. See kubernetes-sigs/karpenter#1231 and the "Considerations" section at https://docs.aws.amazon.com/eks/latest/userguide/device-management.html.

Note also I was wrong in that the autoscaler doesn't exist as an EKS Addon (i.e. you can't install it using the AWS EKS API when you provision a cluster). We'll need to use provider-helm and https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler/charts to install it.