Open
Description
Description
When the cluster up fails to bring up spot nodes (as a requirement for the autoscaler to infer the instances' resource specs), the advertised memory of a node (as specified in the metadata file) is the only option to use.
The problem is that the advertised memory is higher than the actual available memory on the instance. Investigate if multiplying the advertised memory with a percentage leads to an accurate approximation of the actual instances' memory specs across a wide array of instance types.
Underestimating is preferable to overestimating.
Configuration
When cluster.yaml is configured with:
# ...
min_instances: 0
instance_type: <something rare that has no available capacity>
spot: true
spot_config:
on_demand_base_capacity: 0
on_demand_percentage_above_base_capacity: 0
on_demand_backup: false
# ...