Unschedulable: 0/2 nodes are available: 2 Insufficient cpu.

[SinaChavoshi]

The error message for resource shortage is incorrect. In this instance i was requesting 20 cpus where my cluster has a max limit of 10.

https://screenshot.googleplex.com/yvuBNnYqn0p

[hongye-sun]

This is a message from k8s. It says that you have two nodes in your cluster but none of them meet the requirements of requested CPUs.

[paveldournov]

@SinaChavoshi - please scale out the cluster by adding more nodes to the pool, or setup auto-scaling for the node pool.

@jlewi - should we consider setting up auto-scaling automatically in the 1-click deploy app?

[jlewi]

Autoscaling should already be enabled:
https://github.com/kubeflow/kubeflow/blob/797bcb7407a589bacc35b9624120f51f36a83468/deployment/gke/deployment_manager_configs/cluster-kubeflow.yaml#L46

My hunch is @SinaChavoshi is requesting a single pod with 20 CPUs. The current default VM type is 8 CPU
https://github.com/kubeflow/kubeflow/blob/797bcb7407a589bacc35b9624120f51f36a83468/deployment/gke/deployment_manager_configs/cluster.jinja#L118

This was by design to try to fit into the free tier.

There are two options here

1. Add a node pool with larger VM types

   • You can do this by adding a node pool to your DM configs like this
     https://github.com/kubeflow/kubeflow/blob/797bcb7407a589bacc35b9624120f51f36a83468/deployment/gke/deployment_manager_configs/cluster.jinja#L131

   • And then reapplying

2. Modify the node auto-provisioning settings to provision larger nodes.

It looks like auto-provisioning should be enabled:
https://github.com/kubeflow/kubeflow/blob/797bcb7407a589bacc35b9624120f51f36a83468/deployment/gke/deployment_manager_configs/cluster.jinja#L96

And should be able to provision larger nodes.

@SinaChavoshi Which version of Kubeflow are you using? I think we only turned on auto-provisioning in 0.4.

If you look at the K8s events for the resource that isn't getting scheduled (e.g. the pod) it should provide more information about why autoscaling didn't kick in.

[vicaire]

Resolving since there does not seem to be any next action to take. Please re-open if something else is needed.