TF serving GPU test failing

[lluunn]

https://k8s-testgrid.appspot.com/sig-big-data#kubeflow-presubmit

[lluunn]

NAME READY STATUS RESTARTS AGE
inception-gpu-v1-74fc6fd6cf-ft226 0/1 Pending 0 15m

pending
Events:
Type Reason Age From Message

---

Warning FailedScheduling 2s (x57 over 15m) default-scheduler 0/6 nodes are available: 6 Insufficient nvidia.com/gpu.

[lluunn]

passing now

[lluunn]

Forgot to mention to fix.

kubectl get pods --all-namespaces -o wide --sort-by="{.spec.nodeName}" | grep Running | grep gke-kubeflow-testing-gpu-pool

Find the workload (which should be GCed), and delete manully