Start performance of 1.2.0 using vm-driver=none?

[BenHall]

Hello,

Is there any way to identify potentially bottlenecks with 1.2.0. We're seeing significantly longer start times (2m30s) than with 1.0 and previously. We set the log-level to 9 but nothing is jumping out. All the images and assets (kubeadm binary etc) have been pre-cached.

A couple of items we have spotted:

1. What we see is a long pause (60seconds) between the CNI starting and API and etcd being started. With the verify step, this blocks the user. Here are some timings we see:

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE
kube-system   coredns-5c98db65d4-j2q5l           1/1     Running   0          83s
kube-system   coredns-5c98db65d4-ltbrs           1/1     Running   0          83s
kube-system   etcd-minikube                      1/1     Running   0          19s
kube-system   kube-addon-manager-minikube        1/1     Running   0          18s
kube-system   kube-apiserver-minikube            1/1     Running   0          7s
kube-system   kube-controller-manager-minikube   1/1     Running   0          9s
kube-system   kube-proxy-x9pnh                   1/1     Running   0          82s
kube-system   kube-scheduler-minikube            1/1     Running   0          6s
kube-system   storage-provisioner                1/1     Running   0          80s

2. Addon enablement takes 30-60seconds to detect and start the deployment.

How can we start investigating where the performance issues and delays might be coming from?

[tstromberg]

To narrow it down, do you mind checking if performance characteristics are the same across minikube versions for the same release of Kubernetes? You can pass --kubernetes-version=v1.14.3 to both v1.0 and v1.2.0 for instance.

Normally I would also add checking for consitency with the same --iso-url, but the none driver doesn't use it.

Otherwise, I looked through the changelog for anything relating to the none driver or health checks since 1.0, but nothing jumped out at me. If it isn't a difference in the Kubernetes version, the next step would be git bisect. It should only take ~8 runs to isolate the behavior difference to a single git commit.

[BenHall]

@tstromberg Thanks! As I'm not missing something obvious I'll do some investigations between the different versions and report what I find

[tstromberg]

@BenHall - any luck?

[BenHall]

@tstromberg I'm finding it hard to isolate. With 1.2 I can't downgrade, and I can't use 1.15 with 1.1.

There are two issues I think

1. I understand why you added the verifying step, but kubectl reports "Running" while the checks are still happening and waiting for everything to start. Is it possible to "skip" this stage?

Update: I see a waitUntilHealthy but I can't see a way to set this. Is it in 1.3?

2. Is it possible to view the logs of kubeadm?

[tstromberg]

I think the answer is to set --wait=false. Once that's set, everything else is for the most part kubeadm. I'll leave #4990 as it seems like a vaild enhancement.