Delete and init kubeadm on subsequent starts #13121

spowelljr · 2021-12-07T22:57:42Z

Problem:
--extra-options flags would not be respected on subsequent starts because they weren't being applied to the kube init phase steps during reconfiguration. Simply passing the flags into each steps is not possible because some flags are only available for kube init and not kube init phase (--skip-phases for example).

Solution:
Instead of trying to reconfigure, if re-starting kubeadm we delete and re-init it, there's nearly no time increase is doing this instead of the previous way of running specific phases. If running start on a running cluster it will skip any of this just like before.

Before:

$ minikube start --extra-config=kubeadm.skip-phases=addon/kube-proxy
...
$ kubectl --context minikube -n kube-system get daemonset -owide -l k8s-app=kube-proxy
No resources found in kube-system namespace.
$ minikube stop
$ minikube start
...
$ kubectl --context minikube -n kube-system get daemonset -owide -l k8s-app=kube-proxy
NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE   CONTAINERS   IMAGES                          SELECTOR
kube-proxy   0         0         0       0            0           kubernetes.io/os=linux   5s    kube-proxy   k8s.gcr.io/kube-proxy:v1.21.2   k8s-app=kube-proxy

After:

$ minikube start --extra-config=kubeadm.skip-phases=addon/kube-proxy
...
$ kubectl --context minikube -n kube-system get daemonset -owide -l k8s-app=kube-proxy
No resources found in kube-system namespace.
$ minikube stop
$ minikube start
...
$ kubectl --context minikube -n kube-system get daemonset -owide -l k8s-app=kube-proxy
No resources found in kube-system namespace.

spowelljr · 2021-12-08T00:35:42Z

/ok-to-test

minikube-pr-bot · 2021-12-08T01:41:29Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13121) |
+----------------+----------+---------------------+
| minikube start | 47.0s    | 45.9s               |
| enable ingress | 31.2s    | 31.2s               |
+----------------+----------+---------------------+

Times for minikube start: 50.2s 46.1s 46.9s 45.7s 46.0s
Times for minikube (PR 13121) start: 45.7s 45.1s 45.6s 46.3s 47.1s

Times for minikube ingress: 30.3s 30.3s 32.7s 31.2s 31.2s
Times for minikube (PR 13121) ingress: 29.8s 31.2s 31.4s 31.3s 32.2s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13121) |
+----------------+----------+---------------------+
| minikube start | 22.6s    | 22.2s               |
| enable ingress | 25.9s    | 27.7s               |
+----------------+----------+---------------------+

Times for minikube start: 22.2s 22.8s 23.8s 21.8s 22.6s
Times for minikube (PR 13121) start: 22.4s 22.5s 22.0s 22.6s 21.4s

Times for minikube ingress: 25.9s 26.0s 25.9s 25.9s 25.9s
Times for minikube (PR 13121) ingress: 25.9s 26.4s 25.9s 33.9s 26.5s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13121) |
+----------------+----------+---------------------+
| minikube start | 40.7s    | 42.3s               |
| enable ingress | 24.7s    | 32.3s               |
+----------------+----------+---------------------+

Times for minikube start: 36.9s 42.4s 37.4s 42.5s 44.3s
Times for minikube (PR 13121) start: 45.4s 41.9s 41.4s 42.1s 40.8s

Times for minikube ingress: 22.7s 19.4s 20.2s 32.4s 28.9s
Times for minikube (PR 13121) ingress: 28.9s 62.5s 32.4s 18.9s 18.9s

minikube-pr-bot · 2021-12-08T03:38:55Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Hyperkit_macOS	TestHyperKitDriverInstallOrUpdate (gopogh)	0.00 (chart)
KVM_Linux_containerd	TestMultiNode/serial/RestartMultiNode (gopogh)	0.00 (chart)
KVM_Linux	TestNetworkPlugins/group/calico/Start (gopogh)	0.00 (chart)
KVM_Linux	TestStartStop/group/newest-cni/serial/EnableAddonWhileActive (gopogh)	0.00 (chart)
KVM_Linux	TestStartStop/group/newest-cni/serial/FirstStart (gopogh)	0.00 (chart)
Docker_Linux_containerd	TestRunningBinaryUpgrade (gopogh)	0.65 (chart)
KVM_Linux	TestNetworkPlugins/group/kindnet/Start (gopogh)	1.61 (chart)
KVM_Linux_containerd	TestRunningBinaryUpgrade (gopogh)	3.19 (chart)
Docker_Windows	TestPause/serial/Pause (gopogh)	9.18 (chart)
Docker_Windows	TestStartStop/group/embed-certs/serial/AddonExistsAfterStop (gopogh)	9.90 (chart)
Docker_Windows	TestStartStop/group/embed-certs/serial/SecondStart (gopogh)	9.90 (chart)
Docker_Windows	TestStartStop/group/embed-certs/serial/UserAppExistsAfterStop (gopogh)	9.90 (chart)
Docker_Windows	TestStartStop/group/embed-certs/serial/VerifyKubernetesImages (gopogh)	9.90 (chart)
Docker_Windows	TestPause/serial/PauseAgain (gopogh)	14.29 (chart)
Docker_Windows	TestStartStop/group/embed-certs/serial/Pause (gopogh)	27.72 (chart)
Docker_Windows	TestNetworkPlugins/group/bridge/DNS (gopogh)	37.50 (chart)
Docker_Windows	TestNetworkPlugins/group/kubenet/Start (gopogh)	41.58 (chart)
Docker_Windows	TestNetworkPlugins/group/custom-weave/Start (gopogh)	66.34 (chart)
Hyper-V_Windows	TestNoKubernetes/serial/StartWithK8s (gopogh)	69.23 (chart)
Docker_Windows	TestFunctional/parallel/DryRun (gopogh)	84.16 (chart)
Docker_Windows	TestFunctional/parallel/ImageCommands/ImageLoadFromFile (gopogh)	84.16 (chart)
Docker_Windows	TestFunctional/parallel/InternationalLanguage (gopogh)	84.16 (chart)
Docker_Windows	TestInsufficientStorage (gopogh)	86.14 (chart)
Docker_Windows	TestFunctional/parallel/StatusCmd (gopogh)	87.13 (chart)
Docker_Windows	TestNoKubernetes/serial/StartWithStopK8s (gopogh)	88.89 (chart)
Docker_Windows	TestPause/serial/VerifyStatus (gopogh)	94.90 (chart)
Docker_Windows	TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh)	97.03 (chart)
Docker_Windows	TestMultiNode/serial/CopyFile (gopogh)	100.00 (chart)
Docker_Windows	TestNetworkPlugins/group/cilium/Start (gopogh)	100.00 (chart)
Docker_Windows	TestNoKubernetes/serial/StartWithK8s (gopogh)	100.00 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

pkg/minikube/bootstrapper/kubeadm/kubeadm.go

klaases

I am curious if the infos that have an error should be errors.

medyagh · 2021-12-10T20:50:16Z

the test for docker_linux is not come yet

medyagh · 2021-12-10T20:50:24Z

/retest-this-please

minikube-pr-bot · 2021-12-11T00:54:14Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_Linux	TestNetworkPlugins/group/kubenet/Start (gopogh)	0.00 (chart)
Docker_Linux	TestRunningBinaryUpgrade (gopogh)	0.00 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/DeployApp (gopogh)	0.00 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/EnableAddonAfterStop (gopogh)	0.00 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/EnableAddonWhileActive (gopogh)	0.00 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/FirstStart (gopogh)	0.00 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/Stop (gopogh)	0.00 (chart)
Docker_Linux_containerd	TestRunningBinaryUpgrade (gopogh)	0.61 (chart)
KVM_Linux_containerd	TestKubernetesUpgrade (gopogh)	0.83 (chart)
KVM_Linux_containerd	TestNetworkPlugins/group/custom-weave/Start (gopogh)	0.83 (chart)
KVM_Linux	TestFunctional/parallel/MountCmd/specific-port (gopogh)	1.08 (chart)
KVM_Linux_containerd	TestRunningBinaryUpgrade (gopogh)	4.13 (chart)
Docker_Linux_containerd	TestPause/serial/Pause (gopogh)	4.38 (chart)
Docker_Linux_containerd	TestPause/serial/VerifyStatus (gopogh)	4.38 (chart)
Docker_macOS	TestRunningBinaryUpgrade (gopogh)	6.71 (chart)
Docker_Linux_containerd	TestPause/serial/PauseAgain (gopogh)	7.50 (chart)
Docker_Linux	TestStartStop/group/old-k8s-version/serial/Pause (gopogh)	8.43 (chart)
Docker_Cloud_Shell	TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh)	54.20 (chart)
Docker_Cloud_Shell	TestFunctional/parallel/TunnelCmd/serial/WaitService/IngressIP (gopogh)	54.20 (chart)
Docker_macOS	TestNetworkPlugins/group/kubenet/DNS (gopogh)	65.15 (chart)
Docker_macOS	TestNetworkPlugins/group/kindnet/Start (gopogh)	72.27 (chart)
Docker_macOS	TestNetworkPlugins/group/calico/Start (gopogh)	73.11 (chart)
Docker_macOS	TestNetworkPlugins/group/bridge/DNS (gopogh)	73.26 (chart)
Docker_macOS	TestNetworkPlugins/group/enable-default-cni/DNS (gopogh)	76.04 (chart)
Docker_macOS	TestDownloadOnly/v1.16.0/preload-exists (gopogh)	100.00 (chart)

To see the flake rates of all tests by environment, click here.

minikube-pr-bot · 2021-12-11T01:05:45Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13121) |
+----------------+----------+---------------------+
| minikube start | 47.0s    | 46.5s               |
| enable ingress | 30.9s    | 30.4s               |
+----------------+----------+---------------------+

Times for minikube start: 49.1s 46.8s 46.8s 46.1s 46.1s
Times for minikube (PR 13121) start: 46.4s 46.8s 46.6s 45.8s 46.9s

Times for minikube ingress: 32.3s 30.2s 30.7s 30.3s 30.7s
Times for minikube (PR 13121) ingress: 30.2s 30.8s 31.3s 29.4s 30.3s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13121) |
+----------------+----------+---------------------+
| minikube start | 22.9s    | 22.4s               |
| enable ingress | 27.0s    | 25.9s               |
+----------------+----------+---------------------+

Times for minikube ingress: 26.9s 29.9s 25.5s 26.4s 26.4s
Times for minikube (PR 13121) ingress: 26.4s 26.4s 25.9s 25.4s 25.4s

Times for minikube start: 24.5s 22.3s 22.8s 22.4s 22.6s
Times for minikube (PR 13121) start: 23.2s 22.5s 22.8s 21.6s 21.8s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13121) |
+----------------+----------+---------------------+
| minikube start | 39.5s    | 41.9s               |
| enable ingress | 29.3s    | 31.3s               |
+----------------+----------+---------------------+

Times for minikube start: 29.3s 42.7s 41.6s 42.4s 41.2s
Times for minikube (PR 13121) start: 41.2s 41.9s 42.1s 42.4s 42.0s

Times for minikube ingress: 30.9s 31.4s 31.4s 31.4s 21.4s
Times for minikube (PR 13121) ingress: 30.9s 31.5s 31.4s 31.4s 31.4s

pkg/minikube/bootstrapper/kubeadm/kubeadm.go

klaases

LGTM

minikube-pr-bot · 2021-12-14T21:10:06Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 13121) |
+----------------+----------+---------------------+
| minikube start | 47.6s    | 46.0s               |
| enable ingress | 28.6s    | 28.7s               |
+----------------+----------+---------------------+

Times for minikube start: 46.3s 44.8s 51.0s 45.3s 50.3s
Times for minikube (PR 13121) start: 43.8s 51.2s 45.0s 44.3s 45.9s

Times for minikube ingress: 29.1s 29.6s 25.6s 29.1s 29.6s
Times for minikube (PR 13121) ingress: 27.1s 29.1s 29.1s 28.6s 29.6s

docker driver with docker runtime
error collecting results for docker driver: timing run 0 with minikube: timing cmd: [out/minikube addons enable ingress]: waiting for minikube: exit status 10
docker driver with containerd runtime
error collecting results for docker driver: timing run 0 with minikube: timing cmd: [out/minikube addons enable ingress]: waiting for minikube: exit status 10

minikube-pr-bot · 2021-12-15T03:21:38Z

These are the flake rates of all failed tests.

Environment	Failed Tests	Flake Rate (%)
Docker_macOS	TestAddons/parallel/HelmTiller (gopogh)	0.00 (chart)
Hyperkit_macOS	TestNoKubernetes/serial/Stop (gopogh)	0.00 (chart)
none_Linux	TestStoppedBinaryUpgrade/Upgrade (gopogh)	0.83 (chart)
Docker_macOS	TestFunctional/parallel/MySQL (gopogh)	0.85 (chart)
Docker_macOS	TestSkaffold (gopogh)	0.85 (chart)
Hyperkit_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	1.10 (chart)
Docker_macOS	TestFunctional/parallel/TunnelCmd/serial/AccessDirect (gopogh)	1.71 (chart)
Docker_macOS	TestFunctional/parallel/TunnelCmd/serial/WaitService/Setup (gopogh)	1.71 (chart)
Hyperkit_macOS	TestFunctional/parallel/MountCmd/any-port (gopogh)	2.20 (chart)
Hyperkit_macOS	TestFunctional/parallel/PersistentVolumeClaim (gopogh)	2.20 (chart)
Docker_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddonActivation (gopogh)	2.56 (chart)
Hyperkit_macOS	TestFunctional/parallel/ImageCommands/ImageBuild (gopogh)	3.30 (chart)
Hyperkit_macOS	TestIngressAddonLegacy/serial/ValidateIngressAddons (gopogh)	3.30 (chart)
Docker_macOS	TestAddons/parallel/CSI (gopogh)	4.27 (chart)
Docker_macOS	TestAddons/serial/GCPAuth (gopogh)	4.27 (chart)
Docker_macOS	TestFunctional/parallel/PersistentVolumeClaim (gopogh)	4.27 (chart)
Hyperkit_macOS	TestNetworkPlugins/group/kindnet/Start (gopogh)	4.40 (chart)
Docker_macOS	TestMissingContainerUpgrade (gopogh)	5.98 (chart)
Docker_macOS	TestFunctional/parallel/ImageCommands/ImageBuild (gopogh)	6.84 (chart)
Docker_macOS	TestFunctional/parallel/MountCmd/any-port (gopogh)	6.84 (chart)
Hyperkit_macOS	TestStartStop/group/no-preload/serial/AddonExistsAfterStop (gopogh)	8.79 (chart)
Hyperkit_macOS	TestPreload (gopogh)	9.89 (chart)
Docker_Linux_containerd	TestStoppedBinaryUpgrade/MinikubeLogs (gopogh)	10.66 (chart)
Docker_Linux_containerd	TestStoppedBinaryUpgrade/Upgrade (gopogh)	10.66 (chart)
Hyperkit_macOS	TestMultiNode/serial/DeployApp2Nodes (gopogh)	10.99 (chart)
Hyperkit_macOS	TestMultiNode/serial/PingHostFrom2Pods (gopogh)	10.99 (chart)
Docker_macOS	TestFunctional/parallel/DashboardCmd (gopogh)	11.11 (chart)
Docker_macOS	TestNoKubernetes/serial/StartWithK8s (gopogh)	11.48 (chart)
Docker_macOS	TestNoKubernetes/serial/StartWithStopK8s (gopogh)	11.48 (chart)
KVM_Linux_containerd	TestRunningBinaryUpgrade (gopogh)	11.48 (chart)
More tests...	Continued...

Too many tests failed - See test logs for more details.

To see the flake rates of all tests by environment, click here.

k8s-ci-robot · 2021-12-15T22:09:08Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: klaases, sharifelgamal, spowelljr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sharifelgamal,spowelljr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 7, 2021

k8s-ci-robot requested review from afbjorklund and blueelvis December 7, 2021 22:57

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 7, 2021

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Dec 8, 2021

klaases reviewed Dec 10, 2021

View reviewed changes

pkg/minikube/bootstrapper/kubeadm/kubeadm.go Outdated Show resolved Hide resolved

klaases reviewed Dec 10, 2021

View reviewed changes

pkg/minikube/bootstrapper/kubeadm/kubeadm.go Outdated Show resolved Hide resolved

klaases reviewed Dec 10, 2021

View reviewed changes

pkg/minikube/bootstrapper/kubeadm/kubeadm.go Outdated Show resolved Hide resolved

klaases reviewed Dec 10, 2021

View reviewed changes

pkg/minikube/bootstrapper/kubeadm/kubeadm.go Outdated Show resolved Hide resolved

klaases reviewed Dec 10, 2021

View reviewed changes

pkg/minikube/bootstrapper/kubeadm/kubeadm.go Outdated Show resolved Hide resolved

klaases reviewed Dec 10, 2021

View reviewed changes

pkg/minikube/bootstrapper/kubeadm/kubeadm.go Outdated Show resolved Hide resolved

klaases reviewed Dec 10, 2021

View reviewed changes

pkg/minikube/bootstrapper/kubeadm/kubeadm.go Outdated Show resolved Hide resolved

klaases reviewed Dec 10, 2021

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 14, 2021

spowelljr added 3 commits December 14, 2021 11:15

delete and init kubeadm on subsequent starts

74831f7

addressed comments

e91dc21

remove comment

90dc83d

spowelljr force-pushed the fixSecondStart branch from 9137eb9 to 90dc83d Compare December 14, 2021 19:16

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 14, 2021

klaases reviewed Dec 14, 2021

View reviewed changes

pkg/minikube/bootstrapper/kubeadm/kubeadm.go Outdated Show resolved Hide resolved

klaases approved these changes Dec 14, 2021

View reviewed changes

change milliseconds to seconds

78aef25

sharifelgamal approved these changes Dec 15, 2021

View reviewed changes

spowelljr merged commit 1d43372 into kubernetes:master Dec 15, 2021

spowelljr deleted the fixSecondStart branch December 15, 2021 22:28

This was referenced Jan 27, 2022

Error while configuring RBAC rules #13431

Closed

Revert "Delete and init kubeadm on subsequent starts" #13506

Merged

Pods not started after minikube restart (minikube v1.25.X, docker driver) #13503

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Delete and init kubeadm on subsequent starts #13121

Delete and init kubeadm on subsequent starts #13121

spowelljr commented Dec 7, 2021 •

edited

Loading

spowelljr commented Dec 8, 2021

minikube-pr-bot commented Dec 8, 2021

minikube-pr-bot commented Dec 8, 2021

klaases left a comment

medyagh commented Dec 10, 2021

medyagh commented Dec 10, 2021

minikube-pr-bot commented Dec 11, 2021

minikube-pr-bot commented Dec 11, 2021

klaases left a comment

minikube-pr-bot commented Dec 14, 2021

minikube-pr-bot commented Dec 15, 2021

k8s-ci-robot commented Dec 15, 2021

Delete and init kubeadm on subsequent starts #13121

Delete and init kubeadm on subsequent starts #13121

Conversation

spowelljr commented Dec 7, 2021 • edited Loading

spowelljr commented Dec 8, 2021

minikube-pr-bot commented Dec 8, 2021

minikube-pr-bot commented Dec 8, 2021

klaases left a comment

Choose a reason for hiding this comment

medyagh commented Dec 10, 2021

medyagh commented Dec 10, 2021

minikube-pr-bot commented Dec 11, 2021

minikube-pr-bot commented Dec 11, 2021

klaases left a comment

Choose a reason for hiding this comment

minikube-pr-bot commented Dec 14, 2021

minikube-pr-bot commented Dec 15, 2021

k8s-ci-robot commented Dec 15, 2021

spowelljr commented Dec 7, 2021 •

edited

Loading