Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

multi node docker ubuntu failures FreshStart #9096

Closed
medyagh opened this issue Aug 26, 2020 · 5 comments
Closed

multi node docker ubuntu failures FreshStart #9096

medyagh opened this issue Aug 26, 2020 · 5 comments
Labels
co/multinode Issues related to multinode clusters kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@medyagh
Copy link
Member

medyagh commented Aug 26, 2020 •
edited
Loading

as seen on all gh action tests on Head


2020-08-26T04:58:56.9072888Z === RUN   TestMultiNode/serial/FreshStart2Nodes
2020-08-26T04:58:56.9075646Z     multinode_test.go:69: (dbg) Run:  ./minikube-linux-amd64 start -p multinode-20200826045856-2663 --wait=true --memory=2200 --nodes=2 --driver=docker
2020-08-26T05:06:10.7895409Z     multinode_test.go:69: (dbg) Non-zero exit: ./minikube-linux-amd64 start -p multinode-20200826045856-2663 --wait=true --memory=2200 --nodes=2 --driver=docker: exit status 70 (7m13.881071933s)
2020-08-26T05:06:10.7907618Z         
2020-08-26T05:06:10.7936580Z         -- stdout --
2020-08-26T05:06:10.7937350Z         	* [multinode-20200826045856-2663] minikube v1.12.3 on Ubuntu 18.04
2020-08-26T05:06:10.7937874Z         	  - MINIKUBE_HOME=/home/runner/work/minikube/minikube/minikube_binaries/testhome
2020-08-26T05:06:10.7938466Z         	  - KUBECONFIG=/home/runner/work/minikube/minikube/minikube_binaries/testhome/kubeconfig
2020-08-26T05:06:10.7938659Z         	* Using the docker driver based on user configuration
2020-08-26T05:06:10.7939572Z         	* Starting control plane node multinode-20200826045856-2663 in cluster multinode-20200826045856-2663
2020-08-26T05:06:10.7939781Z         	* Pulling base image ...
2020-08-26T05:06:10.7940039Z         	* Downloading Kubernetes v1.18.3 preload ...
2020-08-26T05:06:10.7940310Z         	* Creating docker container (CPUs=2, Memory=2200MB) ...
2020-08-26T05:06:10.7945045Z         	* Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
2020-08-26T05:06:10.8160812Z         	* Verifying Kubernetes components...
2020-08-26T05:06:10.8162490Z         	* Enabled addons: default-storageclass, storage-provisioner
2020-08-26T05:06:10.8165907Z         	
2020-08-26T05:06:10.8166419Z         	* To track progress on multi-node clusters, see https://github.com/kubernetes/minikube/issues/7538.
2020-08-26T05:06:10.8166587Z         	
2020-08-26T05:06:10.8167380Z         	* Starting node multinode-20200826045856-2663-m02 in cluster multinode-20200826045856-2663
2020-08-26T05:06:10.8167562Z         	* Creating docker container (CPUs=2, Memory=2200MB) ...
2020-08-26T05:06:10.8167720Z         	* Found network options:
2020-08-26T05:06:10.8168037Z         	  - NO_PROXY=192.168.39.2
2020-08-26T05:06:10.8168331Z         	* Preparing Kubernetes v1.18.3 on Docker 19.03.8 ...
2020-08-26T05:06:10.8168995Z         	  - env NO_PROXY=192.168.39.2
2020-08-26T05:06:10.8169153Z         
2020-08-26T05:06:10.8169568Z         -- /stdout --
2020-08-26T05:06:10.8169815Z         ** stderr ** 
2020-08-26T05:06:10.8172591Z         	    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 27.00 MiB     > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 67.98 MiB     > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 72.00 MiB     > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 72.00 MiB     > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 72.00 MiB     > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 72.00 MiB     > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 72.00 MiB     > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 90.16 MiB     > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 118.00 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 145.31 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 175.70 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 207.00 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 236.80 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 265.62 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 292.77 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 325.47 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 355.52 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 396.61 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 435.51 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 467.38 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 502.90 MiB    > preloaded-images-k8s-v5-v1.18.3-docker-overlay2-amd64.tar.lz4: 510.91 MiB! Multi-node clusters are currently experimental and might exhibit unintended behavior.
2020-08-26T05:06:10.8173269Z         	* 
2020-08-26T05:06:10.8174098Z         	X failed to start node: adding node: joining cluster: joining cp: cmd failed: sudo env PATH=/var/lib/minikube/binaries/v1.18.3:$PATH kubeadm join control-plane.minikube.internal:8443 --token 2lrdvr.0h89vixoujkl5pmg     --discovery-token-ca-cert-hash sha256:40abf7f9670033945ec051f89536289f8e49d56e412f4fccbeb5a1a7c0f35336 --ignore-preflight-errors=all --node-name=multinode-20200826045856-2663-m02
2020-08-26T05:06:10.8174768Z         	-- stdout --
2020-08-26T05:06:10.8175092Z         	[preflight] Running pre-flight checks
2020-08-26T05:06:10.8175280Z         	[preflight] The system verification failed. Printing the output from the verification:
2020-08-26T05:06:10.8175777Z         	�[0;37mKERNEL_VERSION�[0m: �[0;32m5.3.0-1034-azure�[0m
2020-08-26T05:06:10.8176251Z         	�[0;37mDOCKER_VERSION�[0m: �[0;32m19.03.8�[0m
2020-08-26T05:06:10.8176626Z         	�[0;37mDOCKER_GRAPH_DRIVER�[0m: �[0;32moverlay2�[0m
2020-08-26T05:06:10.8177068Z         	�[0;37mOS�[0m: �[0;32mLinux�[0m
2020-08-26T05:06:10.8177525Z         	�[0;37mCGROUPS_CPU�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8178235Z         	�[0;37mCGROUPS_CPUACCT�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8178730Z         	�[0;37mCGROUPS_CPUSET�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8179191Z         	�[0;37mCGROUPS_DEVICES�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8179650Z         	�[0;37mCGROUPS_FREEZER�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8180106Z         	�[0;37mCGROUPS_MEMORY�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8180353Z         	
2020-08-26T05:06:10.8180780Z         	-- /stdout --
2020-08-26T05:06:10.8181062Z         	** stderr ** 
2020-08-26T05:06:10.8182545Z         	W0826 05:01:02.921276     647 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
2020-08-26T05:06:10.8183001Z         		[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
2020-08-26T05:06:10.8183579Z         		[WARNING Swap]: running with swap on is not supported. Please disable swap
2020-08-26T05:06:10.8184375Z         		[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.3.0-1034-azure\n", err: exit status 1
2020-08-26T05:06:10.8184966Z         		[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
2020-08-26T05:06:10.8185853Z         	error execution phase preflight: couldn't validate the identity of the API Server: Get https://control-plane.minikube.internal:8443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-08-26T05:06:10.8186464Z         	To see the stack trace of this error execute with --v=5 or higher
2020-08-26T05:06:10.8186748Z         	
2020-08-26T05:06:10.8186925Z         	** /stderr **
2020-08-26T05:06:10.8187752Z         	: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.18.3:$PATH kubeadm join control-plane.minikube.internal:8443 --token 2lrdvr.0h89vixoujkl5pmg     --discovery-token-ca-cert-hash sha256:40abf7f9670033945ec051f89536289f8e49d56e412f4fccbeb5a1a7c0f35336 --ignore-preflight-errors=all --node-name=multinode-20200826045856-2663-m02": Process exited with status 1
2020-08-26T05:06:10.8188137Z         	stdout:
2020-08-26T05:06:10.8188496Z         	[preflight] Running pre-flight checks
2020-08-26T05:06:10.8188785Z         	[preflight] The system verification failed. Printing the output from the verification:
2020-08-26T05:06:10.8189563Z         	�[0;37mKERNEL_VERSION�[0m: �[0;32m5.3.0-1034-azure�[0m
2020-08-26T05:06:10.8190060Z         	�[0;37mDOCKER_VERSION�[0m: �[0;32m19.03.8�[0m
2020-08-26T05:06:10.8190534Z         	�[0;37mDOCKER_GRAPH_DRIVER�[0m: �[0;32moverlay2�[0m
2020-08-26T05:06:10.8190970Z         	�[0;37mOS�[0m: �[0;32mLinux�[0m
2020-08-26T05:06:10.8191407Z         	�[0;37mCGROUPS_CPU�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8191861Z         	�[0;37mCGROUPS_CPUACCT�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8192453Z         	�[0;37mCGROUPS_CPUSET�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8192906Z         	�[0;37mCGROUPS_DEVICES�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8193357Z         	�[0;37mCGROUPS_FREEZER�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8194246Z         	�[0;37mCGROUPS_MEMORY�[0m: �[0;32menabled�[0m
2020-08-26T05:06:10.8194540Z         	
2020-08-26T05:06:10.8194717Z         	stderr:
2020-08-26T05:06:10.8195218Z         	W0826 05:01:02.921276     647 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.
2020-08-26T05:06:10.8195590Z         		[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
2020-08-26T05:06:10.8195932Z         		[WARNING Swap]: running with swap on is not supported. Please disable swap
2020-08-26T05:06:10.8196530Z         		[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.3.0-1034-azure\n", err: exit status 1
2020-08-26T05:06:10.8197093Z         		[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
2020-08-26T05:06:10.8198031Z         	error execution phase preflight: couldn't validate the identity of the API Server: Get https://control-plane.minikube.internal:8443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-08-26T05:06:10.8198660Z         	To see the stack trace of this error execute with --v=5 or higher
2020-08-26T05:06:10.8198936Z         	
2020-08-26T05:06:10.8199088Z         	* 
2020-08-26T05:06:10.8199267Z         	* minikube is exiting due to an error. If the above message is not useful, open an issue:
2020-08-26T05:06:10.8199683Z         	  - https://github.com/kubernetes/minikube/issues/new/choose
2020-08-26T05:06:10.8199965Z         
2020-08-26T05:06:10.8200127Z         ** /stderr **
2020-08-26T05:06:10.8200658Z     multinode_test.go:71: failed to start cluster. args "./minikube-linux-amd64 start -p multinode-20200826045856-2663 --wait=true --memory=2200 --nodes=2 --driver=docker" : exit status 70
2020-08-26T05:06:10.8201052Z     helpers_test.go:215: -----------------------post-mortem--------------------------------
2020-08-26T05:06:10.8201586Z     helpers_test.go:223: ======>  post-mortem[TestMultiNode/serial/FreshStart2Nodes]: docker inspect <======
2020-08-26T05:06:10.8202285Z     helpers_test.go:224: (dbg) Run:  docker inspect multinode-20200826045856-2663
2020-08-26T05:06:10.8554665Z     helpers_test.go:228: (dbg) docker inspect multinode-20200826045856-2663:
2020-08-26T05:06:10.8554885Z         
2020-08-26T05:06:10.8555185Z         -- stdout --
2020-08-26T05:06:10.8555334Z         	[
2020-08-26T05:06:10.8555487Z         	    {
2020-08-26T05:06:10.8555658Z         	        "Id": "d26688304ef1fd85ae8c2a746752458d873139fed10cd9c078833b72c1cc75f6",
2020-08-26T05:06:10.8556039Z         	        "Created": "2020-08-26T04:59:37.721360456Z",
2020-08-26T05:06:10.8556210Z         	        "Path": "/usr/local/bin/entrypoint",
2020-08-26T05:06:10.8556371Z         	        "Args": [
2020-08-26T05:06:10.8556970Z         	            "/sbin/init"
2020-08-26T05:06:10.8557162Z         	        ],
2020-08-26T05:06:10.8557336Z         	        "State": {
2020-08-26T05:06:10.8557530Z         	            "Status": "running",
2020-08-26T05:06:10.8557864Z         	            "Running": true,
2020-08-26T05:06:10.8558036Z         	            "Paused": false,
2020-08-26T05:06:10.8558226Z         	            "Restarting": false,
2020-08-26T05:06:10.8562951Z         	            "OOMKilled": false,
2020-08-26T05:06:10.8563341Z         	            "Dead": false,
2020-08-26T05:06:10.8563868Z         	            "Pid": 3498,
2020-08-26T05:06:10.8564611Z         	            "ExitCode": 0,
2020-08-26T05:06:10.8564903Z         	            "Error": "",
2020-08-26T05:06:10.8565883Z         	            "StartedAt": "2020-08-26T04:59:38.736590179Z",
2020-08-26T05:06:10.8566291Z         	            "FinishedAt": "0001-01-01T00:00:00Z"
2020-08-26T05:06:10.8566462Z         	        },
2020-08-26T05:06:10.8566665Z         	        "Image": "sha256:db6a3f9d436891c483cb3ce322c6ecb7952d29e37e2aa5028f10bcd32f3e5c54",
2020-08-26T05:06:10.8566888Z         	        "ResolvConfPath": "/var/lib/docker/containers/d26688304ef1fd85ae8c2a746752458d873139fed10cd9c078833b72c1cc75f6/resolv.conf",
2020-08-26T05:06:10.8567110Z         	        "HostnamePath": "/var/lib/docker/containers/d26688304ef1fd85ae8c2a746752458d873139fed10cd9c078833b72c1cc75f6/hostname",
2020-08-26T05:06:10.8567327Z         	        "HostsPath": "/var/lib/docker/containers/d26688304ef1fd85ae8c2a746752458d873139fed10cd9c078833b72c1cc75f6/hosts",
2020-08-26T05:06:10.8568110Z         	        "LogPath": "/var/lib/docker/containers/d26688304ef1fd85ae8c2a746752458d873139fed10cd9c078833b72c1cc75f6/d26688304ef1fd85ae8c2a746752458d873139fed10cd9c078833b72c1cc75f6-json.log",
2020-08-26T05:06:10.8568632Z         	        "Name": "/multinode-20200826045856-2663",
2020-08-26T05:06:10.8568903Z         	        "RestartCount": 0,
2020-08-26T05:06:10.8569257Z         	        "Driver": "overlay2",
2020-08-26T05:06:10.8569541Z         	        "Platform": "linux",
2020-08-26T05:06:10.8569734Z         	        "MountLabel": "",
2020-08-26T05:06:10.8614930Z         	        "ProcessLabel": "",
2020-08-26T05:06:10.8615506Z         	        "AppArmorProfile": "unconfined",
2020-08-26T05:06:10.8615801Z         	        "ExecIDs": null,
2020-08-26T05:06:10.8616087Z         	        "HostConfig": {
2020-08-26T05:06:10.8616361Z         	            "Binds": [
2020-08-26T05:06:10.8617120Z         	                "multinode-20200826045856-2663:/var",
2020-08-26T05:06:10.8617450Z         	                "/lib/modules:/lib/modules:ro"
2020-08-26T05:06:10.8617769Z         	            ],
2020-08-26T05:06:10.8618225Z         	            "ContainerIDFile": "",
2020-08-26T05:06:10.8618498Z         	            "LogConfig": {
2020-08-26T05:06:10.8618978Z         	                "Type": "json-file",
2020-08-26T05:06:10.8619279Z         	                "Config": {}
2020-08-26T05:06:10.8619566Z         	            },
2020-08-26T05:06:10.8620058Z         	            "NetworkMode": "multinode-20200826045856-2663",
2020-08-26T05:06:10.8620363Z         	            "PortBindings": {
2020-08-26T05:06:10.8620626Z         	                "22/tcp": [
2020-08-26T05:06:10.8620911Z         	                    {
2020-08-26T05:06:10.8621205Z         	                        "HostIp": "127.0.0.1",
2020-08-26T05:06:10.8621508Z         	                        "HostPort": ""
2020-08-26T05:06:10.8621794Z         	                    }
2020-08-26T05:06:10.8622066Z         	                ],
2020-08-26T05:06:10.8622342Z         	                "2376/tcp": [
2020-08-26T05:06:10.8622631Z         	                    {
2020-08-26T05:06:10.8622965Z         	                        "HostIp": "127.0.0.1",
2020-08-26T05:06:10.8623266Z         	                        "HostPort": ""
2020-08-26T05:06:10.8623547Z         	                    }
2020-08-26T05:06:10.8623815Z         	                ],
2020-08-26T05:06:10.8624080Z         	                "5000/tcp": [
2020-08-26T05:06:10.8624359Z         	                    {
2020-08-26T05:06:10.8624656Z         	                        "HostIp": "127.0.0.1",
2020-08-26T05:06:10.8624955Z         	                        "HostPort": ""
2020-08-26T05:06:10.8625237Z         	                    }
2020-08-26T05:06:10.8625515Z         	                ],
2020-08-26T05:06:10.8625798Z         	                "8443/tcp": [
2020-08-26T05:06:10.8626072Z         	                    {
2020-08-26T05:06:10.8626358Z         	                        "HostIp": "127.0.0.1",
2020-08-26T05:06:10.8626877Z         	                        "HostPort": ""
2020-08-26T05:06:10.8627158Z         	                    }
2020-08-26T05:06:10.8627431Z         	                ]
2020-08-26T05:06:10.8627681Z         	            },
2020-08-26T05:06:10.8627953Z         	            "RestartPolicy": {
2020-08-26T05:06:10.8628227Z         	                "Name": "no",
2020-08-26T05:06:10.8628512Z         	                "MaximumRetryCount": 0
2020-08-26T05:06:10.8628777Z         	            },
2020-08-26T05:06:10.8629046Z         	            "AutoRemove": false,
2020-08-26T05:06:10.8629322Z         	            "VolumeDriver": "",
2020-08-26T05:06:10.8629950Z         	            "VolumesFrom": null,
2020-08-26T05:06:10.8632740Z         	            "CapAdd": null,
2020-08-26T05:06:10.8632918Z         	            "CapDrop": null,
2020-08-26T05:06:10.8633089Z         	            "Capabilities": null,
2020-08-26T05:06:10.8633254Z         	            "Dns": [],
2020-08-26T05:06:10.8633422Z         	            "DnsOptions": [],
2020-08-26T05:06:10.8633592Z         	            "DnsSearch": [],
2020-08-26T05:06:10.8634104Z         	            "ExtraHosts": null,
2020-08-26T05:06:10.8634253Z         	            "GroupAdd": null,
2020-08-26T05:06:10.8634421Z         	            "IpcMode": "private",
2020-08-26T05:06:10.8634590Z         	            "Cgroup": "",
2020-08-26T05:06:10.8634756Z         	            "Links": null,
2020-08-26T05:06:10.8635049Z         	            "OomScoreAdj": 0,
2020-08-26T05:06:10.8635221Z         	            "PidMode": "",
2020-08-26T05:06:10.8635391Z         	            "Privileged": true,
2020-08-26T05:06:10.8635560Z         	            "PublishAllPorts": false,
2020-08-26T05:06:10.8635708Z         	            "ReadonlyRootfs": false,
2020-08-26T05:06:10.8635873Z         	            "SecurityOpt": [
2020-08-26T05:06:10.8636044Z         	                "seccomp=unconfined",
2020-08-26T05:06:10.8636215Z         	                "apparmor=unconfined",
2020-08-26T05:06:10.8636381Z         	                "label=disable"
2020-08-26T05:06:10.8636545Z         	            ],
2020-08-26T05:06:10.8636703Z         	            "Tmpfs": {
2020-08-26T05:06:10.8636865Z         	                "/run": "",
2020-08-26T05:06:10.8637012Z         	                "/tmp": ""
2020-08-26T05:06:10.8637169Z         	            },
2020-08-26T05:06:10.8637334Z         	            "UTSMode": "",
2020-08-26T05:06:10.8637506Z         	            "UsernsMode": "",
2020-08-26T05:06:10.8637671Z         	            "ShmSize": 67108864,
2020-08-26T05:06:10.8637836Z         	            "Runtime": "runc",
2020-08-26T05:06:10.8638000Z         	            "ConsoleSize": [
2020-08-26T05:06:10.8638146Z         	                0,
2020-08-26T05:06:10.8638311Z         	                0
2020-08-26T05:06:10.8638471Z         	            ],
2020-08-26T05:06:10.8638625Z         	            "Isolation": "",
2020-08-26T05:06:10.8638785Z         	            "CpuShares": 0,
2020-08-26T05:06:10.8638951Z         	            "Memory": 2306867200,
2020-08-26T05:06:10.8639118Z         	            "NanoCpus": 2000000000,
2020-08-26T05:06:10.8639287Z         	            "CgroupParent": "",
2020-08-26T05:06:10.8639431Z         	            "BlkioWeight": 0,
2020-08-26T05:06:10.8639598Z         	            "BlkioWeightDevice": [],
2020-08-26T05:06:10.8639761Z         	            "BlkioDeviceReadBps": null,
2020-08-26T05:06:10.8639937Z         	            "BlkioDeviceWriteBps": null,
2020-08-26T05:06:10.8640105Z         	            "BlkioDeviceReadIOps": null,
2020-08-26T05:06:10.8640274Z         	            "BlkioDeviceWriteIOps": null,
2020-08-26T05:06:10.8640440Z         	            "CpuPeriod": 0,
2020-08-26T05:06:10.8640604Z         	            "CpuQuota": 0,
2020-08-26T05:06:10.8640751Z         	            "CpuRealtimePeriod": 0,
2020-08-26T05:06:10.8640921Z         	            "CpuRealtimeRuntime": 0,
2020-08-26T05:06:10.8641088Z         	            "CpusetCpus": "",
2020-08-26T05:06:10.8641251Z         	            "CpusetMems": "",
2020-08-26T05:06:10.8641478Z         	            "Devices": [],
2020-08-26T05:06:10.8641646Z         	            "DeviceCgroupRules": null,
2020-08-26T05:06:10.8641811Z         	            "DeviceRequests": null,
2020-08-26T05:06:10.8641978Z         	            "KernelMemory": 0,
2020-08-26T05:06:10.8642125Z         	            "KernelMemoryTCP": 0,
2020-08-26T05:06:10.8642293Z         	            "MemoryReservation": 0,
2020-08-26T05:06:10.8642461Z         	            "MemorySwap": 4613734400,
2020-08-26T05:06:10.8642631Z         	            "MemorySwappiness": null,
2020-08-26T05:06:10.8642798Z         	            "OomKillDisable": false,
2020-08-26T05:06:10.8642964Z         	            "PidsLimit": null,
2020-08-26T05:06:10.8643122Z         	            "Ulimits": null,
2020-08-26T05:06:10.8643285Z         	            "CpuCount": 0,
2020-08-26T05:06:10.8643429Z         	            "CpuPercent": 0,
2020-08-26T05:06:10.8643596Z         	            "IOMaximumIOps": 0,
2020-08-26T05:06:10.8643769Z         	            "IOMaximumBandwidth": 0,
2020-08-26T05:06:10.8643934Z         	            "MaskedPaths": null,
2020-08-26T05:06:10.8644102Z         	            "ReadonlyPaths": null
2020-08-26T05:06:10.8644253Z         	        },
2020-08-26T05:06:10.8644410Z         	        "GraphDriver": {
2020-08-26T05:06:10.8644552Z         	            "Data": {
2020-08-26T05:06:10.8649112Z         	                "LowerDir": "/var/lib/docker/overlay2/dcd2f6e3a65512004c4c869215cd2db7abaa31c3765516b008f1bede76533935-init/diff:/var/lib/docker/overlay2/bdfcf34e42db0d06df50ccb17e3361ca9e2a25004255504d848a913549e70bd9/diff:/var/lib/docker/overlay2/e11747f3ace81f52bb22d6b4d16e664cb953c61469078c2c7469398102afc13c/diff:/var/lib/docker/overlay2/6846d970461339621f3dbddcc46d2683b22ee25792def1c427631068ac0339b7/diff:/var/lib/docker/overlay2/85dedece81a416b4693fae2aa4cb4ee3b4f3fff1cee90734f1f43e43b2c5cf3c/diff:/var/lib/docker/overlay2/b28b8a65dd0e921977b9dfaa57eafd78e3504d8a45cedf163744db6891c4c3f0/diff:/var/lib/docker/overlay2/200f720d20b9778c4e59e415960c6218cef961ef82ef6e57b36b6ea2b02cabfc/diff:/var/lib/docker/overlay2/9277732ed70e60d7e63f8995fdf1e2fef648248954a0bd1012f2306c53e5b36f/diff:/var/lib/docker/overlay2/0e639e400489e3dc97e31eb509004e174efd73d9f2c81e685cca6f0b26e4b848/diff:/var/lib/docker/overlay2/c3595b75bed927bc4b8948c6c43b6dffb83e06dce5445c6b404636c56147d614/diff:/var/lib/docker/overlay2/89842cbb614e7edcacdc10854cef815a67ef3808effabb8799a2674a8875ec4d/diff:/var/lib/docker/overlay2/7a4d02f68f77bf5ce4b7a32121cf0f9db65b05e15542977c48bb3fe670da758c/diff:/var/lib/docker/overlay2/0f03c9f1cb8562de380fd6c047885bba349a49046ba2a7d3259fd4f4bdc8fcb7/diff:/var/lib/docker/overlay2/24bf6b818b1213edba4c97960ca56693fc980c8ac950cfc0da73d3b55eb6116c/diff:/var/lib/docker/overlay2/0441b0f2228429311d844e3f1c6020c58ccd7e9aff1260cd2d254dc0155219db/diff:/var/lib/docker/overlay2/961c3df7ab8558e3364ae0e53eb503be7d3b47d3bf7682b8cbf74d0009db44f3/diff:/var/lib/docker/overlay2/016df74aa0e6f203afb58f71911510d209cd4cfc0983fc8361dc3207e7292f35/diff:/var/lib/docker/overlay2/968d0629fb6068df166c54ab7661a2392f91c3765a183b353aae005fe9bc86cc/diff:/var/lib/docker/overlay2/5dae8042dc8abc4d706b9b6173bc6021d55f5f249e1422b0847988d01ac0fa98/diff:/var/lib/docker/overlay2/8a67f91347ba4d1b995ccd65817bdc9a150e7444330e6d58961ab918013c58fd/diff:/var/lib/docker/overlay2/2d1dc962b8852a97af315c15e59367867382b2e674a78a45ed751a87bbcbc501/diff:/var/lib/docker/overlay2/261cd4309d9f344a468cf08b34e900f1c2e68c153a2b293d840b3e0b3dec29aa/diff:/var/lib/docker/overlay2/36298c55befa9decb3d9f3bfc229bb934c2a1ea8e5934d4a8f300da45b8748c5/diff:/var/lib/docker/overlay2/6f0638c3c1ba009ce61f5d12f289f664f6f6f7137184100cd5352cb2ad219f5f/diff:/var/lib/docker/overlay2/d46dc8f5ea7e979ab9b8e93a1d1249269167b199ecaf58bcb09ba0ee6a4c618e/diff:/var/lib/docker/overlay2/a90b6279aef0754e7fd49e0c5d8c1794f69e1862fefe7e2ff7bdf270155c1d6e/diff:/var/lib/docker/overlay2/daae120b505acdd9659becc32bb117e6876c748fdbac1d403aa704b418972d5c/diff:/var/lib/docker/overlay2/1a0198ae09cd15c9b877f0d366e8e57041d348bb4d3521cb0342e04afe77c25c/diff:/var/lib/docker/overlay2/4b312026ae01c37af8da37546f1f9ab698ceb9a16b41731258b6d0187eeb17ef/diff",
2020-08-26T05:06:10.8650137Z         	                "MergedDir": "/var/lib/docker/overlay2/dcd2f6e3a65512004c4c869215cd2db7abaa31c3765516b008f1bede76533935/merged",
2020-08-26T05:06:10.8650374Z         	                "UpperDir": "/var/lib/docker/overlay2/dcd2f6e3a65512004c4c869215cd2db7abaa31c3765516b008f1bede76533935/diff",
2020-08-26T05:06:10.8650606Z         	                "WorkDir": "/var/lib/docker/overlay2/dcd2f6e3a65512004c4c869215cd2db7abaa31c3765516b008f1bede76533935/work"
2020-08-26T05:06:10.8650791Z         	            },
2020-08-26T05:06:10.8650960Z         	            "Name": "overlay2"
2020-08-26T05:06:10.8651095Z         	        },
2020-08-26T05:06:10.8651255Z         	        "Mounts": [
2020-08-26T05:06:10.8651418Z         	            {
2020-08-26T05:06:10.8651590Z         	                "Type": "volume",
2020-08-26T05:06:10.8651992Z         	                "Name": "multinode-20200826045856-2663",
2020-08-26T05:06:10.8652414Z         	                "Source": "/var/lib/docker/volumes/multinode-20200826045856-2663/_data",
2020-08-26T05:06:10.8652610Z         	                "Destination": "/var",
2020-08-26T05:06:10.8652785Z         	                "Driver": "local",
2020-08-26T05:06:10.8652955Z         	                "Mode": "z",
2020-08-26T05:06:10.8653161Z         	                "RW": true,
2020-08-26T05:06:10.8653344Z         	                "Propagation": ""
2020-08-26T05:06:10.8653505Z         	            },
2020-08-26T05:06:10.8653663Z         	            {
2020-08-26T05:06:10.8653831Z         	                "Type": "bind",
2020-08-26T05:06:10.8654004Z         	                "Source": "/lib/modules",
2020-08-26T05:06:10.8654181Z         	                "Destination": "/lib/modules",
2020-08-26T05:06:10.8654335Z         	                "Mode": "ro",
2020-08-26T05:06:10.8654504Z         	                "RW": false,
2020-08-26T05:06:10.8654683Z         	                "Propagation": "rprivate"
2020-08-26T05:06:10.8654848Z         	            }
2020-08-26T05:06:10.8655004Z         	        ],
2020-08-26T05:06:10.8655160Z         	        "Config": {
2020-08-26T05:06:10.8655531Z         	            "Hostname": "multinode-20200826045856-2663",
2020-08-26T05:06:10.8655704Z         	            "Domainname": "",
2020-08-26T05:06:10.8655853Z         	            "User": "root",
2020-08-26T05:06:10.8656014Z         	            "AttachStdin": false,
2020-08-26T05:06:10.8656183Z         	            "AttachStdout": false,
2020-08-26T05:06:10.8656348Z         	            "AttachStderr": false,
2020-08-26T05:06:10.8656512Z         	            "ExposedPorts": {
2020-08-26T05:06:10.8656675Z         	                "22/tcp": {},
2020-08-26T05:06:10.8656838Z         	                "2376/tcp": {},
2020-08-26T05:06:10.8657009Z         	                "5000/tcp": {},
2020-08-26T05:06:10.8657157Z         	                "8443/tcp": {}
2020-08-26T05:06:10.8657321Z         	            },
2020-08-26T05:06:10.8657484Z         	            "Tty": true,
2020-08-26T05:06:10.8657647Z         	            "OpenStdin": false,
2020-08-26T05:06:10.8657814Z         	            "StdinOnce": false,
2020-08-26T05:06:10.8657976Z         	            "Env": [
2020-08-26T05:06:10.8658145Z         	                "container=docker",
2020-08-26T05:06:10.8658317Z         	                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
2020-08-26T05:06:10.8658499Z         	            ],
2020-08-26T05:06:10.8658656Z         	            "Cmd": null,
2020-08-26T05:06:10.8659147Z         	            "Image": "gcr.io/k8s-minikube/kicbase:v0.0.12-snapshot2@sha256:2e78dd54231714527631b27e9d6674549d9114d7be1ff55ed417d2bc514a63f4",
2020-08-26T05:06:10.8659340Z         	            "Volumes": null,
2020-08-26T05:06:10.8659504Z         	            "WorkingDir": "",
2020-08-26T05:06:10.8659667Z         	            "Entrypoint": [
2020-08-26T05:06:10.8659902Z         	                "/usr/local/bin/entrypoint",
2020-08-26T05:06:10.8660073Z         	                "/sbin/init"
2020-08-26T05:06:10.8660230Z         	            ],
2020-08-26T05:06:10.8660368Z         	            "OnBuild": null,
2020-08-26T05:06:10.8660532Z         	            "Labels": {
2020-08-26T05:06:10.8660707Z         	                "created_by.minikube.sigs.k8s.io": "true",
2020-08-26T05:06:10.8661115Z         	                "mode.minikube.sigs.k8s.io": "multinode-20200826045856-2663",
2020-08-26T05:06:10.8661509Z         	                "name.minikube.sigs.k8s.io": "multinode-20200826045856-2663",
2020-08-26T05:06:10.8661696Z         	                "role.minikube.sigs.k8s.io": ""
2020-08-26T05:06:10.8661858Z         	            },
2020-08-26T05:06:10.8662021Z         	            "StopSignal": "SIGRTMIN+3"
2020-08-26T05:06:10.8662158Z         	        },
2020-08-26T05:06:10.8662315Z         	        "NetworkSettings": {
2020-08-26T05:06:10.8662478Z         	            "Bridge": "",
2020-08-26T05:06:10.8662662Z         	            "SandboxID": "d6bd9993aa578bbf0a2e6f63200311e9f9c388ff1e32154d4b2dae98ad1fb667",
2020-08-26T05:06:10.8662848Z         	            "HairpinMode": false,
2020-08-26T05:06:10.8663017Z         	            "LinkLocalIPv6Address": "",
2020-08-26T05:06:10.8663182Z         	            "LinkLocalIPv6PrefixLen": 0,
2020-08-26T05:06:10.8663397Z         	            "Ports": {
2020-08-26T05:06:10.8663553Z         	                "22/tcp": [
2020-08-26T05:06:10.8663722Z         	                    {
2020-08-26T05:06:10.8663906Z         	                        "HostIp": "127.0.0.1",
2020-08-26T05:06:10.8664091Z         	                        "HostPort": "32771"
2020-08-26T05:06:10.8664262Z         	                    }
2020-08-26T05:06:10.8664430Z         	                ],
2020-08-26T05:06:10.8664595Z         	                "2376/tcp": [
2020-08-26T05:06:10.8664760Z         	                    {
2020-08-26T05:06:10.8664922Z         	                        "HostIp": "127.0.0.1",
2020-08-26T05:06:10.8665113Z         	                        "HostPort": "32770"
2020-08-26T05:06:10.8665290Z         	                    }
2020-08-26T05:06:10.8665446Z         	                ],
2020-08-26T05:06:10.8665609Z         	                "5000/tcp": [
2020-08-26T05:06:10.8665785Z         	                    {
2020-08-26T05:06:10.8665968Z         	                        "HostIp": "127.0.0.1",
2020-08-26T05:06:10.8666151Z         	                        "HostPort": "32769"
2020-08-26T05:06:10.8666304Z         	                    }
2020-08-26T05:06:10.8666471Z         	                ],
2020-08-26T05:06:10.8666640Z         	                "8443/tcp": [
2020-08-26T05:06:10.8666814Z         	                    {
2020-08-26T05:06:10.8666988Z         	                        "HostIp": "127.0.0.1",
2020-08-26T05:06:10.8667166Z         	                        "HostPort": "32768"
2020-08-26T05:06:10.8667334Z         	                    }
2020-08-26T05:06:10.8667478Z         	                ]
2020-08-26T05:06:10.8667636Z         	            },
2020-08-26T05:06:10.8667804Z         	            "SandboxKey": "/var/run/docker/netns/d6bd9993aa57",
2020-08-26T05:06:10.8667985Z         	            "SecondaryIPAddresses": null,
2020-08-26T05:06:10.8668151Z         	            "SecondaryIPv6Addresses": null,
2020-08-26T05:06:10.8668316Z         	            "EndpointID": "",
2020-08-26T05:06:10.8668478Z         	            "Gateway": "",
2020-08-26T05:06:10.8668644Z         	            "GlobalIPv6Address": "",
2020-08-26T05:06:10.8668793Z         	            "GlobalIPv6PrefixLen": 0,
2020-08-26T05:06:10.8668954Z         	            "IPAddress": "",
2020-08-26T05:06:10.8669115Z         	            "IPPrefixLen": 0,
2020-08-26T05:06:10.8669276Z         	            "IPv6Gateway": "",
2020-08-26T05:06:10.8669433Z         	            "MacAddress": "",
2020-08-26T05:06:10.8669588Z         	            "Networks": {
2020-08-26T05:06:10.8669963Z         	                "multinode-20200826045856-2663": {
2020-08-26T05:06:10.8670210Z         	                    "IPAMConfig": {
2020-08-26T05:06:10.8670379Z         	                        "IPv4Address": "192.168.39.2"
2020-08-26T05:06:10.8670557Z         	                    },
2020-08-26T05:06:10.8670730Z         	                    "Links": null,
2020-08-26T05:06:10.8670904Z         	                    "Aliases": [
2020-08-26T05:06:10.8671086Z         	                        "d26688304ef1"
2020-08-26T05:06:10.8671254Z         	                    ],
2020-08-26T05:06:10.8671443Z         	                    "NetworkID": "57f3b77d9a8320ea32a759da98ce2728614445cf54149c74bf84a3b11020fd8e",
2020-08-26T05:06:10.8671654Z         	                    "EndpointID": "f2c6d6e69f26b513f38856420cb33fbd24ec3465f600f5db9f3f36cc5eeccb1b",
2020-08-26T05:06:10.8671850Z         	                    "Gateway": "192.168.39.1",
2020-08-26T05:06:10.8672013Z         	                    "IPAddress": "192.168.39.2",
2020-08-26T05:06:10.8672194Z         	                    "IPPrefixLen": 24,
2020-08-26T05:06:10.8672372Z         	                    "IPv6Gateway": "",
2020-08-26T05:06:10.8672548Z         	                    "GlobalIPv6Address": "",
2020-08-26T05:06:10.8672723Z         	                    "GlobalIPv6PrefixLen": 0,
2020-08-26T05:06:10.8672905Z         	                    "MacAddress": "02:42:c0:a8:27:02",
2020-08-26T05:06:10.8673138Z         	                    "DriverOpts": null
2020-08-26T05:06:10.8673307Z         	                }
2020-08-26T05:06:10.8673447Z         	            }
2020-08-26T05:06:10.8673595Z         	        }
2020-08-26T05:06:10.8676691Z         	    }
2020-08-26T05:06:10.8676891Z         	]
2020-08-26T05:06:10.8677028Z         
2020-08-26T05:06:10.8677436Z         -- /stdout --
2020-08-26T05:06:10.8677894Z     helpers_test.go:232: (dbg) Run:  ./minikube-linux-amd64 status --format={{.Host}} -p multinode-20200826045856-2663 -n multinode-20200826045856-2663
2020-08-26T05:06:11.2725322Z     helpers_test.go:237: <<< TestMultiNode/serial/FreshStart2Nodes FAILED: start of post-mortem logs <<<
2020-08-26T05:06:11.2726052Z     helpers_test.go:238: ======>  post-mortem[TestMultiNode/serial/FreshStart2Nodes]: minikube logs <======
2020-08-26T05:06:11.2726667Z     helpers_test.go:240: (dbg) Run:  ./minikube-linux-amd64 -p multinode-20200826045856-2663 logs -n 25
2020-08-26T05:06:12.9795380Z     helpers_test.go:240: (dbg) Done: ./minikube-linux-amd64 -p multinode-20200826045856-2663 logs -n 25: (1.707285321s)
2020-08-26T05:06:12.9854294Z     helpers_test.go:245: TestMultiNode/serial/FreshStart2Nodes logs: 
2020-08-26T05:06:12.9855656Z         -- stdout --
2020-08-26T05:06:12.9855873Z         	* ==> Docker <==
2020-08-26T05:06:12.9856308Z         	* -- Logs begin at Wed 2020-08-26 04:59:39 UTC, end at Wed 2020-08-26 05:06:11 UTC. --
2020-08-26T05:06:12.9856955Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 systemd[1]: Stopped Docker Application Container Engine.
2020-08-26T05:06:12.9857406Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 systemd[1]: Starting Docker Application Container Engine...
2020-08-26T05:06:12.9857905Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.593555611Z" level=info msg="Starting up"
2020-08-26T05:06:12.9858421Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.595423380Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2020-08-26T05:06:12.9858977Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.595439981Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2020-08-26T05:06:12.9859735Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.595459482Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
2020-08-26T05:06:12.9860280Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.595470382Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2020-08-26T05:06:12.9860997Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.600895384Z" level=info msg="parsed scheme: \"unix\"" module=grpc
2020-08-26T05:06:12.9862220Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.600936685Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
2020-08-26T05:06:12.9863007Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.600952786Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}" module=grpc
2020-08-26T05:06:12.9863549Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.600962086Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
2020-08-26T05:06:12.9864082Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.607529131Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
2020-08-26T05:06:12.9864713Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.612626820Z" level=warning msg="Your kernel does not support cgroup rt period"
2020-08-26T05:06:12.9865248Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.612649421Z" level=warning msg="Your kernel does not support cgroup rt runtime"
2020-08-26T05:06:12.9865754Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.612656321Z" level=warning msg="Your kernel does not support cgroup blkio weight"
2020-08-26T05:06:12.9866271Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.612662522Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
2020-08-26T05:06:12.9866761Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.612790526Z" level=info msg="Loading containers: start."
2020-08-26T05:06:12.9867380Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.713635478Z" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
2020-08-26T05:06:12.9867895Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.749847426Z" level=info msg="Loading containers: done."
2020-08-26T05:06:12.9868549Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.759015167Z" level=warning msg="Not using native diff for overlay2, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled" storage-driver=overlay2
2020-08-26T05:06:12.9869124Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.759197073Z" level=info msg="Docker daemon" commit=afacb8b7f0 graphdriver(s)=overlay2 version=19.03.8
2020-08-26T05:06:12.9869632Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.759249175Z" level=info msg="Daemon has completed initialization"
2020-08-26T05:06:12.9870068Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 systemd[1]: Started Docker Application Container Engine.
2020-08-26T05:06:12.9870554Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.770999413Z" level=info msg="API listen on /var/run/docker.sock"
2020-08-26T05:06:12.9871640Z         	* Aug 26 04:59:59 multinode-20200826045856-2663 dockerd[335]: time="2020-08-26T04:59:59.771059315Z" level=info msg="API listen on [::]:2376"
2020-08-26T05:06:12.9871815Z         	* 
2020-08-26T05:06:12.9871968Z         	* ==> container status <==
2020-08-26T05:06:12.9872234Z         	* CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID
2020-08-26T05:06:12.9872441Z         	* 01b26f0089fad       67da37a9a360e       5 minutes ago       Running             coredns                   0                   6598e821da156
2020-08-26T05:06:12.9872934Z         	* 9f49c9d37d59e       9c3ca9f065bb1       5 minutes ago       Running             storage-provisioner       0                   ebaefcbe7223c
2020-08-26T05:06:12.9873423Z         	* c90835d84eea1       3439b7546f29b       5 minutes ago       Running             kube-proxy                0                   1e3764e4476cd
2020-08-26T05:06:12.9873611Z         	* d60a75013b599       303ce5db0e90d       6 minutes ago       Running             etcd                      0                   4a8c6c3a3e9bb
2020-08-26T05:06:12.9874691Z         	* 81878e5316959       da26705ccb4b5       6 minutes ago       Running             kube-controller-manager   0                   848b220ccdf88
2020-08-26T05:06:12.9875308Z         	* 6dd21bbfa3221       76216c34ed0c7       6 minutes ago       Running             kube-scheduler            0                   66ce0f402a81d
2020-08-26T05:06:12.9875787Z         	* 96896e99b4b55       7e28efa976bd1       6 minutes ago       Running             kube-apiserver            0                   b8d074d7a6150
2020-08-26T05:06:12.9876055Z         	* 
2020-08-26T05:06:12.9876213Z         	* ==> coredns [01b26f0089fa] <==
2020-08-26T05:06:12.9876366Z         	* .:53
2020-08-26T05:06:12.9876536Z         	* [INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7
2020-08-26T05:06:12.9876849Z         	* CoreDNS-1.6.7
2020-08-26T05:06:12.9877304Z         	* linux/amd64, go1.13.6, da7f65b
2020-08-26T05:06:12.9877443Z         	* 
2020-08-26T05:06:12.9877592Z         	* ==> describe nodes <==
2020-08-26T05:06:12.9877948Z         	* Name:               multinode-20200826045856-2663
2020-08-26T05:06:12.9878115Z         	* Roles:              master
2020-08-26T05:06:12.9878283Z         	* Labels:             beta.kubernetes.io/arch=amd64
2020-08-26T05:06:12.9878452Z         	*                     beta.kubernetes.io/os=linux
2020-08-26T05:06:12.9878617Z         	*                     kubernetes.io/arch=amd64
2020-08-26T05:06:12.9878995Z         	*                     kubernetes.io/hostname=multinode-20200826045856-2663
2020-08-26T05:06:12.9879157Z         	*                     kubernetes.io/os=linux
2020-08-26T05:06:12.9879336Z         	*                     minikube.k8s.io/commit=5bb793b6310f687df35093dcfb16552647376820
2020-08-26T05:06:12.9879717Z         	*                     minikube.k8s.io/name=multinode-20200826045856-2663
2020-08-26T05:06:12.9879901Z         	*                     minikube.k8s.io/updated_at=2020_08_26T05_00_16_0700
2020-08-26T05:06:12.9880183Z         	*                     minikube.k8s.io/version=v1.12.3
2020-08-26T05:06:12.9880565Z         	*                     node-role.kubernetes.io/master=
2020-08-26T05:06:12.9880951Z         	* Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
2020-08-26T05:06:12.9881125Z         	*                     node.alpha.kubernetes.io/ttl: 0
2020-08-26T05:06:12.9881495Z         	*                     volumes.kubernetes.io/controller-managed-attach-detach: true
2020-08-26T05:06:12.9881689Z         	* CreationTimestamp:  Wed, 26 Aug 2020 05:00:13 +0000
2020-08-26T05:06:12.9881830Z         	* Taints:             <none>
2020-08-26T05:06:12.9881991Z         	* Unschedulable:      false
2020-08-26T05:06:12.9882140Z         	* Lease:
2020-08-26T05:06:12.9882475Z         	*   HolderIdentity:  multinode-20200826045856-2663
2020-08-26T05:06:12.9882644Z         	*   AcquireTime:     <unset>
2020-08-26T05:06:12.9882807Z         	*   RenewTime:       Wed, 26 Aug 2020 05:06:06 +0000
2020-08-26T05:06:12.9882961Z         	* Conditions:
2020-08-26T05:06:12.9883141Z         	*   Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
2020-08-26T05:06:12.9883695Z         	*   ----             ------  -----------------                 ------------------                ------                       -------
2020-08-26T05:06:12.9883890Z         	*   MemoryPressure   False   Wed, 26 Aug 2020 05:05:27 +0000   Wed, 26 Aug 2020 05:00:08 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
2020-08-26T05:06:12.9884106Z         	*   DiskPressure     False   Wed, 26 Aug 2020 05:05:27 +0000   Wed, 26 Aug 2020 05:00:08 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
2020-08-26T05:06:12.9884311Z         	*   PIDPressure      False   Wed, 26 Aug 2020 05:05:27 +0000   Wed, 26 Aug 2020 05:00:08 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
2020-08-26T05:06:12.9884529Z         	*   Ready            True    Wed, 26 Aug 2020 05:05:27 +0000   Wed, 26 Aug 2020 05:00:26 +0000   KubeletReady                 kubelet is posting ready status
2020-08-26T05:06:12.9884716Z         	* Addresses:
2020-08-26T05:06:12.9884870Z         	*   InternalIP:  192.168.39.2
2020-08-26T05:06:12.9885203Z         	*   Hostname:    multinode-20200826045856-2663
2020-08-26T05:06:12.9885645Z         	* Capacity:
2020-08-26T05:06:12.9885796Z         	*   cpu:                2
2020-08-26T05:06:12.9886126Z         	*   ephemeral-storage:  87218124Ki
2020-08-26T05:06:12.9886510Z         	*   hugepages-1Gi:      0
2020-08-26T05:06:12.9886798Z         	*   hugepages-2Mi:      0
2020-08-26T05:06:12.9886951Z         	*   memory:             7093492Ki
2020-08-26T05:06:12.9887107Z         	*   pods:               110
2020-08-26T05:06:12.9887260Z         	* Allocatable:
2020-08-26T05:06:12.9887410Z         	*   cpu:                2
2020-08-26T05:06:12.9887725Z         	*   ephemeral-storage:  87218124Ki
2020-08-26T05:06:12.9888026Z         	*   hugepages-1Gi:      0
2020-08-26T05:06:12.9888296Z         	*   hugepages-2Mi:      0
2020-08-26T05:06:12.9888459Z         	*   memory:             7093492Ki
2020-08-26T05:06:12.9888627Z         	*   pods:               110
2020-08-26T05:06:12.9888781Z         	* System Info:
2020-08-26T05:06:12.9888943Z         	*   Machine ID:                 6c1a5a3510854d6a8d7a3dc1f70f471a
2020-08-26T05:06:12.9889325Z         	*   System UUID:                0781b9be-47d6-42cc-a920-3341c092e68c
2020-08-26T05:06:12.9889709Z         	*   Boot ID:                    53fe6350-f65e-4ad1-811b-dbeaeb24acb6
2020-08-26T05:06:12.9890050Z         	*   Kernel Version:             5.3.0-1034-azure
2020-08-26T05:06:12.9890215Z         	*   OS Image:                   Ubuntu 20.04 LTS
2020-08-26T05:06:12.9890355Z         	*   Operating System:           linux
2020-08-26T05:06:12.9890515Z         	*   Architecture:               amd64
2020-08-26T05:06:12.9890673Z         	*   Container Runtime Version:  docker://19.3.8
2020-08-26T05:06:12.9890885Z         	*   Kubelet Version:            v1.18.3
2020-08-26T05:06:12.9891280Z         	*   Kube-Proxy Version:         v1.18.3
2020-08-26T05:06:12.9891645Z         	* Non-terminated Pods:          (7 in total)
2020-08-26T05:06:12.9891838Z         	*   Namespace                   Name                                                     CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
2020-08-26T05:06:12.9892312Z         	*   ---------                   ----                                                     ------------  ----------  ---------------  -------------  ---
2020-08-26T05:06:12.9892799Z         	*   kube-system                 coredns-66bff467f8-gh45l                                 100m (5%)     0 (0%)      70Mi (1%)        170Mi (2%)     5m50s
2020-08-26T05:06:12.9893290Z         	*   kube-system                 etcd-multinode-20200826045856-2663                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         5m56s
2020-08-26T05:06:12.9893774Z         	*   kube-system                 kube-apiserver-multinode-20200826045856-2663             250m (12%)    0 (0%)      0 (0%)           0 (0%)         5m56s
2020-08-26T05:06:12.9894317Z         	*   kube-system                 kube-controller-manager-multinode-20200826045856-2663    200m (10%)    0 (0%)      0 (0%)           0 (0%)         5m55s
2020-08-26T05:06:12.9894784Z         	*   kube-system                 kube-proxy-jkqql                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         5m50s
2020-08-26T05:06:12.9895270Z         	*   kube-system                 kube-scheduler-multinode-20200826045856-2663             100m (5%)     0 (0%)      0 (0%)           0 (0%)         5m55s
2020-08-26T05:06:12.9895746Z         	*   kube-system                 storage-provisioner                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         5m49s
2020-08-26T05:06:12.9895927Z         	* Allocated resources:
2020-08-26T05:06:12.9896089Z         	*   (Total limits may be over 100 percent, i.e., overcommitted.)
2020-08-26T05:06:12.9896263Z         	*   Resource           Requests    Limits
2020-08-26T05:06:12.9896590Z         	*   --------           --------    ------
2020-08-26T05:06:12.9896757Z         	*   cpu                650m (32%)  0 (0%)
2020-08-26T05:06:12.9896920Z         	*   memory             70Mi (1%)   170Mi (2%)
2020-08-26T05:06:12.9897246Z         	*   ephemeral-storage  0 (0%)      0 (0%)
2020-08-26T05:06:12.9897545Z         	*   hugepages-1Gi      0 (0%)      0 (0%)
2020-08-26T05:06:12.9897933Z         	*   hugepages-2Mi      0 (0%)      0 (0%)
2020-08-26T05:06:12.9898097Z         	* Events:
2020-08-26T05:06:12.9898267Z         	*   Type    Reason                   Age                  From                                       Message
2020-08-26T05:06:12.9898688Z         	*   ----    ------                   ----                 ----                                       -------
2020-08-26T05:06:12.9899204Z         	*   Normal  NodeHasSufficientMemory  6m5s (x4 over 6m6s)  kubelet, multinode-20200826045856-2663     Node multinode-20200826045856-2663 status is now: NodeHasSufficientMemory
2020-08-26T05:06:12.9899731Z         	*   Normal  NodeHasNoDiskPressure    6m5s (x4 over 6m6s)  kubelet, multinode-20200826045856-2663     Node multinode-20200826045856-2663 status is now: NodeHasNoDiskPressure
2020-08-26T05:06:12.9900252Z         	*   Normal  NodeHasSufficientPID     6m5s (x4 over 6m6s)  kubelet, multinode-20200826045856-2663     Node multinode-20200826045856-2663 status is now: NodeHasSufficientPID
2020-08-26T05:06:12.9900736Z         	*   Normal  Starting                 5m56s                kubelet, multinode-20200826045856-2663     Starting kubelet.
2020-08-26T05:06:12.9901254Z         	*   Normal  NodeHasSufficientMemory  5m56s                kubelet, multinode-20200826045856-2663     Node multinode-20200826045856-2663 status is now: NodeHasSufficientMemory
2020-08-26T05:06:12.9901772Z         	*   Normal  NodeHasNoDiskPressure    5m56s                kubelet, multinode-20200826045856-2663     Node multinode-20200826045856-2663 status is now: NodeHasNoDiskPressure
2020-08-26T05:06:12.9902287Z         	*   Normal  NodeHasSufficientPID     5m56s                kubelet, multinode-20200826045856-2663     Node multinode-20200826045856-2663 status is now: NodeHasSufficientPID
2020-08-26T05:06:12.9902771Z         	*   Normal  NodeAllocatableEnforced  5m56s                kubelet, multinode-20200826045856-2663     Updated Node Allocatable limit across pods
2020-08-26T05:06:12.9903221Z         	*   Normal  Starting                 5m49s                kube-proxy, multinode-20200826045856-2663  Starting kube-proxy.
2020-08-26T05:06:12.9903719Z         	*   Normal  NodeReady                5m46s                kubelet, multinode-20200826045856-2663     Node multinode-20200826045856-2663 status is now: NodeReady
2020-08-26T05:06:12.9903905Z         	* 
2020-08-26T05:06:12.9904031Z         	* ==> dmesg <==
2020-08-26T05:06:12.9904471Z         	* [Aug26 04:55] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
2020-08-26T05:06:12.9904953Z         	* [  +0.140055] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
2020-08-26T05:06:12.9905146Z         	*               * this clock source is slow. Consider trying other clock sources
2020-08-26T05:06:12.9905331Z         	* [  +1.482726] platform eisa.0: EISA: Cannot allocate resource for mainboard
2020-08-26T05:06:12.9905519Z         	* [  +0.007298] platform eisa.0: Cannot allocate resource for EISA slot 1
2020-08-26T05:06:12.9905705Z         	* [  +0.007329] platform eisa.0: Cannot allocate resource for EISA slot 2
2020-08-26T05:06:12.9905891Z         	* [  +0.007049] platform eisa.0: Cannot allocate resource for EISA slot 3
2020-08-26T05:06:12.9906069Z         	* [  +0.007251] platform eisa.0: Cannot allocate resource for EISA slot 4
2020-08-26T05:06:12.9906242Z         	* [  +0.016569] platform eisa.0: Cannot allocate resource for EISA slot 5
2020-08-26T05:06:12.9906392Z         	* [  +0.000001] platform eisa.0: Cannot allocate resource for EISA slot 6
2020-08-26T05:06:12.9906579Z         	* [  +0.000001] platform eisa.0: Cannot allocate resource for EISA slot 7
2020-08-26T05:06:12.9906741Z         	* [  +0.000001] platform eisa.0: Cannot allocate resource for EISA slot 8
2020-08-26T05:06:12.9906918Z         	* [  +0.202907] Unstable clock detected, switching default tracing clock to "global"
2020-08-26T05:06:12.9907141Z         	*               If you want to keep using the local clock, then add:
2020-08-26T05:06:12.9907309Z         	*                 "trace_clock=local"
2020-08-26T05:06:12.9907467Z         	*               on the kernel command line
2020-08-26T05:06:12.9907973Z         	* [  +3.552588] systemd[1]: Configuration file /etc/systemd/system/runner-provisioner.service is marked executable. Please remove executable permission bits. Proceeding anyway.
2020-08-26T05:06:12.9908411Z         	* [  +0.017206] systemd[1]: /etc/systemd/system/runner-provisioner.service:3: Invalid URL, ignoring: None
2020-08-26T05:06:12.9908605Z         	* [  +7.823845] new mount options do not match the existing superblock, will be ignored
2020-08-26T05:06:12.9908794Z         	* [  +5.345906] kauditd_printk_skb: 19 callbacks suppressed
2020-08-26T05:06:12.9908942Z         	* [  +0.061924] Started bpfilter
2020-08-26T05:06:12.9909100Z         	* 
2020-08-26T05:06:12.9909254Z         	* ==> etcd [d60a75013b59] <==
2020-08-26T05:06:12.9909439Z         	* raft2020/08/26 05:00:09 INFO: 6c80de388e5020e8 switched to configuration voters=(7818493287602331880)
2020-08-26T05:06:12.9909846Z         	* 2020-08-26 05:00:09.326266 W | auth: simple token is not cryptographically signed
2020-08-26T05:06:12.9910273Z         	* 2020-08-26 05:00:09.328121 I | etcdserver: starting server... [version: 3.4.3, cluster version: to_be_decided]
2020-08-26T05:06:12.9910847Z         	* 2020-08-26 05:00:09.330392 I | embed: ClientTLS: cert = /var/lib/minikube/certs/etcd/server.crt, key = /var/lib/minikube/certs/etcd/server.key, trusted-ca = /var/lib/minikube/certs/etcd/ca.crt, client-cert-auth = true, crl-file = 
2020-08-26T05:06:12.9911336Z         	* 2020-08-26 05:00:09.331143 I | embed: listening for metrics on http://127.0.0.1:2381
2020-08-26T05:06:12.9911816Z         	* 2020-08-26 05:00:09.331338 I | etcdserver: 6c80de388e5020e8 as single-node; fast-forwarding 9 ticks (election ticks 10)
2020-08-26T05:06:12.9912230Z         	* 2020-08-26 05:00:09.331506 I | embed: listening for peers on 192.168.39.2:2380
2020-08-26T05:06:12.9912435Z         	* raft2020/08/26 05:00:09 INFO: 6c80de388e5020e8 switched to configuration voters=(7818493287602331880)
2020-08-26T05:06:12.9912951Z         	* 2020-08-26 05:00:09.331821 I | etcdserver/membership: added member 6c80de388e5020e8 [https://192.168.39.2:2380] to cluster e20ba2e00cb0e827
2020-08-26T05:06:12.9913181Z         	* raft2020/08/26 05:00:09 INFO: 6c80de388e5020e8 is starting a new election at term 1
2020-08-26T05:06:12.9913364Z         	* raft2020/08/26 05:00:09 INFO: 6c80de388e5020e8 became candidate at term 2
2020-08-26T05:06:12.9939027Z         	* raft2020/08/26 05:00:09 INFO: 6c80de388e5020e8 received MsgVoteResp from 6c80de388e5020e8 at term 2
2020-08-26T05:06:12.9939532Z         	* raft2020/08/26 05:00:09 INFO: 6c80de388e5020e8 became leader at term 2
2020-08-26T05:06:12.9939730Z         	* raft2020/08/26 05:00:09 INFO: raft.node: 6c80de388e5020e8 elected leader 6c80de388e5020e8 at term 2
2020-08-26T05:06:12.9940605Z         	* 2020-08-26 05:00:09.508568 I | etcdserver: setting up the initial cluster version to 3.4
2020-08-26T05:06:12.9941033Z         	* 2020-08-26 05:00:09.509090 N | etcdserver/membership: set the initial cluster version to 3.4
2020-08-26T05:06:12.9941440Z         	* 2020-08-26 05:00:09.509267 I | etcdserver/api: enabled capabilities for version 3.4
2020-08-26T05:06:12.9941990Z         	* 2020-08-26 05:00:09.509408 I | etcdserver: published {Name:multinode-20200826045856-2663 ClientURLs:[https://192.168.39.2:2379]} to cluster e20ba2e00cb0e827
2020-08-26T05:06:12.9942415Z         	* 2020-08-26 05:00:09.509901 I | embed: ready to serve client requests
2020-08-26T05:06:12.9942822Z         	* 2020-08-26 05:00:09.511482 I | embed: serving client requests on 192.168.39.2:2379
2020-08-26T05:06:12.9943209Z         	* 2020-08-26 05:00:09.518873 I | embed: ready to serve client requests
2020-08-26T05:06:12.9943596Z         	* 2020-08-26 05:00:09.520223 I | embed: serving client requests on 127.0.0.1:2379
2020-08-26T05:06:12.9944103Z         	* 2020-08-26 05:00:40.893167 W | wal: sync duration of 1.069312304s, expected less than 1s
2020-08-26T05:06:12.9944698Z         	* 2020-08-26 05:00:40.947705 W | etcdserver: read-only range request "key:\"/registry/services/endpoints/kube-system/k8s.io-minikube-hostpath\" " with result "range_response_count:1 size:1109" took too long (709.653638ms) to execute
2020-08-26T05:06:12.9945236Z         	* 2020-08-26 05:00:51.345631 W | etcdserver: read-only range request "key:\"/registry/health\" " with result "range_response_count:0 size:5" took too long (197.994998ms) to execute
2020-08-26T05:06:12.9945418Z         	* 
2020-08-26T05:06:12.9945574Z         	* ==> kernel <==
2020-08-26T05:06:12.9945736Z         	*  05:06:12 up 11 min,  0 users,  load average: 0.83, 0.80, 0.51
2020-08-26T05:06:12.9946186Z         	* Linux multinode-20200826045856-2663 5.3.0-1034-azure #35~18.04.1-Ubuntu SMP Mon Jul 13 12:54:45 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
2020-08-26T05:06:12.9946373Z         	* PRETTY_NAME="Ubuntu 20.04 LTS"
2020-08-26T05:06:12.9946525Z         	* 
2020-08-26T05:06:12.9946846Z         	* ==> kube-apiserver [96896e99b4b5] <==
2020-08-26T05:06:12.9947252Z         	* I0826 05:00:13.142785       1 shared_informer.go:223] Waiting for caches to sync for crd-autoregister
2020-08-26T05:06:12.9947483Z         	* E0826 05:00:13.207086       1 controller.go:152] Unable to remove old endpoints from kubernetes service: StorageError: key not found, Code: 1, Key: /registry/masterleases/192.168.39.2, ResourceVersion: 0, AdditionalErrorMsg: 
2020-08-26T05:06:12.9947911Z         	* I0826 05:00:13.255826       1 shared_informer.go:230] Caches are synced for crd-autoregister 
2020-08-26T05:06:12.9948226Z         	* I0826 05:00:13.255877       1 shared_informer.go:230] Caches are synced for cluster_authentication_trust_controller 
2020-08-26T05:06:12.9948409Z         	* I0826 05:00:13.255897       1 cache.go:39] Caches are synced for APIServiceRegistrationController controller
2020-08-26T05:06:12.9948595Z         	* I0826 05:00:13.255909       1 cache.go:39] Caches are synced for AvailableConditionController controller
2020-08-26T05:06:12.9948772Z         	* I0826 05:00:13.255940       1 cache.go:39] Caches are synced for autoregister controller
2020-08-26T05:06:12.9948952Z         	* I0826 05:00:14.139584       1 controller.go:130] OpenAPI AggregationController: action for item : Nothing (removed from the queue).
2020-08-26T05:06:12.9949140Z         	* I0826 05:00:14.139731       1 controller.go:130] OpenAPI AggregationController: action for item k8s_internal_local_delegation_chain_0000000000: Nothing (removed from the queue).
2020-08-26T05:06:12.9949564Z         	* I0826 05:00:14.147247       1 storage_scheduling.go:134] created PriorityClass system-node-critical with value 2000001000
2020-08-26T05:06:12.9950030Z         	* I0826 05:00:14.152727       1 storage_scheduling.go:134] created PriorityClass system-cluster-critical with value 2000000000
2020-08-26T05:06:12.9950226Z         	* I0826 05:00:14.152752       1 storage_scheduling.go:143] all system priority classes are created successfully or already exist.
2020-08-26T05:06:12.9950411Z         	* I0826 05:00:14.534274       1 controller.go:606] quota admission added evaluator for: roles.rbac.authorization.k8s.io
2020-08-26T05:06:12.9950594Z         	* I0826 05:00:14.563785       1 controller.go:606] quota admission added evaluator for: rolebindings.rbac.authorization.k8s.io
2020-08-26T05:06:12.9950772Z         	* W0826 05:00:14.704074       1 lease.go:224] Resetting endpoints for master service "kubernetes" to [192.168.39.2]
2020-08-26T05:06:12.9950951Z         	* I0826 05:00:14.705127       1 controller.go:606] quota admission added evaluator for: endpoints
2020-08-26T05:06:12.9951132Z         	* I0826 05:00:14.709495       1 controller.go:606] quota admission added evaluator for: endpointslices.discovery.k8s.io
2020-08-26T05:06:12.9951299Z         	* I0826 05:00:15.503704       1 controller.go:606] quota admission added evaluator for: serviceaccounts
2020-08-26T05:06:12.9951523Z         	* I0826 05:00:16.080130       1 controller.go:606] quota admission added evaluator for: deployments.apps
2020-08-26T05:06:12.9951701Z         	* I0826 05:00:16.142242       1 controller.go:606] quota admission added evaluator for: daemonsets.apps
2020-08-26T05:06:12.9951880Z         	* I0826 05:00:16.550990       1 controller.go:606] quota admission added evaluator for: leases.coordination.k8s.io
2020-08-26T05:06:12.9952063Z         	* I0826 05:00:22.305369       1 controller.go:606] quota admission added evaluator for: replicasets.apps
2020-08-26T05:06:12.9952240Z         	* I0826 05:00:22.380444       1 controller.go:606] quota admission added evaluator for: controllerrevisions.apps
2020-08-26T05:06:12.9952923Z         	* I0826 05:00:40.950224       1 trace.go:116] Trace[387093642]: "Get" url:/api/v1/namespaces/kube-system/endpoints/k8s.io-minikube-hostpath,user-agent:storage-provisioner/v0.0.0 (linux/amd64) kubernetes/$Format,client:192.168.39.2 (started: 2020-08-26 05:00:40.237585161 +0000 UTC m=+31.983603517) (total time: 712.602265ms):
2020-08-26T05:06:12.9953133Z         	* Trace[387093642]: [712.520162ms] [712.511661ms] About to write a response
2020-08-26T05:06:12.9953287Z         	* 
2020-08-26T05:06:12.9953594Z         	* ==> kube-controller-manager [81878e531695] <==
2020-08-26T05:06:12.9953861Z         	* I0826 05:00:22.302257       1 shared_informer.go:230] Caches are synced for expand 
2020-08-26T05:06:12.9954901Z         	* I0826 05:00:22.309441       1 event.go:278] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"kube-system", Name:"coredns", UID:"6aa9a124-08ae-4414-aef1-92466af446c7", APIVersion:"apps/v1", ResourceVersion:"205", FieldPath:""}): type: 'Normal' reason: 'ScalingReplicaSet' Scaled up replica set coredns-66bff467f8 to 2
2020-08-26T05:06:12.9955179Z         	* I0826 05:00:22.310352       1 shared_informer.go:230] Caches are synced for GC 
2020-08-26T05:06:12.9956086Z         	* I0826 05:00:22.320059       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"34169fe4-b2a3-4495-a883-8d625452cbf4", APIVersion:"apps/v1", ResourceVersion:"301", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-66bff467f8-s7qqb
2020-08-26T05:06:12.9956338Z         	* I0826 05:00:22.321063       1 shared_informer.go:230] Caches are synced for TTL 
2020-08-26T05:06:12.9957059Z         	* I0826 05:00:22.326944       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"kube-system", Name:"coredns-66bff467f8", UID:"34169fe4-b2a3-4495-a883-8d625452cbf4", APIVersion:"apps/v1", ResourceVersion:"301", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: coredns-66bff467f8-gh45l
2020-08-26T05:06:12.9957390Z         	* I0826 05:00:22.351193       1 shared_informer.go:230] Caches are synced for daemon sets 
2020-08-26T05:06:12.9957584Z         	* I0826 05:00:22.359627       1 shared_informer.go:230] Caches are synced for stateful set 
2020-08-26T05:06:12.9958315Z         	* I0826 05:00:22.399117       1 event.go:278] Event(v1.ObjectReference{Kind:"DaemonSet", Namespace:"kube-system", Name:"kube-proxy", UID:"28a3b31e-d276-4ef9-b0ce-daeb5feef858", APIVersion:"apps/v1", ResourceVersion:"215", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-jkqql
2020-08-26T05:06:12.9958572Z         	* I0826 05:00:22.435712       1 shared_informer.go:230] Caches are synced for bootstrap_signer 
2020-08-26T05:06:12.9958759Z         	* I0826 05:00:22.459431       1 shared_informer.go:230] Caches are synced for service account 
2020-08-26T05:06:12.9958947Z
@medyagh medyagh added co/multinode Issues related to multinode clusters kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Aug 26, 2020
@sharifelgamal
Copy link
Collaborator

So the important part of the error is:

error execution phase preflight: couldn't validate the identity of the API Server: Get "https://control-plane.minikube.internal:8443/api/v1/namespaces/kube-public/configmaps/cluster-info?timeout=10s": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
2020-09-01T16:01:25.4674600Z         	To see the stack trace of this error execute with --v=5 or higher

So either the control plane is legitimately timing out responding to the join command, or is actually rejecting it because of some token nonsense. It only seems to be happening on the gh action machines, so I'll open a PR with some extra debugging so I can pin down what's actually happening.

@sharifelgamal
Copy link
Collaborator

Using #9146 as my debugging playground.

@medyagh
Copy link
Member Author

medyagh commented Sep 2, 2020

any new detaills @sharifelgamal ? maybe we can add a docker network ls

and docker docker network inspect bridge... in your debug pr (in gh action) to see if network is werid or not

@sharifelgamal
Copy link
Collaborator

sharifelgamal commented Sep 2, 2020 •
edited
Loading

network info:

[
    {
        "Name": "bridge",
        "Id": "3c4aafba5e8effc74ae96dcc139cf397065abf97a152900fce8c38cd2e6eb7ea",
        "Created": "2020-09-02T20:05:57.7131582Z",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {},
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

seems pretty normal to me

@sharifelgamal
Copy link
Collaborator

Turns out static IP was breaking this, the revert PR at #9166 will fix this for now. We'll have to be mindful of this test when we re-implement static IP.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
co/multinode Issues related to multinode clusters kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@medyagh @sharifelgamal