`minikube start` on M1 Mac failing with `'kubelet --version': exit status 255` #12274

rams3sh · 2021-08-16T11:08:19Z

Steps to reproduce the issue:

Run the following command :-
minikube start --driver=docker

Full output of failed command:

Log file created at: 2021/08/16 14:07:42 Running on machine: FDLDL064 Binary: Built with gc go1.16.4 for linux/amd64 Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg I0816 14:07:42.554271 202041 out.go:286] Setting OutFile to fd 1 ... I0816 14:07:42.554436 202041 out.go:338] isatty.IsTerminal(1) = true I0816 14:07:42.554443 202041 out.go:299] Setting ErrFile to fd 2... I0816 14:07:42.554451 202041 out.go:338] isatty.IsTerminal(2) = true I0816 14:07:42.554716 202041 root.go:312] Updating PATH: /home/rseshadri/.minikube/bin I0816 14:07:42.555439 202041 out.go:293] Setting JSON to false I0816 14:07:42.576973 202041 start.go:111] hostinfo: {"hostname":"FDLDL064","uptime":331985,"bootTime":1628771077,"procs":320,"os":"linux","platform":"ubuntu","platformFamily":"debian","platformVersion":"20.04","kernelVersion":"5.11.0-25-generic","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"host","hostId":"6499bef5-befd-4de8-b950-457d91d90574"} I0816 14:07:42.577154 202041 start.go:121] virtualization: kvm host I0816 14:07:42.582284 202041 out.go:165] 😄 minikube v1.22.0 on Ubuntu 20.04 I0816 14:07:42.583324 202041 notify.go:169] Checking for updates... I0816 14:07:42.584078 202041 driver.go:335] Setting default libvirt URI to qemu:///system I0816 14:07:42.645531 202041 docker.go:132] docker version: linux-20.10.7 I0816 14:07:42.645589 202041 cli_runner.go:115] Run: docker system info --format "{{json .}}" I0816 14:07:42.692299 202041 info.go:263] docker info: {ID:63YF:7UB3:XY35:AJ3T:V4DH:3LU5:UWNZ:C3KS:Y2MC:L4BM:UYRV:WLCK Containers:15 ContainersRunning:0 ContainersPaused:0 ContainersStopped:15 Images:26 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus: Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization: Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:22 OomKillDisable:true NGoroutines:34 SystemTime:2021-08-16 14:07:42.677547407 +0530 IST LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.11.0-25-generic OperatingSystem:Ubuntu 20.04.2 LTS OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:4 MemTotal:16665108480 GenericResources: DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:FDLDL064 Labels:[] ExperimentalBuild:false ServerVersion:20.10.7 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID: Expected:} RuncCommit:{ID: Expected:} InitCommit:{ID: Expected:} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense: Warnings: ServerErrors:[] ClientInfo:{Debug:false Plugins:[] Warnings:}} I0816 14:07:42.692356 202041 docker.go:244] overlay module found I0816 14:07:42.696004 202041 out.go:165] ✨ Using the docker driver based on user configuration I0816 14:07:42.696053 202041 start.go:278] selected driver: docker I0816 14:07:42.696056 202041 start.go:751] validating driver "docker" against I0816 14:07:42.696089 202041 start.go:762] status for docker: {Installed:true Healthy:true Running:false NeedsImprovement:false Error: Reason: Fix: Doc:} I0816 14:07:42.696350 202041 cli_runner.go:115] Run: docker system info --format "{{json .}}" I0816 14:07:42.738341 202041 info.go:263] docker info: {ID:63YF:7UB3:XY35:AJ3T:V4DH:3LU5:UWNZ:C3KS:Y2MC:L4BM:UYRV:WLCK Containers:15 ContainersRunning:0 ContainersPaused:0 ContainersStopped:15 Images:26 Driver:overlay2 DriverStatus:[[Backing Filesystem extfs] [Supports d_type true] [Native Overlay Diff true] [userxattr false]] SystemStatus: Plugins:{Volume:[local] Network:[bridge host ipvlan macvlan null overlay] Authorization: Log:[awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog]} MemoryLimit:true SwapLimit:true KernelMemory:true KernelMemoryTCP:true CPUCfsPeriod:true CPUCfsQuota:true CPUShares:true CPUSet:true PidsLimit:true IPv4Forwarding:true BridgeNfIptables:true BridgeNfIP6Tables:true Debug:false NFd:22 OomKillDisable:true NGoroutines:34 SystemTime:2021-08-16 14:07:42.72940272 +0530 IST LoggingDriver:json-file CgroupDriver:cgroupfs NEventsListener:0 KernelVersion:5.11.0-25-generic OperatingSystem:Ubuntu 20.04.2 LTS OSType:linux Architecture:x86_64 IndexServerAddress:https://index.docker.io/v1/ RegistryConfig:{AllowNondistributableArtifactsCIDRs:[] AllowNondistributableArtifactsHostnames:[] InsecureRegistryCIDRs:[127.0.0.0/8] IndexConfigs:{DockerIo:{Name:docker.io Mirrors:[] Secure:true Official:true}} Mirrors:[]} NCPU:4 MemTotal:16665108480 GenericResources: DockerRootDir:/var/lib/docker HTTPProxy: HTTPSProxy: NoProxy: Name:FDLDL064 Labels:[] ExperimentalBuild:false ServerVersion:20.10.7 ClusterStore: ClusterAdvertise: Runtimes:{Runc:{Path:runc}} DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID: Expected:} RuncCommit:{ID: Expected:} InitCommit:{ID: Expected:} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense: Warnings: ServerErrors:[] ClientInfo:{Debug:false Plugins:[] Warnings:}} I0816 14:07:42.738407 202041 start_flags.go:261] no existing cluster config was found, will generate one from the flags I0816 14:07:42.759051 202041 start_flags.go:342] Using suggested 3900MB memory alloc based on sys=15893MB, container=15893MB I0816 14:07:42.759121 202041 start_flags.go:669] Wait components to verify : map[apiserver:true system_pods:true] I0816 14:07:42.759134 202041 cni.go:93] Creating CNI manager for "" I0816 14:07:42.759139 202041 cni.go:167] CNI unnecessary in this configuration, recommending no CNI I0816 14:07:42.759142 202041 start_flags.go:275] config: {Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 Memory:3900 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.21.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false} I0816 14:07:42.762790 202041 out.go:165] 👍 Starting control plane node minikube in cluster minikube I0816 14:07:42.762869 202041 cache.go:117] Beginning downloading kic base image for docker with docker I0816 14:07:42.769290 202041 out.go:165] 🚜 Pulling base image ... I0816 14:07:42.769333 202041 preload.go:134] Checking if preload exists for k8s version v1.21.2 and runtime docker I0816 14:07:42.769376 202041 preload.go:150] Found local preload: /home/rseshadri/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v11-v1.21.2-docker-overlay2-amd64.tar.lz4 I0816 14:07:42.769380 202041 cache.go:56] Caching tarball of preloaded images I0816 14:07:42.769428 202041 image.go:75] Checking for gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 in local docker daemon I0816 14:07:42.769560 202041 preload.go:174] Found /home/rseshadri/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v11-v1.21.2-docker-overlay2-amd64.tar.lz4 in cache, skipping download I0816 14:07:42.769574 202041 cache.go:59] Finished verifying existence of preloaded tar for v1.21.2 on docker I0816 14:07:42.769872 202041 profile.go:148] Saving config to /home/rseshadri/.minikube/profiles/minikube/config.json ... I0816 14:07:42.769885 202041 lock.go:36] WriteFile acquiring /home/rseshadri/.minikube/profiles/minikube/config.json: {Name:mkf6684a51bf53a67befbbcfcbdbaaa517795e7e Clock:{} Delay:500ms Timeout:1m0s Cancel:} I0816 14:07:42.814546 202041 image.go:79] Found gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 in local docker daemon, skipping pull I0816 14:07:42.814557 202041 cache.go:139] gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 exists in daemon, skipping load I0816 14:07:42.814567 202041 cache.go:205] Successfully downloaded all kic artifacts I0816 14:07:42.814585 202041 start.go:313] acquiring machines lock for minikube: {Name:mk9eb9753f6d172b5bde2444a2583cf5bcb0bdf8 Clock:{} Delay:500ms Timeout:10m0s Cancel:} I0816 14:07:42.814670 202041 start.go:317] acquired machines lock for "minikube" in 75.148µs I0816 14:07:42.814693 202041 start.go:89] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 Memory:3900 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.21.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.21.2 ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false} &{Name: IP: Port:8443 KubernetesVersion:v1.21.2 ControlPlane:true Worker:true} I0816 14:07:42.814741 202041 start.go:126] createHost starting for "" (driver="docker") I0816 14:07:42.818350 202041 out.go:192] 🔥 Creating docker container (CPUs=2, Memory=3900MB) ... I0816 14:07:42.818537 202041 start.go:160] libmachine.API.Create for "minikube" (driver="docker") I0816 14:07:42.818552 202041 client.go:168] LocalClient.Create starting I0816 14:07:42.818596 202041 main.go:130] libmachine: Reading certificate data from /home/rseshadri/.minikube/certs/ca.pem I0816 14:07:42.818621 202041 main.go:130] libmachine: Decoding PEM data... I0816 14:07:42.818636 202041 main.go:130] libmachine: Parsing certificate... I0816 14:07:42.818723 202041 main.go:130] libmachine: Reading certificate data from /home/rseshadri/.minikube/certs/cert.pem I0816 14:07:42.818739 202041 main.go:130] libmachine: Decoding PEM data... I0816 14:07:42.818747 202041 main.go:130] libmachine: Parsing certificate... I0816 14:07:42.819009 202041 cli_runner.go:115] Run: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" W0816 14:07:42.854560 202041 cli_runner.go:162] docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1 I0816 14:07:42.854605 202041 network_create.go:255] running [docker network inspect minikube] to gather additional debugging logs... I0816 14:07:42.854616 202041 cli_runner.go:115] Run: docker network inspect minikube W0816 14:07:42.888780 202041 cli_runner.go:162] docker network inspect minikube returned with exit code 1 I0816 14:07:42.888794 202041 network_create.go:258] error running [docker network inspect minikube]: docker network inspect minikube: exit status 1 stdout: []

stderr:
Error: No such network: minikube
I0816 14:07:42.888801 202041 network_create.go:260] output of [docker network inspect minikube]: -- stdout --
[]

-- /stdout --
** stderr **
Error: No such network: minikube

** /stderr **
I0816 14:07:42.888853 202041 cli_runner.go:115] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0816 14:07:42.922837 202041 network.go:288] reserving subnet 192.168.49.0 for 1m0s: &{mu:{state:0 sema:0} read:{v:{m:map[] amended:true}} dirty:map[192.168.49.0:0xc0000c7028] misses:0}
I0816 14:07:42.922863 202041 network.go:235] using free private subnet 192.168.49.0/24: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:}}
I0816 14:07:42.922878 202041 network_create.go:106] attempt to create docker network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 1500 ...
I0816 14:07:42.922912 202041 cli_runner.go:115] Run: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=1500 --label=created_by.minikube.sigs.k8s.io=true minikube
I0816 14:07:43.039016 202041 network_create.go:90] docker network minikube 192.168.49.0/24 created
I0816 14:07:43.039040 202041 kic.go:106] calculated static IP "192.168.49.2" for the "minikube" container
I0816 14:07:43.039085 202041 cli_runner.go:115] Run: docker ps -a --format {{.Names}}
I0816 14:07:43.083367 202041 cli_runner.go:115] Run: docker volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
I0816 14:07:43.124482 202041 oci.go:102] Successfully created a docker volume minikube
I0816 14:07:43.124534 202041 cli_runner.go:115] Run: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 -d /var/lib
I0816 14:07:44.139895 202041 cli_runner.go:168] Completed: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 -d /var/lib: (1.01533568s)
I0816 14:07:44.139907 202041 oci.go:106] Successfully prepared a docker volume minikube
W0816 14:07:44.139928 202041 oci.go:135] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
W0816 14:07:44.139934 202041 oci.go:119] Your kernel does not support memory limit capabilities or the cgroup is not mounted.
I0816 14:07:44.139978 202041 cli_runner.go:115] Run: docker info --format "'{{json .SecurityOptions}}'"
I0816 14:07:44.140000 202041 preload.go:134] Checking if preload exists for k8s version v1.21.2 and runtime docker
I0816 14:07:44.140012 202041 kic.go:179] Starting extracting preloaded images to volume ...
I0816 14:07:44.140049 202041 cli_runner.go:115] Run: docker run --rm --entrypoint /usr/bin/tar -v /home/rseshadri/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v11-v1.21.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 -I lz4 -xf /preloaded.tar -C /extractDir
I0816 14:07:44.202540 202041 cli_runner.go:115] Run: docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var --security-opt apparmor=unconfined --cpus=2 -e container=docker --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79
I0816 14:07:44.920110 202041 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Running}}
I0816 14:07:44.979887 202041 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0816 14:07:45.054012 202041 cli_runner.go:115] Run: docker exec minikube stat /var/lib/dpkg/alternatives/iptables
I0816 14:07:45.258524 202041 oci.go:278] the created container "minikube" has a running status.
I0816 14:07:45.258546 202041 kic.go:210] Creating ssh key for kic: /home/rseshadri/.minikube/machines/minikube/id_rsa...
I0816 14:07:45.316482 202041 kic_runner.go:188] docker (temp): /home/rseshadri/.minikube/machines/minikube/id_rsa.pub --> /home/docker/.ssh/authorized_keys (381 bytes)
I0816 14:07:45.541289 202041 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0816 14:07:45.626588 202041 kic_runner.go:94] Run: chown docker:docker /home/docker/.ssh/authorized_keys
I0816 14:07:45.626598 202041 kic_runner.go:115] Args: [docker exec --privileged minikube chown docker:docker /home/docker/.ssh/authorized_keys]
I0816 14:07:54.673465 202041 cli_runner.go:168] Completed: docker run --rm --entrypoint /usr/bin/tar -v /home/rseshadri/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v11-v1.21.2-docker-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 -I lz4 -xf /preloaded.tar -C /extractDir: (10.533332284s)
I0816 14:07:54.673497 202041 kic.go:188] duration metric: took 10.533478 seconds to extract preloaded images to volume
I0816 14:07:54.673648 202041 cli_runner.go:115] Run: docker container inspect minikube --format={{.State.Status}}
I0816 14:07:54.728192 202041 machine.go:88] provisioning docker machine ...
I0816 14:07:54.728223 202041 ubuntu.go:169] provisioning hostname "minikube"
I0816 14:07:54.728269 202041 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0816 14:07:54.771078 202041 main.go:130] libmachine: Using SSH client type: native
I0816 14:07:54.771223 202041 main.go:130] libmachine: &{{{ 0 [] [] []} docker [0x8022e0] 0x8022a0 [] 0s} 127.0.0.1 49177 }
I0816 14:07:54.771231 202041 main.go:130] libmachine: About to run SSH command:
sudo hostname minikube && echo "minikube" | sudo tee /etc/hostname
I0816 14:07:55.025683 202041 main.go:130] libmachine: SSH cmd err, output: : minikube

I0816 14:07:55.025731 202041 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0816 14:07:55.073434 202041 main.go:130] libmachine: Using SSH client type: native
I0816 14:07:55.073550 202041 main.go:130] libmachine: &{{{ 0 [] [] []} docker [0x8022e0] 0x8022a0 [] 0s} 127.0.0.1 49177 }
I0816 14:07:55.073561 202041 main.go:130] libmachine: About to run SSH command:

	if ! grep -xq '.*\sminikube' /etc/hosts; then
		if grep -xq '127.0.1.1\s.*' /etc/hosts; then
			sudo sed -i 's/^127.0.1.1\s.*/127.0.1.1 minikube/g' /etc/hosts;
		else 
			echo '127.0.1.1 minikube' | sudo tee -a /etc/hosts; 
		fi
	fi

I0816 14:07:55.242160 202041 main.go:130] libmachine: SSH cmd err, output: :
I0816 14:07:55.242175 202041 ubuntu.go:175] set auth options {CertDir:/home/rseshadri/.minikube CaCertPath:/home/rseshadri/.minikube/certs/ca.pem CaPrivateKeyPath:/home/rseshadri/.minikube/certs/ca-key.pem CaCertRemotePath:/etc/docker/ca.pem ServerCertPath:/home/rseshadri/.minikube/machines/server.pem ServerKeyPath:/home/rseshadri/.minikube/machines/server-key.pem ClientKeyPath:/home/rseshadri/.minikube/certs/key.pem ServerCertRemotePath:/etc/docker/server.pem ServerKeyRemotePath:/etc/docker/server-key.pem ClientCertPath:/home/rseshadri/.minikube/certs/cert.pem ServerCertSANs:[] StorePath:/home/rseshadri/.minikube}
I0816 14:07:55.242186 202041 ubuntu.go:177] setting up certificates
I0816 14:07:55.242192 202041 provision.go:83] configureAuth start
I0816 14:07:55.242236 202041 cli_runner.go:115] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" minikube
I0816 14:07:55.296162 202041 provision.go:137] copyHostCerts
I0816 14:07:55.296205 202041 exec_runner.go:145] found /home/rseshadri/.minikube/key.pem, removing ...
I0816 14:07:55.296209 202041 exec_runner.go:190] rm: /home/rseshadri/.minikube/key.pem
I0816 14:07:55.296261 202041 exec_runner.go:152] cp: /home/rseshadri/.minikube/certs/key.pem --> /home/rseshadri/.minikube/key.pem (1675 bytes)
I0816 14:07:55.296327 202041 exec_runner.go:145] found /home/rseshadri/.minikube/ca.pem, removing ...
I0816 14:07:55.296331 202041 exec_runner.go:190] rm: /home/rseshadri/.minikube/ca.pem
I0816 14:07:55.296354 202041 exec_runner.go:152] cp: /home/rseshadri/.minikube/certs/ca.pem --> /home/rseshadri/.minikube/ca.pem (1086 bytes)
I0816 14:07:55.296404 202041 exec_runner.go:145] found /home/rseshadri/.minikube/cert.pem, removing ...
I0816 14:07:55.296407 202041 exec_runner.go:190] rm: /home/rseshadri/.minikube/cert.pem
I0816 14:07:55.296428 202041 exec_runner.go:152] cp: /home/rseshadri/.minikube/certs/cert.pem --> /home/rseshadri/.minikube/cert.pem (1131 bytes)
I0816 14:07:55.296468 202041 provision.go:111] generating server cert: /home/rseshadri/.minikube/machines/server.pem ca-key=/home/rseshadri/.minikube/certs/ca.pem private-key=/home/rseshadri/.minikube/certs/ca-key.pem org=rseshadri.minikube san=[192.168.49.2 127.0.0.1 localhost 127.0.0.1 minikube minikube]
I0816 14:07:55.463290 202041 provision.go:171] copyRemoteCerts
I0816 14:07:55.463363 202041 ssh_runner.go:149] Run: sudo mkdir -p /etc/docker /etc/docker /etc/docker
I0816 14:07:55.463407 202041 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0816 14:07:55.510555 202041 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49177 SSHKeyPath:/home/rseshadri/.minikube/machines/minikube/id_rsa Username:docker}
I0816 14:07:55.647288 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/certs/ca.pem --> /etc/docker/ca.pem (1086 bytes)
I0816 14:07:55.776337 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/machines/server.pem --> /etc/docker/server.pem (1208 bytes)
I0816 14:07:55.874150 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/machines/server-key.pem --> /etc/docker/server-key.pem (1679 bytes)
I0816 14:07:55.971601 202041 provision.go:86] duration metric: configureAuth took 729.401524ms
I0816 14:07:55.971613 202041 ubuntu.go:193] setting minikube options for container-runtime
I0816 14:07:55.971750 202041 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0816 14:07:56.006771 202041 main.go:130] libmachine: Using SSH client type: native
I0816 14:07:56.006887 202041 main.go:130] libmachine: &{{{ 0 [] [] []} docker [0x8022e0] 0x8022a0 [] 0s} 127.0.0.1 49177 }
I0816 14:07:56.006894 202041 main.go:130] libmachine: About to run SSH command:
df --output=fstype / | tail -n 1
I0816 14:07:56.260434 202041 main.go:130] libmachine: SSH cmd err, output: : overlay

I0816 14:07:56.260456 202041 ubuntu.go:71] root file system type: overlay
I0816 14:07:56.260885 202041 provision.go:308] Updating docker unit: /lib/systemd/system/docker.service ...
I0816 14:07:56.260997 202041 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0816 14:07:56.320806 202041 main.go:130] libmachine: Using SSH client type: native
I0816 14:07:56.320953 202041 main.go:130] libmachine: &{{{ 0 [] [] []} docker [0x8022e0] 0x8022a0 [] 0s} 127.0.0.1 49177 }
I0816 14:07:56.321029 202041 main.go:130] libmachine: About to run SSH command:
sudo mkdir -p /lib/systemd/system && printf %s "[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60

[Service]
Type=notify
Restart=on-failure

This file is a systemd drop-in unit that inherits from the base dockerd configuration.

The base configuration already specifies an 'ExecStart=...' command. The first directive

here is to clear out that command inherited from the base configuration. Without this,

the command from the base configuration and the command specified here are treated as

a sequence of commands, which is not the desired behavior, nor is it valid -- systemd

will catch this invalid input and refuse to start the service with an error like:

Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.

NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other

container runtimes. If left unlimited, it may result in OOM issues with MySQL.

ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID

Having non-zero Limit*s causes performance problems due to accounting overhead

in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

Uncomment TasksMax if your systemd version supports it.

Only systemd 226 and above support this version.

TasksMax=infinity
TimeoutStartSec=0

set delegate yes so that systemd does not reset the cgroups of docker containers

Delegate=yes

kill only the docker process, not all processes in the cgroup

KillMode=process

[Install]
WantedBy=multi-user.target
" | sudo tee /lib/systemd/system/docker.service.new
I0816 14:07:56.596635 202041 main.go:130] libmachine: SSH cmd err, output: : [Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
Requires=docker.socket
StartLimitBurst=3
StartLimitIntervalSec=60

[Service]
Type=notify
Restart=on-failure

This file is a systemd drop-in unit that inherits from the base dockerd configuration.

The base configuration already specifies an 'ExecStart=...' command. The first directive

here is to clear out that command inherited from the base configuration. Without this,

the command from the base configuration and the command specified here are treated as

a sequence of commands, which is not the desired behavior, nor is it valid -- systemd

will catch this invalid input and refuse to start the service with an error like:

Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.

NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other

container runtimes. If left unlimited, it may result in OOM issues with MySQL.

ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
ExecReload=/bin/kill -s HUP $MAINPID

Having non-zero Limit*s causes performance problems due to accounting overhead

in the kernel. We recommend using cgroups to do container-local accounting.

LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity

Uncomment TasksMax if your systemd version supports it.

Only systemd 226 and above support this version.

TasksMax=infinity
TimeoutStartSec=0

set delegate yes so that systemd does not reset the cgroups of docker containers

Delegate=yes

kill only the docker process, not all processes in the cgroup

KillMode=process

[Install]
WantedBy=multi-user.target

I0816 14:07:56.596704 202041 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0816 14:07:56.644217 202041 main.go:130] libmachine: Using SSH client type: native
I0816 14:07:56.644370 202041 main.go:130] libmachine: &{{{ 0 [] [] []} docker [0x8022e0] 0x8022a0 [] 0s} 127.0.0.1 49177 }
I0816 14:07:56.644386 202041 main.go:130] libmachine: About to run SSH command:
sudo diff -u /lib/systemd/system/docker.service /lib/systemd/system/docker.service.new || { sudo mv /lib/systemd/system/docker.service.new /lib/systemd/system/docker.service; sudo systemctl -f daemon-reload && sudo systemctl -f enable docker && sudo systemctl -f restart docker; }
I0816 14:07:59.417023 202041 main.go:130] libmachine: SSH cmd err, output: : --- /lib/systemd/system/docker.service 2021-06-02 11:54:50.000000000 +0000
+++ /lib/systemd/system/docker.service.new 2021-08-16 08:37:56.594056366 +0000
@@ -1,30 +1,32 @@
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
+BindsTo=containerd.service
After=network-online.target firewalld.service containerd.service
Wants=network-online.target
-Requires=docker.socket containerd.service
+Requires=docker.socket
+StartLimitBurst=3
+StartLimitIntervalSec=60

[Service]
Type=notify
-# the default is not to use systemd for cgroups because the delegate issues still
-# exists and systemd currently does not support the cgroup feature set required
-# for containers run by docker
-ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
-ExecReload=/bin/kill -s HUP $MAINPID
-TimeoutSec=0
-RestartSec=2
-Restart=always

-# Note that StartLimit* options were moved from "Service" to "Unit" in systemd 229.
-# Both the old, and new location are accepted by systemd 229 and up, so using the old location
-# to make them work for either version of systemd.
-StartLimitBurst=3
+Restart=on-failure

-# Note that StartLimitInterval was renamed to StartLimitIntervalSec in systemd 230.
-# Both the old, and new name are accepted by systemd 230 and up, so using the old name to make
-# this option work for either version of systemd.
-StartLimitInterval=60s
+
+
+# This file is a systemd drop-in unit that inherits from the base dockerd configuration.
+# The base configuration already specifies an 'ExecStart=...' command. The first directive
+# here is to clear out that command inherited from the base configuration. Without this,
+# the command from the base configuration and the command specified here are treated as
+# a sequence of commands, which is not the desired behavior, nor is it valid -- systemd
+# will catch this invalid input and refuse to start the service with an error like:
+# Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.
+
+# NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other
+# container runtimes. If left unlimited, it may result in OOM issues with MySQL.
+ExecStart=
+ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --default-ulimit=nofile=1048576:1048576 --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=docker --insecure-registry 10.96.0.0/12
+ExecReload=/bin/kill -s HUP $MAINPID

Having non-zero Limit*s causes performance problems due to accounting overhead

in the kernel. We recommend using cgroups to do container-local accounting.

@@ -32,16 +34,16 @@
LimitNPROC=infinity
LimitCORE=infinity

-# Comment TasksMax if your systemd version does not support it.
-# Only systemd 226 and above support this option.
+# Uncomment TasksMax if your systemd version supports it.
+# Only systemd 226 and above support this version.
TasksMax=infinity
+TimeoutStartSec=0

set delegate yes so that systemd does not reset the cgroups of docker containers

Delegate=yes

kill only the docker process, not all processes in the cgroup

KillMode=process
-OOMScoreAdjust=-500

[Install]
WantedBy=multi-user.target
Synchronizing state of docker.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable docker

I0816 14:07:59.417063 202041 machine.go:91] provisioned docker machine in 4.688859388s
I0816 14:07:59.417070 202041 client.go:171] LocalClient.Create took 16.598515625s
I0816 14:07:59.417078 202041 start.go:168] duration metric: libmachine.API.Create for "minikube" took 16.598540305s
I0816 14:07:59.417082 202041 start.go:267] post-start starting for "minikube" (driver="docker")
I0816 14:07:59.417085 202041 start.go:277] creating required directories: [/etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs]
I0816 14:07:59.417136 202041 ssh_runner.go:149] Run: sudo mkdir -p /etc/kubernetes/addons /etc/kubernetes/manifests /var/tmp/minikube /var/lib/minikube /var/lib/minikube/certs /var/lib/minikube/images /var/lib/minikube/binaries /tmp/gvisor /usr/share/ca-certificates /etc/ssl/certs
I0816 14:07:59.417170 202041 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0816 14:07:59.459843 202041 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49177 SSHKeyPath:/home/rseshadri/.minikube/machines/minikube/id_rsa Username:docker}
I0816 14:07:59.629652 202041 ssh_runner.go:149] Run: cat /etc/os-release
I0816 14:07:59.654861 202041 main.go:130] libmachine: Couldn't set key PRIVACY_POLICY_URL, no corresponding struct field found
I0816 14:07:59.654886 202041 main.go:130] libmachine: Couldn't set key VERSION_CODENAME, no corresponding struct field found
I0816 14:07:59.654892 202041 main.go:130] libmachine: Couldn't set key UBUNTU_CODENAME, no corresponding struct field found
I0816 14:07:59.654896 202041 info.go:137] Remote host: Ubuntu 20.04.2 LTS
I0816 14:07:59.654902 202041 filesync.go:126] Scanning /home/rseshadri/.minikube/addons for local assets ...
I0816 14:07:59.654940 202041 filesync.go:126] Scanning /home/rseshadri/.minikube/files for local assets ...
I0816 14:07:59.654966 202041 start.go:270] post-start completed in 237.879908ms
I0816 14:07:59.655225 202041 cli_runner.go:115] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" minikube
I0816 14:07:59.689534 202041 profile.go:148] Saving config to /home/rseshadri/.minikube/profiles/minikube/config.json ...
I0816 14:07:59.689720 202041 ssh_runner.go:149] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0816 14:07:59.689745 202041 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0816 14:07:59.724134 202041 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49177 SSHKeyPath:/home/rseshadri/.minikube/machines/minikube/id_rsa Username:docker}
I0816 14:07:59.867093 202041 start.go:129] duration metric: createHost completed in 17.052340655s
I0816 14:07:59.867103 202041 start.go:80] releasing machines lock for "minikube", held for 17.052415539s
I0816 14:07:59.867174 202041 cli_runner.go:115] Run: docker container inspect -f "{{range .NetworkSettings.Networks}}{{.IPAddress}},{{.GlobalIPv6Address}}{{end}}" minikube
I0816 14:07:59.900775 202041 ssh_runner.go:149] Run: systemctl --version
I0816 14:07:59.900807 202041 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0816 14:07:59.900853 202041 ssh_runner.go:149] Run: curl -sS -m 2 https://k8s.gcr.io/
I0816 14:07:59.900892 202041 cli_runner.go:115] Run: docker container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0816 14:07:59.987285 202041 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49177 SSHKeyPath:/home/rseshadri/.minikube/machines/minikube/id_rsa Username:docker}
I0816 14:07:59.987307 202041 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:49177 SSHKeyPath:/home/rseshadri/.minikube/machines/minikube/id_rsa Username:docker}
I0816 14:08:00.110564 202041 ssh_runner.go:149] Run: sudo systemctl is-active --quiet service containerd
I0816 14:08:00.519782 202041 ssh_runner.go:149] Run: sudo systemctl cat docker.service
I0816 14:08:00.567551 202041 cruntime.go:249] skipping containerd shutdown because we are bound to it
I0816 14:08:00.567616 202041 ssh_runner.go:149] Run: sudo systemctl is-active --quiet service crio
I0816 14:08:00.607289 202041 ssh_runner.go:149] Run: /bin/bash -c "sudo mkdir -p /etc && printf %s "runtime-endpoint: unix:///var/run/dockershim.sock
image-endpoint: unix:///var/run/dockershim.sock
" | sudo tee /etc/crictl.yaml"
I0816 14:08:00.686613 202041 ssh_runner.go:149] Run: sudo systemctl unmask docker.service
I0816 14:08:00.929231 202041 ssh_runner.go:149] Run: sudo systemctl enable docker.socket
I0816 14:08:01.199685 202041 ssh_runner.go:149] Run: sudo systemctl cat docker.service
I0816 14:08:01.246120 202041 ssh_runner.go:149] Run: sudo systemctl daemon-reload
I0816 14:08:01.525694 202041 ssh_runner.go:149] Run: sudo systemctl start docker
I0816 14:08:01.578797 202041 ssh_runner.go:149] Run: docker version --format {{.Server.Version}}
I0816 14:08:01.666758 202041 ssh_runner.go:149] Run: docker version --format {{.Server.Version}}
I0816 14:08:01.759831 202041 out.go:192] 🐳 Preparing Kubernetes v1.21.2 on Docker 20.10.7 ...
I0816 14:08:01.760032 202041 cli_runner.go:115] Run: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0816 14:08:01.818969 202041 ssh_runner.go:149] Run: grep 192.168.49.1 host.minikube.internal$ /etc/hosts
I0816 14:08:01.842549 202041 ssh_runner.go:149] Run: /bin/bash -c "{ grep -v $'\thost.minikube.internal$' "/etc/hosts"; echo "192.168.49.1 host.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0816 14:08:01.906115 202041 preload.go:134] Checking if preload exists for k8s version v1.21.2 and runtime docker
I0816 14:08:01.906156 202041 ssh_runner.go:149] Run: docker images --format {{.Repository}}:{{.Tag}}
I0816 14:08:01.962834 202041 docker.go:535] Got preloaded images: -- stdout --
k8s.gcr.io/kube-apiserver:v1.21.2
k8s.gcr.io/kube-proxy:v1.21.2
k8s.gcr.io/kube-scheduler:v1.21.2
k8s.gcr.io/kube-controller-manager:v1.21.2
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/pause:3.4.1
kubernetesui/dashboard:v2.1.0
k8s.gcr.io/coredns/coredns:v1.8.0
k8s.gcr.io/etcd:3.4.13-0
kubernetesui/metrics-scraper:v1.0.4

-- /stdout --
I0816 14:08:01.962844 202041 docker.go:466] Images already preloaded, skipping extraction
I0816 14:08:01.962881 202041 ssh_runner.go:149] Run: docker images --format {{.Repository}}:{{.Tag}}
I0816 14:08:02.020272 202041 docker.go:535] Got preloaded images: -- stdout --
k8s.gcr.io/kube-apiserver:v1.21.2
k8s.gcr.io/kube-controller-manager:v1.21.2
k8s.gcr.io/kube-scheduler:v1.21.2
k8s.gcr.io/kube-proxy:v1.21.2
gcr.io/k8s-minikube/storage-provisioner:v5
k8s.gcr.io/pause:3.4.1
kubernetesui/dashboard:v2.1.0
k8s.gcr.io/coredns/coredns:v1.8.0
k8s.gcr.io/etcd:3.4.13-0
kubernetesui/metrics-scraper:v1.0.4

-- /stdout --
I0816 14:08:02.020282 202041 cache_images.go:74] Images are preloaded, skipping loading
I0816 14:08:02.020320 202041 ssh_runner.go:149] Run: docker info --format {{.CgroupDriver}}
I0816 14:08:02.155333 202041 cni.go:93] Creating CNI manager for ""
I0816 14:08:02.155342 202041 cni.go:167] CNI unnecessary in this configuration, recommending no CNI
I0816 14:08:02.155347 202041 kubeadm.go:87] Using pod CIDR: 10.244.0.0/16
I0816 14:08:02.155356 202041 kubeadm.go:153] kubeadm options: {CertDir:/var/lib/minikube/certs ServiceCIDR:10.96.0.0/12 PodSubnet:10.244.0.0/16 AdvertiseAddress:192.168.49.2 APIServerPort:8443 KubernetesVersion:v1.21.2 EtcdDataDir:/var/lib/minikube/etcd EtcdExtraArgs:map[] ClusterName:minikube NodeName:minikube DNSDomain:cluster.local CRISocket:/var/run/dockershim.sock ImageRepository: ComponentOptions:[{Component:apiServer ExtraArgs:map[enable-admission-plugins:NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota] Pairs:map[certSANs:["127.0.0.1", "localhost", "192.168.49.2"]]} {Component:controllerManager ExtraArgs:map[allocate-node-cidrs:true leader-elect:false] Pairs:map[]} {Component:scheduler ExtraArgs:map[leader-elect:false] Pairs:map[]}] FeatureArgs:map[] NoTaintMaster:true NodeIP:192.168.49.2 CgroupDriver:cgroupfs ClientCAFile:/var/lib/minikube/certs/ca.crt StaticPodPath:/etc/kubernetes/manifests ControlPlaneAddress:control-plane.minikube.internal KubeProxyOptions:map[]}
I0816 14:08:02.155440 202041 kubeadm.go:157] kubeadm config:
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
advertiseAddress: 192.168.49.2
bindPort: 8443
bootstrapTokens:

groups:
- system:bootstrappers:kubeadm:default-node-token
  ttl: 24h0m0s
  usages:
- signing
- authentication
  nodeRegistration:
  criSocket: /var/run/dockershim.sock
  name: "minikube"
  kubeletExtraArgs:
  node-ip: 192.168.49.2
  taints: []

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
apiServer:
certSANs: ["127.0.0.1", "localhost", "192.168.49.2"]
extraArgs:
enable-admission-plugins: "NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"
controllerManager:
extraArgs:
allocate-node-cidrs: "true"
leader-elect: "false"
scheduler:
extraArgs:
leader-elect: "false"
certificatesDir: /var/lib/minikube/certs
clusterName: mk
controlPlaneEndpoint: control-plane.minikube.internal:8443
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/minikube/etcd
extraArgs:
proxy-refresh-interval: "70000"
kubernetesVersion: v1.21.2
networking:
dnsDomain: cluster.local
podSubnet: "10.244.0.0/16"
serviceSubnet: 10.96.0.0/12

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
x509:
clientCAFile: /var/lib/minikube/certs/ca.crt
cgroupDriver: cgroupfs
clusterDomain: "cluster.local"

disable disk resource management by default

imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests

apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
clusterCIDR: "10.244.0.0/16"
metricsBindAddress: 0.0.0.0:10249
conntrack:
maxPerCore: 0

I0816 14:08:02.155498 202041 kubeadm.go:909] kubelet [Unit]
Wants=docker.socket

[Service]
ExecStart=
ExecStart=/var/lib/minikube/binaries/v1.21.2/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --config=/var/lib/kubelet/config.yaml --container-runtime=docker --hostname-override=minikube --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=192.168.49.2

[Install]
config:
{KubernetesVersion:v1.21.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:}
I0816 14:08:02.155535 202041 ssh_runner.go:149] Run: sudo ls /var/lib/minikube/binaries/v1.21.2
I0816 14:08:02.196445 202041 binaries.go:44] Found k8s binaries, skipping transfer
I0816 14:08:02.196483 202041 ssh_runner.go:149] Run: sudo mkdir -p /etc/systemd/system/kubelet.service.d /lib/systemd/system /var/tmp/minikube
I0816 14:08:02.233094 202041 ssh_runner.go:316] scp memory --> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf (334 bytes)
I0816 14:08:02.301884 202041 ssh_runner.go:316] scp memory --> /lib/systemd/system/kubelet.service (352 bytes)
I0816 14:08:02.371809 202041 ssh_runner.go:316] scp memory --> /var/tmp/minikube/kubeadm.yaml.new (1867 bytes)
I0816 14:08:02.435891 202041 ssh_runner.go:149] Run: grep 192.168.49.2 control-plane.minikube.internal$ /etc/hosts
I0816 14:08:02.459067 202041 ssh_runner.go:149] Run: /bin/bash -c "{ grep -v $'\tcontrol-plane.minikube.internal$' "/etc/hosts"; echo "192.168.49.2 control-plane.minikube.internal"; } > /tmp/h.$$; sudo cp /tmp/h.$$ "/etc/hosts""
I0816 14:08:02.522363 202041 certs.go:52] Setting up /home/rseshadri/.minikube/profiles/minikube for IP: 192.168.49.2
I0816 14:08:02.522397 202041 certs.go:179] skipping minikubeCA CA generation: /home/rseshadri/.minikube/ca.key
I0816 14:08:02.522412 202041 certs.go:179] skipping proxyClientCA CA generation: /home/rseshadri/.minikube/proxy-client-ca.key
I0816 14:08:02.522446 202041 certs.go:294] generating minikube-user signed cert: /home/rseshadri/.minikube/profiles/minikube/client.key
I0816 14:08:02.522450 202041 crypto.go:69] Generating cert /home/rseshadri/.minikube/profiles/minikube/client.crt with IP's: []
I0816 14:08:02.742439 202041 crypto.go:157] Writing cert to /home/rseshadri/.minikube/profiles/minikube/client.crt ...
I0816 14:08:02.742477 202041 lock.go:36] WriteFile acquiring /home/rseshadri/.minikube/profiles/minikube/client.crt: {Name:mka25b8dfe6da1dd652fd59311787990fd05efd3 Clock:{} Delay:500ms Timeout:1m0s Cancel:}
I0816 14:08:02.742670 202041 crypto.go:165] Writing key to /home/rseshadri/.minikube/profiles/minikube/client.key ...
I0816 14:08:02.742681 202041 lock.go:36] WriteFile acquiring /home/rseshadri/.minikube/profiles/minikube/client.key: {Name:mk7e9ceedb068f2b0d0843bed28e8546418acedb Clock:{} Delay:500ms Timeout:1m0s Cancel:}
I0816 14:08:02.742780 202041 certs.go:294] generating minikube signed cert: /home/rseshadri/.minikube/profiles/minikube/apiserver.key.dd3b5fb2
I0816 14:08:02.742784 202041 crypto.go:69] Generating cert /home/rseshadri/.minikube/profiles/minikube/apiserver.crt.dd3b5fb2 with IP's: [192.168.49.2 10.96.0.1 127.0.0.1 10.0.0.1]
I0816 14:08:02.863474 202041 crypto.go:157] Writing cert to /home/rseshadri/.minikube/profiles/minikube/apiserver.crt.dd3b5fb2 ...
I0816 14:08:02.863484 202041 lock.go:36] WriteFile acquiring /home/rseshadri/.minikube/profiles/minikube/apiserver.crt.dd3b5fb2: {Name:mk60d063bec22fa616627410348563d568dcc50f Clock:{} Delay:500ms Timeout:1m0s Cancel:}
I0816 14:08:02.863601 202041 crypto.go:165] Writing key to /home/rseshadri/.minikube/profiles/minikube/apiserver.key.dd3b5fb2 ...
I0816 14:08:02.863607 202041 lock.go:36] WriteFile acquiring /home/rseshadri/.minikube/profiles/minikube/apiserver.key.dd3b5fb2: {Name:mkb5799fb1f22e98e82cb75c6852708ea2b88868 Clock:{} Delay:500ms Timeout:1m0s Cancel:}
I0816 14:08:02.863663 202041 certs.go:305] copying /home/rseshadri/.minikube/profiles/minikube/apiserver.crt.dd3b5fb2 -> /home/rseshadri/.minikube/profiles/minikube/apiserver.crt
I0816 14:08:02.863710 202041 certs.go:309] copying /home/rseshadri/.minikube/profiles/minikube/apiserver.key.dd3b5fb2 -> /home/rseshadri/.minikube/profiles/minikube/apiserver.key
I0816 14:08:02.863750 202041 certs.go:294] generating aggregator signed cert: /home/rseshadri/.minikube/profiles/minikube/proxy-client.key
I0816 14:08:02.863753 202041 crypto.go:69] Generating cert /home/rseshadri/.minikube/profiles/minikube/proxy-client.crt with IP's: []
I0816 14:08:02.957187 202041 crypto.go:157] Writing cert to /home/rseshadri/.minikube/profiles/minikube/proxy-client.crt ...
I0816 14:08:02.957197 202041 lock.go:36] WriteFile acquiring /home/rseshadri/.minikube/profiles/minikube/proxy-client.crt: {Name:mke4e49b3d8ccd8d8b01db8f87825bbff4624c24 Clock:{} Delay:500ms Timeout:1m0s Cancel:}
I0816 14:08:02.957306 202041 crypto.go:165] Writing key to /home/rseshadri/.minikube/profiles/minikube/proxy-client.key ...
I0816 14:08:02.957311 202041 lock.go:36] WriteFile acquiring /home/rseshadri/.minikube/profiles/minikube/proxy-client.key: {Name:mk8edbe22e8858e7690765fc132af2735d30dccb Clock:{} Delay:500ms Timeout:1m0s Cancel:}
I0816 14:08:02.957439 202041 certs.go:369] found cert: /home/rseshadri/.minikube/certs/home/rseshadri/.minikube/certs/ca-key.pem (1675 bytes)
I0816 14:08:02.957464 202041 certs.go:369] found cert: /home/rseshadri/.minikube/certs/home/rseshadri/.minikube/certs/ca.pem (1086 bytes)
I0816 14:08:02.957482 202041 certs.go:369] found cert: /home/rseshadri/.minikube/certs/home/rseshadri/.minikube/certs/cert.pem (1131 bytes)
I0816 14:08:02.957500 202041 certs.go:369] found cert: /home/rseshadri/.minikube/certs/home/rseshadri/.minikube/certs/key.pem (1675 bytes)
I0816 14:08:02.958184 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/profiles/minikube/apiserver.crt --> /var/lib/minikube/certs/apiserver.crt (1399 bytes)
I0816 14:08:03.045735 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/profiles/minikube/apiserver.key --> /var/lib/minikube/certs/apiserver.key (1675 bytes)
I0816 14:08:03.135713 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/profiles/minikube/proxy-client.crt --> /var/lib/minikube/certs/proxy-client.crt (1147 bytes)
I0816 14:08:03.226138 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/profiles/minikube/proxy-client.key --> /var/lib/minikube/certs/proxy-client.key (1679 bytes)
I0816 14:08:03.314632 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/ca.crt --> /var/lib/minikube/certs/ca.crt (1111 bytes)
I0816 14:08:03.401961 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/ca.key --> /var/lib/minikube/certs/ca.key (1679 bytes)
I0816 14:08:03.494163 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/proxy-client-ca.crt --> /var/lib/minikube/certs/proxy-client-ca.crt (1119 bytes)
I0816 14:08:03.591570 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/proxy-client-ca.key --> /var/lib/minikube/certs/proxy-client-ca.key (1679 bytes)
I0816 14:08:03.680700 202041 ssh_runner.go:316] scp /home/rseshadri/.minikube/ca.crt --> /usr/share/ca-certificates/minikubeCA.pem (1111 bytes)
I0816 14:08:03.774548 202041 ssh_runner.go:316] scp memory --> /var/lib/minikube/kubeconfig (738 bytes)
I0816 14:08:03.843380 202041 ssh_runner.go:149] Run: openssl version
I0816 14:08:03.867972 202041 ssh_runner.go:149] Run: sudo /bin/bash -c "test -s /usr/share/ca-certificates/minikubeCA.pem && ln -fs /usr/share/ca-certificates/minikubeCA.pem /etc/ssl/certs/minikubeCA.pem"
I0816 14:08:03.917171 202041 ssh_runner.go:149] Run: ls -la /usr/share/ca-certificates/minikubeCA.pem
I0816 14:08:03.940902 202041 certs.go:410] hashing: -rw-r--r-- 1 root root 1111 Jul 8 04:17 /usr/share/ca-certificates/minikubeCA.pem
I0816 14:08:03.940944 202041 ssh_runner.go:149] Run: openssl x509 -hash -noout -in /usr/share/ca-certificates/minikubeCA.pem
I0816 14:08:03.969153 202041 ssh_runner.go:149] Run: sudo /bin/bash -c "test -L /etc/ssl/certs/b5213941.0 || ln -fs /etc/ssl/certs/minikubeCA.pem /etc/ssl/certs/b5213941.0"
I0816 14:08:04.018313 202041 kubeadm.go:390] StartCluster: {Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.25@sha256:6f936e3443b95cd918d77623bf7b595653bb382766e280290a02b4a349e88b79 Memory:3900 CPUs:2 DiskSize:20000 VMDriver: Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.21.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.49.2 Port:8443 KubernetesVersion:v1.21.2 ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false}
I0816 14:08:04.018385 202041 ssh_runner.go:149] Run: docker ps --filter status=paused --filter=name=k8s_.*(kube-system) --format={{.ID}}
I0816 14:08:04.070214 202041 ssh_runner.go:149] Run: sudo ls /var/lib/kubelet/kubeadm-flags.env /var/lib/kubelet/config.yaml /var/lib/minikube/etcd
I0816 14:08:04.110014 202041 ssh_runner.go:149] Run: sudo cp /var/tmp/minikube/kubeadm.yaml.new /var/tmp/minikube/kubeadm.yaml
I0816 14:08:04.147255 202041 kubeadm.go:220] ignoring SystemVerification for kubeadm because of docker driver
I0816 14:08:04.147306 202041 ssh_runner.go:149] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0816 14:08:04.183312 202041 kubeadm.go:151] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:

stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0816 14:08:04.183332 202041 ssh_runner.go:240] Start: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.21.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
W0816 14:09:06.217236 202041 out.go:230] 💢 initialization failed, will try again: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.21.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.21.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 5.11.0-25-generic
DOCKER_VERSION: 20.10.7
DOCKER_GRAPH_DRIVER: overlay2
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled

stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.11.0-25-generic\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': signal: killed
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

I0816 14:09:06.217350 202041 ssh_runner.go:149] Run: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.21.2:$PATH kubeadm reset --cri-socket /var/run/dockershim.sock --force"
I0816 14:09:06.398485 202041 ssh_runner.go:149] Run: sudo systemctl stop -f kubelet
I0816 14:09:06.437162 202041 ssh_runner.go:149] Run: docker ps -a --filter=name=k8s_.*(kube-system) --format={{.ID}}
I0816 14:09:06.489333 202041 kubeadm.go:220] ignoring SystemVerification for kubeadm because of docker driver
I0816 14:09:06.489367 202041 ssh_runner.go:149] Run: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf
I0816 14:09:06.526594 202041 kubeadm.go:151] config check failed, skipping stale config cleanup: sudo ls -la /etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf: Process exited with status 2
stdout:

stderr:
ls: cannot access '/etc/kubernetes/admin.conf': No such file or directory
ls: cannot access '/etc/kubernetes/kubelet.conf': No such file or directory
ls: cannot access '/etc/kubernetes/controller-manager.conf': No such file or directory
ls: cannot access '/etc/kubernetes/scheduler.conf': No such file or directory
I0816 14:09:06.526625 202041 ssh_runner.go:240] Start: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.21.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables"
I0816 14:10:07.650951 202041 kubeadm.go:392] StartCluster complete in 2m3.632639842s
I0816 14:10:07.651002 202041 ssh_runner.go:149] Run: docker ps -a --filter=name=k8s_kube-apiserver --format={{.ID}}
I0816 14:10:07.716785 202041 logs.go:270] 0 containers: []
W0816 14:10:07.716797 202041 logs.go:272] No container was found matching "kube-apiserver"
I0816 14:10:07.716836 202041 ssh_runner.go:149] Run: docker ps -a --filter=name=k8s_etcd --format={{.ID}}
I0816 14:10:07.769475 202041 logs.go:270] 0 containers: []
W0816 14:10:07.769483 202041 logs.go:272] No container was found matching "etcd"
I0816 14:10:07.769517 202041 ssh_runner.go:149] Run: docker ps -a --filter=name=k8s_coredns --format={{.ID}}
I0816 14:10:07.822992 202041 logs.go:270] 0 containers: []
W0816 14:10:07.823027 202041 logs.go:272] No container was found matching "coredns"
I0816 14:10:07.823073 202041 ssh_runner.go:149] Run: docker ps -a --filter=name=k8s_kube-scheduler --format={{.ID}}
I0816 14:10:07.891713 202041 logs.go:270] 0 containers: []
W0816 14:10:07.891722 202041 logs.go:272] No container was found matching "kube-scheduler"
I0816 14:10:07.891794 202041 ssh_runner.go:149] Run: docker ps -a --filter=name=k8s_kube-proxy --format={{.ID}}
I0816 14:10:07.947744 202041 logs.go:270] 0 containers: []
W0816 14:10:07.947753 202041 logs.go:272] No container was found matching "kube-proxy"
I0816 14:10:07.947830 202041 ssh_runner.go:149] Run: docker ps -a --filter=name=k8s_kubernetes-dashboard --format={{.ID}}
I0816 14:10:07.999969 202041 logs.go:270] 0 containers: []
W0816 14:10:07.999978 202041 logs.go:272] No container was found matching "kubernetes-dashboard"
I0816 14:10:08.000014 202041 ssh_runner.go:149] Run: docker ps -a --filter=name=k8s_storage-provisioner --format={{.ID}}
I0816 14:10:08.058055 202041 logs.go:270] 0 containers: []
W0816 14:10:08.058064 202041 logs.go:272] No container was found matching "storage-provisioner"
I0816 14:10:08.058098 202041 ssh_runner.go:149] Run: docker ps -a --filter=name=k8s_kube-controller-manager --format={{.ID}}
I0816 14:10:08.112316 202041 logs.go:270] 0 containers: []
W0816 14:10:08.112324 202041 logs.go:272] No container was found matching "kube-controller-manager"
I0816 14:10:08.112330 202041 logs.go:123] Gathering logs for kubelet ...
I0816 14:10:08.112336 202041 ssh_runner.go:149] Run: /bin/bash -c "sudo journalctl -u kubelet -n 400"
I0816 14:10:08.162297 202041 logs.go:123] Gathering logs for dmesg ...
I0816 14:10:08.162308 202041 ssh_runner.go:149] Run: /bin/bash -c "sudo dmesg -PH -L=never --level warn,err,crit,alert,emerg | tail -n 400"
I0816 14:10:08.232431 202041 logs.go:123] Gathering logs for describe nodes ...
I0816 14:10:08.232442 202041 ssh_runner.go:149] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.21.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig"
W0816 14:10:08.409936 202041 logs.go:130] failed describe nodes: command: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.21.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig" /bin/bash -c "sudo /var/lib/minikube/binaries/v1.21.2/kubectl describe nodes --kubeconfig=/var/lib/minikube/kubeconfig": Process exited with status 1
stdout:

stderr:
The connection to the server localhost:8443 was refused - did you specify the right host or port?
output:
** stderr **
The connection to the server localhost:8443 was refused - did you specify the right host or port?

** /stderr **
I0816 14:10:08.409945 202041 logs.go:123] Gathering logs for Docker ...
I0816 14:10:08.409952 202041 ssh_runner.go:149] Run: /bin/bash -c "sudo journalctl -u docker -n 400"
I0816 14:10:08.464919 202041 logs.go:123] Gathering logs for container status ...
I0816 14:10:08.464930 202041 ssh_runner.go:149] Run: /bin/bash -c "sudo which crictl || echo crictl ps -a || sudo docker ps -a"
I0816 14:10:10.616627 202041 ssh_runner.go:189] Completed: /bin/bash -c "sudo which crictl || echo crictl ps -a || sudo docker ps -a": (2.15168587s)
W0816 14:10:10.616756 202041 out.go:359] Error starting cluster: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.21.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.21.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 5.11.0-25-generic
DOCKER_VERSION: 20.10.7
DOCKER_GRAPH_DRIVER: overlay2
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled

stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.11.0-25-generic\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': signal: killed
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher
W0816 14:10:10.616766 202041 out.go:230]
W0816 14:10:10.616870 202041 out.go:230] 💣 Error starting cluster: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.21.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.21.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 5.11.0-25-generic
DOCKER_VERSION: 20.10.7
DOCKER_GRAPH_DRIVER: overlay2
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled

stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.11.0-25-generic\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': signal: killed
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

W0816 14:10:10.616903 202041 out.go:230]
W0816 14:10:10.617560 202041 out.go:230] ╭────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please attach the following file to the GitHub issue: │
│ - /home/rseshadri/.minikube/logs/lastStart.txt │
│ │
╰────────────────────────────────────────────────────────────────────╯
I0816 14:10:10.624257 202041 out.go:165]
W0816 14:10:10.624388 202041 out.go:230] ❌ Exiting due to GUEST_START: wait: /bin/bash -c "sudo env PATH=/var/lib/minikube/binaries/v1.21.2:$PATH kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.21.2
[preflight] Running pre-flight checks
[preflight] The system verification failed. Printing the output from the verification:
KERNEL_VERSION: 5.11.0-25-generic
DOCKER_VERSION: 20.10.7
DOCKER_GRAPH_DRIVER: overlay2
OS: Linux
CGROUPS_CPU: enabled
CGROUPS_CPUACCT: enabled
CGROUPS_CPUSET: enabled
CGROUPS_DEVICES: enabled
CGROUPS_FREEZER: enabled
CGROUPS_MEMORY: enabled
CGROUPS_PIDS: enabled
CGROUPS_HUGETLB: enabled

stderr:
[WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING SystemVerification]: failed to parse kernel config: unable to load kernel module: "configs", output: "modprobe: FATAL: Module configs not found in directory /lib/modules/5.11.0-25-generic\n", err: exit status 1
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': signal: killed
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

W0816 14:10:10.624415 202041 out.go:230]
W0816 14:10:10.625059 202041 out.go:230] ╭────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please attach the following file to the GitHub issue: │
│ - /home/rseshadri/.minikube/logs/lastStart.txt │
│ │
╰────────────────────────────────────────────────────────────────────╯

Let me know if any other details are required from my end.

The text was updated successfully, but these errors were encountered:

Rits333 · 2021-08-20T05:10:32Z

Hey,
I use the minikube on the virtual box
minikube start --kubernetes-version=v1.20.0 --driver=virtualbox
I use this command if it is helpful to you can go with it and let me know that on which platform you using minikube on Docker or Linux or virtual box.
and if it helps to you please let me know.

spowelljr · 2021-10-06T18:51:22Z

Hi @rams3sh, thanks for reporting your issue with minikube!

First recommendation, I'd suggest updating to the latest version of minikube (v1.23.2).

Could you please try running minikube delete --all and then minikube start --driver=docker --force-systemd=true and let me know if that solves your issue, thanks!

spowelljr · 2021-10-20T18:19:15Z

Hi @rams3sh, we haven't heard back from you, do you still have this issue?
There isn't enough information in this issue to make it actionable, and a long enough duration has passed, so this issue is likely difficult to replicate.

I will close this issue for now but feel free to reopen when you feel ready to provide more information.

DPS0340 · 2021-11-13T17:58:58Z

Hi! I'm struggling with the same errors on macOS m1.
minikube works by ignoring this error message, but custom start options like --extra-config=apiserver.service-node-port-range= won't work.

I think that kubelet couldn't be installed on macos. is it related?

Command to reproduce

minikube start

Main error messages

❌  Exiting due to GUEST_START: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.22.3:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml  --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks

stderr:
        [WARNING Swap]: running with swap on is not supported. Please disable swap
        [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
        [ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': exit status 255
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

DPS0340 · 2021-11-13T18:17:56Z

I was wrong. minikube doesn't start. I was confused with docker desktop k8s.

chris-lsn · 2021-11-28T16:42:23Z

I have this issue on m1 mac.

vimalkansal · 2022-01-03T00:39:47Z

I am also having the same issue. My host system is :

Model Name: MacBook Pro
Chip: Apple M1 Max

And this is the complete output
minikube start --driver=docker --force-systemd=true
😄 minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨ Using the docker driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔥 Creating docker container (CPUs=2, Memory=1988MB) ...
🐳 Preparing Kubernetes v1.22.3 on Docker 20.10.8 ...
💢 initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.22.3:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks

stderr:
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': exit status 255
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

💣 Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.22.3:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks

stderr:
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': exit status 255
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

❌ Exiting due to GUEST_START: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.22.3:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks

stderr:
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': exit status 255
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

rsong0606 · 2022-01-04T22:15:23Z

I have the same issue for M1 Max Chip. Exactly as @vimalkansal.
Here's my output.
(base) runtiansong@runtians-MBP istio-1.12.1 % minikube start
😄 minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨ Using the docker driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
🔄 Restarting existing docker container for "minikube" ...
🐳 Preparing Kubernetes v1.22.3 on Docker 20.10.8 ...
💢 initialization failed, will try again: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.22.3:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks

stderr:
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': exit status 255
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

💣 Error starting cluster: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.22.3:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks

stderr:
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': exit status 255
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

❌ Exiting due to GUEST_START: wait: /bin/bash -c "sudo env PATH="/var/lib/minikube/binaries/v1.22.3:$PATH" kubeadm init --config /var/tmp/minikube/kubeadm.yaml --ignore-preflight-errors=DirAvailable--etc-kubernetes-manifests,DirAvailable--var-lib-minikube,DirAvailable--var-lib-minikube-etcd,FileAvailable--etc-kubernetes-manifests-kube-scheduler.yaml,FileAvailable--etc-kubernetes-manifests-kube-apiserver.yaml,FileAvailable--etc-kubernetes-manifests-kube-controller-manager.yaml,FileAvailable--etc-kubernetes-manifests-etcd.yaml,Port-10250,Swap,Mem,SystemVerification,FileContent--proc-sys-net-bridge-bridge-nf-call-iptables": Process exited with status 1
stdout:
[init] Using Kubernetes version: v1.22.3
[preflight] Running pre-flight checks

stderr:
[WARNING Swap]: running with swap on is not supported. Please disable swap
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
[ERROR KubeletVersion]: couldn't get kubelet version: cannot execute 'kubelet --version': exit status 255
[preflight] If you know what you are doing, you can make a check non-fatal with --ignore-preflight-errors=...
To see the stack trace of this error execute with --v=5 or higher

spowelljr · 2022-01-04T23:03:21Z

Thanks to @rsong0606 we found that we're downloading an amd64 version of kubelet, an arm64 binary would instead be looking for /lib/ld-linux-aarch64.so.1

afbjorklund · 2022-01-05T15:19:55Z

Same as:

kubelet depending on missing /lib64/ld-linux-x86-64.so.2 on v0.0.28 m1 mac kicbase image #13190

afbjorklund · 2022-01-05T15:27:31Z

@sharifelgamal : the best fix would be to add the arch after the os in the cache.

Add architecture to the binaries and images in the cache #9593

Same way that we fixed kubectl, when it was using the Linux binary on a Mac...

addon enable fails after update: kubectl: cannot execute binary file #6559

spowelljr · 2022-01-06T01:05:14Z

To anyone coming across this problem:

Make sure you're using the arm64 binary of minikube
Run minikube delete --all --purge
Try starting minikube again

rsong0606 · 2022-01-06T01:43:56Z

It is working now. Thanks @spowelljr
(base) @runtians-MBP istio-1.12.1 % minikube delete --all --purge
🔥 Deleting "minikube" in docker ...
🔥 Removing /Users/runtiansong/.minikube/machines/minikube ...
💀 Removed all traces of the "minikube" cluster.
🔥 Successfully deleted all profiles
💀 Successfully purged minikube directory located at - [/Users/runtiansong/.minikube]
(base) runtiansong@runtians-MBP istio-1.12.1 % minikube start
😄 minikube v1.24.0 on Darwin 12.0.1 (arm64)
✨ Automatically selected the docker driver
👍 Starting control plane node minikube in cluster minikube
🚜 Pulling base image ...
> gcr.io/k8s-minikube/kicbase: 321.58 MiB / 321.58 MiB 100.00% 16.50 MiB p
🔥 Creating docker container (CPUs=2, Memory=8100MB) ...
> kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
> kubeadm.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
> kubelet.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
> kubectl: 41.44 MiB / 41.44 MiB [-------------] 100.00% 22.58 MiB p/s 2.0s
> kubelet: 107.26 MiB / 107.26 MiB [-----------] 100.00% 23.33 MiB p/s 4.8s
> kubeadm: 40.50 MiB / 40.50 MiB [--------------] 100.00% 5.91 MiB p/s 7.1s

▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...

🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
(base) @runtians-MBP istio-1.12.1 %

prashant-shahi · 2022-01-11T11:01:03Z

@spowelljr It works! Thanks 🙏

medyagh · 2022-01-31T19:18:10Z

we could adde arch to file name in cache

celialovesdata · 2023-10-22T03:22:19Z

Hi @rams3sh, thanks for reporting your issue with minikube!

First recommendation, I'd suggest updating to the latest version of minikube (v1.23.2).

Could you please try running minikube delete --all and then minikube start --driver=docker --force-systemd=true and let me know if that solves your issue, thanks!

It works. Thanks.

Edmartt · 2024-02-06T06:18:34Z

To anyone coming across this problem:

1. Make sure you're using the arm64 binary of minikube

2. Run `minikube delete --all --purge`

3. Try starting minikube again

Thank you. I had this issue on Linux too and this solved it

pavanjava · 2024-04-20T15:43:34Z

Hi @rams3sh, thanks for reporting your issue with minikube!

First recommendation, I'd suggest updating to the latest version of minikube (v1.23.2).

Could you please try running minikube delete --all and then minikube start --driver=docker --force-systemd=true and let me know if that solves your issue, thanks!

This solution worked for me.

zazencodes · 2024-05-20T13:18:14Z

I also had to delete my minikube docker container and the docker image (these were for the old amd64 version of minikube that I was using before switching to arm64)

yousefhamza · 2024-07-15T20:15:29Z

I'm on minikube v1.33.1, and still facing this error

I have tried minikube delete --all --purge, and it didn't work

and made sure that my minikube executable is arm64

Should I just give up on using minikube with docker driver?

spowelljr · 2024-07-15T20:30:36Z

Hi @yousefhamza and anyone else that runs into this issue, this is resolved for future releases by #19205.

The problem is you have an amd64 version of the kicbase in your Docker daemon. minikube uses this image to start the cluster, but downloads and uses arm64 binaries, causing an architecture mismatch, causing kubelet --version to fail. You can confirm this by running:

$ docker image inspect gcr.io/k8s-minikube/kicbase:v0.0.44 | grep Architecture

If you run minikube delete --all --purge to clear the minikube cache, and then delete any kicbase image from the Docker daemon (docker rmi gcr.io/k8s-minikube/kicbase:v0.0.44) your issue should be resolved.

yousefhamza · 2024-07-15T21:20:40Z

Thanks @spowelljr for the prompt response 🙏

I have tried this fix already, and now removed all images from my machine, and the command you provided returns

"Architecture": "arm64",

which is the correct architecture.

However minikube start still fails due to kubelet --version error.

Upon closer inspection into Docker, I found that after running minikube start, I end up with 2 docker images, there was none before running the command, and the wrong image is used. do you know why is this happening?

spowelljr · 2024-07-15T22:15:28Z

Are you using an amd64 version of the minikube binary @yousefhamza? Or is something on your machine configuring Docker to pull amd64 images? I am confused why an amd64 image is getting pulled...

Could you run minikube delete --all --purge and remove the image from Docker again. Then download: https://storage.googleapis.com/minikube-builds/19205/minikube-darwin-arm64 and use that to start minikube, that should resolve your issue, but I'm still not sure why an amd64 image is getting downloaded.

If you run the following what's the output?

$ docker pull alpine:latest
$ docker image inspect alpine:latest | grep Architecture

yousefhamza · 2024-07-16T20:58:11Z

My minikube is arm64

yosefhamza@Yousefs-Macbook-Pro-14 ~ % file /opt/homebrew/bin/minikube
/opt/homebrew/bin/minikube: Mach-O 64-bit executable arm64

$ docker pull alpine:latest
$ docker image inspect alpine:latest | grep Architecture

running this returns "Architecture": "amd64",

I delete all images in docker, confirmed that docker is arm64, re-installed minikube, and I still have the same issue

Below are some of the logs when I run minikube start

logs

``` I0716 23:50:15.268718 67496 out.go:177] 👍 Starting "minikube" primary control-plane node in "minikube" cluster 👍 Starting "minikube" primary control-plane node in "minikube" cluster I0716 23:50:15.268749 67496 cache.go:121] Beginning downloading kic base image for docker with docker I0716 23:50:15.268779 67496 out.go:177] 🚜 Pulling base image v0.0.44 ... 🚜 Pulling base image v0.0.44 ... I0716 23:50:15.268809 67496 preload.go:132] Checking if preload exists for k8s version v1.30.0 and runtime docker I0716 23:50:15.268846 67496 image.go:79] Checking for gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e in local docker daemon I0716 23:50:15.322665 67496 cache.go:149] Downloading gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e to local cache I0716 23:50:15.322856 67496 image.go:63] Checking for gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e in local cache directory I0716 23:50:15.322997 67496 image.go:118] Writing gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e to local cache I0716 23:50:15.465046 67496 preload.go:119] Found remote preload: https://storage.googleapis.com/minikube-preloaded-volume-tarballs/v18/v1.30.0/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-arm64.tar.lz4 I0716 23:50:15.465105 67496 cache.go:56] Caching tarball of preloaded images I0716 23:50:15.465288 67496 preload.go:132] Checking if preload exists for k8s version v1.30.0 and runtime docker I0716 23:50:15.465358 67496 out.go:177] 💾 Downloading Kubernetes v1.30.0 preload ... 💾 Downloading Kubernetes v1.30.0 preload ... I0716 23:50:15.465370 67496 preload.go:237] getting checksum for preloaded-images-k8s-v18-v1.30.0-docker-overlay2-arm64.tar.lz4 ... I0716 23:50:15.785194 67496 download.go:107] Downloading: https://storage.googleapis.com/minikube-preloaded-volume-tarballs/v18/v1.30.0/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-arm64.tar.lz4?checksum=md5:677034533668c42fec962cc52f9b3c42 -> /Users/yosefhamza/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-arm64.tar.lz4 > preloaded-images-k8s-v18-v1...: 319.81 MiB / 319.81 MiB 100.00% 4.08 Mi^[[1;2C > gcr.io/k8s-minikube/kicbase...: 262.03 MiB / 435.76 MiB 60.13% 3.54 MiBI0716 23:51:35.615149 67496 preload.go:248] saving checksum for preloaded-images-k8s-v18-v1.30.0-docker-overlay2-arm64.tar.lz4 ... I0716 23:51:35.615351 67496 preload.go:255] verifying checksum of /Users/yosefhamza/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-arm64.tar.lz4 ... > gcr.io/k8s-minikube/kicbase...: 264.63 MiB / 435.76 MiB 60.73% 3.59 MiBI0716 23:51:36.207412 67496 cache.go:59] Finished verifying existence of preloaded tar for v1.30.0 on docker I0716 23:51:36.207608 67496 profile.go:143] Saving config to /Users/yosefhamza/.minikube/profiles/minikube/config.json ... I0716 23:51:36.207629 67496 lock.go:35] WriteFile acquiring /Users/yosefhamza/.minikube/profiles/minikube/config.json: {Name:mkb86a173110334431e32d67c3a6d70e5e697929 Clock:{} Delay:500ms Timeout:1m0s Cancel:} > gcr.io/k8s-minikube/kicbase...: 435.76 MiB / 435.76 MiB 100.00% 4.18 Mi I0716 23:52:00.980190 67496 cache.go:152] successfully saved gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e as a tarball I0716 23:52:00.980212 67496 cache.go:162] Loading gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e from local cache I0716 23:53:28.235842 67496 cache.go:164] successfully loaded and using gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e from cached tarball I0716 23:53:28.235920 67496 cache.go:194] Successfully downloaded all kic artifacts I0716 23:53:28.236071 67496 start.go:360] acquireMachinesLock for minikube: {Name:mk91180e19378540b655fc9b0f5e674054c4ee32 Clock:{} Delay:500ms Timeout:10m0s Cancel:} I0716 23:53:28.236254 67496 start.go:364] duration metric: took 160.792µs to acquireMachinesLock for "minikube" I0716 23:53:28.236329 67496 start.go:93] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e Memory:7792 CPUs:2 DiskSize:20000 Driver:docker HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.30.0 ClusterName:minikube Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin:cni FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.30.0 ContainerRuntime:docker ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/Users:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath:/opt/homebrew/opt/socket_vmnet/bin/socket_vmnet_client SocketVMnetPath:/opt/homebrew/var/run/socket_vmnet StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.30.0 ContainerRuntime:docker ControlPlane:true Worker:true} I0716 23:53:28.236695 67496 start.go:125] createHost starting for "" (driver="docker") I0716 23:53:28.237014 67496 out.go:204] 🔥 Creating docker container (CPUs=2, Memory=7792MB) ... 🔥 Creating docker container (CPUs=2, Memory=7792MB) ...| I0716 23:53:28.237331 67496 start.go:159] libmachine.API.Create for "minikube" (driver="docker") I0716 23:53:28.237363 67496 client.go:168] LocalClient.Create starting I0716 23:53:28.239167 67496 main.go:141] libmachine: Creating CA: /Users/yosefhamza/.minikube/certs/ca.pem / I0716 23:53:28.358628 67496 main.go:141] libmachine: Creating client certificate: /Users/yosefhamza/.minikube/certs/cert.pem I0716 23:53:28.393747 67496 cli_runner.go:164] Run: docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" W0716 23:53:28.438071 67496 cli_runner.go:211] docker network inspect minikube --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}" returned with exit code 1 I0716 23:53:28.438214 67496 network_create.go:281] running [docker network inspect minikube] to gather additional debugging logs... I0716 23:53:28.438229 67496 cli_runner.go:164] Run: docker network inspect minikube - W0716 23:53:28.482760 67496 cli_runner.go:211] docker network inspect minikube returned with exit code 1 I0716 23:53:28.482797 67496 network_create.go:284] error running [docker network inspect minikube]: docker network inspect minikube: exit status 1 stdout: []

stderr:
Error response from daemon: network minikube not found
I0716 23:53:28.482812 67496 network_create.go:286] output of [docker network inspect minikube]: -- stdout --
[]

-- /stdout --
** stderr **
Error response from daemon: network minikube not found

** /stderr **
I0716 23:53:28.482997 67496 cli_runner.go:164] Run: docker network inspect bridge --format "{"Name": "{{.Name}}","Driver": "{{.Driver}}","Subnet": "{{range .IPAM.Config}}{{.Subnet}}{{end}}","Gateway": "{{range .IPAM.Config}}{{.Gateway}}{{end}}","MTU": {{if (index .Options "com.docker.network.driver.mtu")}}{{(index .Options "com.docker.network.driver.mtu")}}{{else}}0{{end}}, "ContainerIPs": [{{range $k,$v := .Containers }}"{{$v.IPv4Address}}",{{end}}]}"
I0716 23:53:28.529891 67496 network.go:206] using free private subnet 192.168.49.0/24: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0x14001654010}
I0716 23:53:28.529951 67496 network_create.go:124] attempt to create docker network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 65535 ...
I0716 23:53:28.530058 67496 cli_runner.go:164] Run: docker network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 -o --ip-masq -o --icc -o com.docker.network.driver.mtu=65535 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=minikube minikube

I0716 23:53:28.849663 67496 network_create.go:108] docker network minikube 192.168.49.0/24 created
I0716 23:53:28.849721 67496 kic.go:121] calculated static IP "192.168.49.2" for the "minikube" container
I0716 23:53:28.850186 67496 cli_runner.go:164] Run: docker ps -a --format {{.Names}}
I0716 23:53:28.911484 67496 cli_runner.go:164] Run: docker volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
\ I0716 23:53:28.957381 67496 oci.go:103] Successfully created a docker volume minikube
I0716 23:53:28.957633 67496 cli_runner.go:164] Run: docker run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -d /var/lib
| I0716 23:53:29.464127 67496 oci.go:107] Successfully prepared a docker volume minikube
I0716 23:53:29.464203 67496 preload.go:132] Checking if preload exists for k8s version v1.30.0 and runtime docker
I0716 23:53:29.464231 67496 kic.go:194] Starting extracting preloaded images to volume ...
I0716 23:53:29.464658 67496 cli_runner.go:164] Run: docker run --rm --entrypoint /usr/bin/tar -v /Users/yosefhamza/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-arm64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -I lz4 -xf /preloaded.tar -C /extractDir
/ I0716 23:53:32.002298 67496 cli_runner.go:217] Completed: docker run --rm --entrypoint /usr/bin/tar -v /Users/yosefhamza/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-docker-overlay2-arm64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e -I lz4 -xf /preloaded.tar -C /extractDir: (2.

</details>

spowelljr · 2024-07-16T21:30:15Z

Thanks for the update and the information!

running this returns "Architecture": "amd64",

There's something wrong here, this is indicating that your Docker is pulling amd64 images by default and is essentially the root of the problem. If I run it on my M1 Mac I get:

"Architecture": "arm64",

I'm not sure if there's an option or config value you modified in Docker somewhere to cause that, but for other use cases outside of minikube you'd likely want to find the cause and resolve it.

But back to the minikube front, this makes sense why it's happening given the above information. When you start minikube for the first time or after minikube delete --all --purge it downloads the kicbase image to the minikube home folder cache (~/.minikube/cache), it them loads the image from the cache into the Docker daemon. This operation is correctly downloading the arm64 version of the kicbase. The problem is that images loaded from a local file lose their sha, so to ensure that the image was unmodified we run docker pull <kicbase_image> with the point to just get the sha attached to the image to made sure unmodified. This docker pull command should have had a --plaform=linux/<arch> flag included in it (resolved in #19205), but since it's missing that, it leaves it up to your Docker to pull the arch it wants, which in your case is defaulted to amd64, it basically looks at the arm64 kicbase image and goes "this isn't the image that I want" and then proceeds to downloads the amd64 version of the kicbase, which is causing your issue. Since sha has priority over tags, when minikube starts it uses the amd64 kicbase and you get the issues you see.

So like I mentioned above, if you're able to figure out why your Docker is pulling amd64 images by default and resolve that, it should also resolve this issue you're experiencing in minikube. Otherwise, you can download and use the binary I'll link below that has the --platform fix implemeneted until the next minikube release that will have the fix included.

https://storage.googleapis.com/minikube-builds/19205/minikube-darwin-arm64

revathyrams · 2024-07-16T23:28:55Z

@yousefhamza, I faced the same issue as well where the kicbase image was arm64, but docker pull alpine would always download amd64 by default. Check your
echo DOCKER_DEFAULT_PLATFORM, for me I had set it to linux/amd64 sometime back. Changing it back to linux/arm64 fixed the issue. Now docker pull pulls arm64 images. I'd highly recommend checking your docker default platform

echo $DOCKER_DEFAULT_PLATFORM

linux/arm64

yousefhamza · 2024-07-17T06:33:08Z

Thanks @spowelljr for explaining the issue 🙏

Thanks @revathyrams for the docker reference 🙏 , I found that it was linux/amd64 and when I fixed it, minikube finished successfully

spowelljr added the kind/support Categorizes issue or PR as a support question. label Aug 24, 2021

spowelljr added triage/needs-information Indicates an issue needs more information in order to work on it. long-term-support Long-term support issues that can't be fixed in code labels Oct 6, 2021

spowelljr closed this as completed Oct 20, 2021

spowelljr reopened this Jan 4, 2022

spowelljr changed the title ~~Getting Error while starting minikube~~ minikube start on M1 Mac failing with kubelet --version Jan 4, 2022

spowelljr changed the title ~~minikube start on M1 Mac failing with kubelet --version~~ minikube start on M1 Mac failing with 'kubelet --version': exit status 255 Jan 4, 2022

spowelljr added arch/arm64 and removed triage/needs-information Indicates an issue needs more information in order to work on it. labels Jan 4, 2022

spowelljr added this to the 1.26.0-candidate milestone Jan 4, 2022

spowelljr self-assigned this Jan 4, 2022

spowelljr modified the milestones: 1.26.0-candidate, 1.25.0 Jan 4, 2022

spowelljr assigned sharifelgamal and unassigned spowelljr Jan 4, 2022

afbjorklund added co/docker-driver Issues related to kubernetes in container kind/bug Categorizes issue or PR as related to a bug. and removed kind/support Categorizes issue or PR as a support question. labels Jan 5, 2022

spowelljr removed the long-term-support Long-term support issues that can't be fixed in code label Jan 5, 2022

spowelljr mentioned this issue Jan 5, 2022

Unable to start minikube on Apple M1 #13288

Closed

leogovan mentioned this issue Jan 12, 2022

Unable to run minikube start #13334

Closed

spowelljr mentioned this issue Jan 19, 2022

Unable to initialise minikube #13203

Closed

spowelljr added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jan 25, 2022

spowelljr mentioned this issue Jan 25, 2022

The connection to the server localhost:8443 was refused - did you specify the right host or port? #13317

Closed

sharifelgamal modified the milestones: 1.25.0, 1.26.0-candidate Jan 26, 2022

spowelljr mentioned this issue Jan 27, 2022

Minikube not working #13484

Closed

sharifelgamal mentioned this issue Feb 1, 2022

add arch to binary and image cache paths #13539

Merged

medyagh closed this as completed in #13539 Feb 10, 2022

speedfinger mentioned this issue Jul 23, 2022

Failed to run Minikube #14623

Closed

red8888 mentioned this issue Mar 17, 2023

Getting 'kubelet --version': exit status 255 even after following all the fixes #16076

Closed

spowelljr mentioned this issue Jul 16, 2024

Improvements for using x86 binary on arm64 machine #19205

Merged

minikube start on M1 Mac failing with 'kubelet --version': exit status 255 #12274

minikube start on M1 Mac failing with 'kubelet --version': exit status 255 #12274

Comments

rams3sh commented Aug 16, 2021

This file is a systemd drop-in unit that inherits from the base dockerd configuration.

The base configuration already specifies an 'ExecStart=...' command. The first directive

here is to clear out that command inherited from the base configuration. Without this,

the command from the base configuration and the command specified here are treated as

a sequence of commands, which is not the desired behavior, nor is it valid -- systemd

will catch this invalid input and refuse to start the service with an error like:

Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.

NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other

container runtimes. If left unlimited, it may result in OOM issues with MySQL.

Having non-zero Limit*s causes performance problems due to accounting overhead

in the kernel. We recommend using cgroups to do container-local accounting.

Uncomment TasksMax if your systemd version supports it.

Only systemd 226 and above support this version.

set delegate yes so that systemd does not reset the cgroups of docker containers

kill only the docker process, not all processes in the cgroup

This file is a systemd drop-in unit that inherits from the base dockerd configuration.

The base configuration already specifies an 'ExecStart=...' command. The first directive

here is to clear out that command inherited from the base configuration. Without this,

the command from the base configuration and the command specified here are treated as

a sequence of commands, which is not the desired behavior, nor is it valid -- systemd

will catch this invalid input and refuse to start the service with an error like:

Service has more than one ExecStart= setting, which is only allowed for Type=oneshot services.

NOTE: default-ulimit=nofile is set to an arbitrary number for consistency with other

container runtimes. If left unlimited, it may result in OOM issues with MySQL.

Having non-zero Limit*s causes performance problems due to accounting overhead

in the kernel. We recommend using cgroups to do container-local accounting.

Uncomment TasksMax if your systemd version supports it.

Only systemd 226 and above support this version.

set delegate yes so that systemd does not reset the cgroups of docker containers

kill only the docker process, not all processes in the cgroup

Having non-zero Limit*s causes performance problems due to accounting overhead

in the kernel. We recommend using cgroups to do container-local accounting.

set delegate yes so that systemd does not reset the cgroups of docker containers

kill only the docker process, not all processes in the cgroup

disable disk resource management by default

imageGCHighThresholdPercent: 100 evictionHard: nodefs.available: "0%" nodefs.inodesFree: "0%" imagefs.available: "0%" failSwapOn: false staticPodPath: /etc/kubernetes/manifests

Rits333 commented Aug 20, 2021 • edited Loading

spowelljr commented Oct 6, 2021

spowelljr commented Oct 20, 2021

DPS0340 commented Nov 13, 2021

DPS0340 commented Nov 13, 2021

chris-lsn commented Nov 28, 2021

vimalkansal commented Jan 3, 2022 • edited Loading

rsong0606 commented Jan 4, 2022

spowelljr commented Jan 4, 2022 • edited Loading

afbjorklund commented Jan 5, 2022 • edited Loading

afbjorklund commented Jan 5, 2022

spowelljr commented Jan 6, 2022

rsong0606 commented Jan 6, 2022 • edited Loading

prashant-shahi commented Jan 11, 2022

medyagh commented Jan 31, 2022

celialovesdata commented Oct 22, 2023

Edmartt commented Feb 6, 2024

pavanjava commented Apr 20, 2024

zazencodes commented May 20, 2024

yousefhamza commented Jul 15, 2024

spowelljr commented Jul 15, 2024

yousefhamza commented Jul 15, 2024

spowelljr commented Jul 15, 2024

yousefhamza commented Jul 16, 2024

spowelljr commented Jul 16, 2024 • edited Loading

revathyrams commented Jul 16, 2024 • edited Loading

yousefhamza commented Jul 17, 2024

`minikube start` on M1 Mac failing with `'kubelet --version': exit status 255` #12274

`minikube start` on M1 Mac failing with `'kubelet --version': exit status 255` #12274

imageGCHighThresholdPercent: 100
evictionHard:
nodefs.available: "0%"
nodefs.inodesFree: "0%"
imagefs.available: "0%"
failSwapOn: false
staticPodPath: /etc/kubernetes/manifests

Rits333 commented Aug 20, 2021 •

edited

Loading

vimalkansal commented Jan 3, 2022 •

edited

Loading

spowelljr commented Jan 4, 2022 •

edited

Loading

afbjorklund commented Jan 5, 2022 •

edited

Loading

rsong0606 commented Jan 6, 2022 •

edited

Loading

spowelljr commented Jul 16, 2024 •

edited

Loading

revathyrams commented Jul 16, 2024 •

edited

Loading