Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weโ€™ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail to start minikube with root-less podman #16919

Open
hugo-syn opened this issue Jul 20, 2023 · 11 comments
Open

Fail to start minikube with root-less podman #16919

hugo-syn opened this issue Jul 20, 2023 · 11 comments
Labels
co/podman-driver podman driver issues kind/support Categorizes issue or PR as a support question. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.

Comments

@hugo-syn
Copy link

hugo-syn commented Jul 20, 2023

What Happened?

I can't start minikube with the podman driver in rootless mode.

minikube start --driver=podman --container-runtime=containerd           
๐Ÿ˜„  minikube v1.31.0 on Debian 12.0
    โ–ช MINIKUBE_ROOTLESS=true
โœจ  Using the podman driver based on user configuration
๐Ÿ“Œ  Using rootless Podman driver
๐Ÿ‘  Starting control plane node minikube in cluster minikube
๐Ÿšœ  Pulling base image ...
E0720 17:41:05.157264  150173 cache.go:190] Error downloading kic artifacts:  not yet implemented, see issue #8426
๐Ÿ”ฅ  Creating podman container (CPUs=2, Memory=7900MB) ...
๐Ÿคฆ  StartHost failed, but will try again: creating host: create: creating: create kic node: create container: podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=7900mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.40: exit status 126
stdout:

stderr:
Error: crun: error stat'ing file `/dev/vboxusb/001/003`: Permission denied: OCI permission denied

๐Ÿ”„  Restarting existing podman container for "minikube" ...
๐Ÿ˜ฟ  Failed to start podman container. Running "minikube delete" may fix it: driver start: start: podman start minikube: exit status 125
stdout:

stderr:
Error: unable to start container "e2fa0aaa486b87b5acedaf360303a2afe419d2386fe4039d3c51c9368dd73537": crun: error stat'ing file `/dev/vboxusb/001/003`: Permission denied: OCI permission denied


โŒ  Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: driver start: start: podman start minikube: exit status 125
stdout:

stderr:
Error: unable to start container "e2fa0aaa486b87b5acedaf360303a2afe419d2386fe4039d3c51c9368dd73537": crun: error stat'ing file `/dev/vboxusb/001/003`: Permission denied: OCI permission denied

Attach the log file

`** /stderr **
I0720 17:52:24.654165 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:52:24.689875 156176 cli_runner.go:164] Run: podman network inspect podman --format "{{range .}}{{if eq .Driver "bridge"}}{{(index .Subnets 0).Subnet}},{{(index .Subnets 0).Gateway}}{{end}}{{end}}"
I0720 17:52:24.725931 156176 network.go:209] using free private subnet 192.168.49.0/24: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc0005f7960}
I0720 17:52:24.725967 156176 network_create.go:123] attempt to create podman network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 0 ...
I0720 17:52:24.726003 156176 cli_runner.go:164] Run: podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=minikube minikube
I0720 17:52:24.764246 156176 network_create.go:107] podman network minikube 192.168.49.0/24 created
I0720 17:52:24.764262 156176 kic.go:117] calculated static IP "192.168.49.2" for the "minikube" container
I0720 17:52:24.764316 156176 cli_runner.go:164] Run: podman ps -a --format {{.Names}}
I0720 17:52:24.798459 156176 cli_runner.go:164] Run: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
I0720 17:52:24.844224 156176 oci.go:103] Successfully created a podman volume minikube
I0720 17:52:24.844264 156176 cli_runner.go:164] Run: podman run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.40 -d /var/lib
I0720 17:52:28.170992 156176 cli_runner.go:217] Completed: podman run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.40 -d /var/lib: (3.326690571s)
I0720 17:52:28.171027 156176 oci.go:107] Successfully prepared a podman volume minikube
I0720 17:52:28.171054 156176 preload.go:132] Checking if preload exists for k8s version v1.27.3 and runtime containerd
I0720 17:52:28.171071 156176 kic.go:190] Starting extracting preloaded images to volume ...
I0720 17:52:28.171126 156176 cli_runner.go:164] Run: podman run --rm --entrypoint /usr/bin/tar --security-opt label=disable -v /home/user/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.27.3-containerd-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.40 -I lz4 -xf /preloaded.tar -C /extractDir
I0720 17:52:39.362900 156176 cli_runner.go:217] Completed: podman run --rm --entrypoint /usr/bin/tar --security-opt label=disable -v /home/user/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.27.3-containerd-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.40 -I lz4 -xf /preloaded.tar -C /extractDir: (11.191727796s)
I0720 17:52:39.362923 156176 kic.go:199] duration metric: took 11.191845 seconds to extract preloaded images to volume
W0720 17:52:39.363205 156176 cgroups_linux.go:77] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
W0720 17:52:39.363268 156176 oci.go:240] Your kernel does not support CPU cfs period/quota or the cgroup is not mounted.
I0720 17:52:39.363308 156176 cli_runner.go:164] Run: podman info --format "'{{json .SecurityOptions}}'"
W0720 17:52:42.924179 156176 cli_runner.go:211] podman info --format "'{{json .SecurityOptions}}'" returned with exit code 125
I0720 17:52:42.924200 156176 cli_runner.go:217] Completed: podman info --format "'{{json .SecurityOptions}}'": (3.560838029s)
I0720 17:52:42.924300 156176 cli_runner.go:164] Run: podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=7900mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.40
W0720 17:52:49.242697 156176 cli_runner.go:211] podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=7900mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.40 returned with exit code 126
I0720 17:52:49.242721 156176 cli_runner.go:217] Completed: podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=7900mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.40: (6.318363385s)
I0720 17:52:49.242772 156176 client.go:171] LocalClient.Create took 24.745587099s
I0720 17:52:51.243476 156176 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0720 17:52:51.243597 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:52:51.381367 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:52:51.490301 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:52:51.490533 156176 retry.go:31] will retry after 331.520036ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:52:51.823338 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:52:51.933714 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:52:52.052250 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:52:52.052448 156176 retry.go:31] will retry after 236.672894ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:52:52.290302 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:52:52.396626 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:52:52.448771 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:52:52.448864 156176 retry.go:31] will retry after 686.150186ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:52:53.135305 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:52:53.253209 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:52:53.371803 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
W0720 17:52:53.372134 156176 start.go:275] error running df -h /var: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
W0720 17:52:53.372198 156176 start.go:242] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:52:53.372312 156176 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0720 17:52:53.372400 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:52:53.489449 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:52:53.599127 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:52:53.599215 156176 retry.go:31] will retry after 368.372712ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:52:53.968426 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:52:54.055536 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:52:54.096379 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:52:54.096463 156176 retry.go:31] will retry after 351.643475ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:52:54.448719 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:52:54.554354 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:52:54.655829 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:52:54.655894 156176 retry.go:31] will retry after 621.330498ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:52:55.278000 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:52:55.413040 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:52:55.538557 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
W0720 17:52:55.538931 156176 start.go:290] error running df -BG /var: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
W0720 17:52:55.538991 156176 start.go:247] error getting GiB of /var that is available: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:52:55.539062 156176 start.go:128] duration metric: createHost completed in 31.046274902s
I0720 17:52:55.539082 156176 start.go:83] releasing machines lock for "minikube", held for 31.046412855s
W0720 17:52:55.539124 156176 start.go:672] error starting host: creating host: create: creating: create kic node: create container: podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=7900mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.40: exit status 126
stdout:

stderr:
Error: crun: error stat'ing file /dev/vboxusb/001/003: Permission denied: OCI permission denied
I0720 17:52:55.541767 156176 cli_runner.go:164] Run: podman container inspect minikube --format={{.State.Status}}
W0720 17:52:55.648448 156176 start.go:677] delete host: Docker machine "minikube" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
W0720 17:52:55.648518 156176 out.go:239] ๐Ÿคฆ StartHost failed, but will try again: creating host: create: creating: create kic node: create container: podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=7900mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.40: exit status 126
stdout:

stderr:
Error: crun: error stat'ing file /dev/vboxusb/001/003: Permission denied: OCI permission denied

I0720 17:52:55.648588 156176 start.go:687] Will try again in 5 seconds ...
I0720 17:53:00.651194 156176 start.go:365] acquiring machines lock for minikube: {Name:mk014e19cf8b44f5e25987f528a80e8a38617d4e Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I0720 17:53:00.651342 156176 start.go:369] acquired machines lock for "minikube" in 109.031ยตs
I0720 17:53:00.651378 156176 start.go:96] Skipping create...Using existing machine configuration
I0720 17:53:00.651387 156176 fix.go:54] fixHost starting:
I0720 17:53:00.651961 156176 cli_runner.go:164] Run: podman container inspect minikube --format={{.State.Status}}
I0720 17:53:00.750839 156176 fix.go:102] recreateIfNeeded on minikube: state= err=
W0720 17:53:00.750854 156176 fix.go:128] unexpected machine state, will restart:
I0720 17:53:00.762916 156176 out.go:177] ๐Ÿ”„ Restarting existing podman container for "minikube" ...
I0720 17:53:00.766778 156176 cli_runner.go:164] Run: podman start minikube
W0720 17:53:01.501354 156176 cli_runner.go:211] podman start minikube returned with exit code 125
I0720 17:53:01.501397 156176 cli_runner.go:164] Run: podman inspect minikube
I0720 17:53:01.574370 156176 errors.go:84] Postmortem inspect ("podman inspect minikube"): -- stdout --
[
{
"Id": "606e6a197505aceb29e4ececd6ebf73942619ce9559ebc04a507a00e3db8b4a2",
"Created": "2023-07-20T17:52:43.060528479+02:00",
"Path": "/usr/local/bin/entrypoint",
"Args": [
"/sbin/init"
],
"State": {
"OciVersion": "1.0.2-dev",
"Status": "created",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 0,
"Error": "",
"StartedAt": "0001-01-01T00:00:00Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "",
"FailingStreak": 0,
"Log": null
},
"CheckpointedAt": "0001-01-01T00:00:00Z",
"RestoredAt": "0001-01-01T00:00:00Z"
},
"Image": "c6cc01e6091959400f260dc442708e7c71630b58dab1f7c344cb00926bd84950",
"ImageDigest": "sha256:8cadf23777709e43eca447c47a45f5a4635615129267ce025193040ec92a1631",
"ImageName": "gcr.io/k8s-minikube/kicbase:v0.0.40",
"Rootfs": "",
"Pod": "",
"ResolvConfPath": "/run/user/1000/containers/vfs-containers/606e6a197505aceb29e4ececd6ebf73942619ce9559ebc04a507a00e3db8b4a2/userdata/resolv.conf",
"HostnamePath": "/run/user/1000/containers/vfs-containers/606e6a197505aceb29e4ececd6ebf73942619ce9559ebc04a507a00e3db8b4a2/userdata/hostname",
"HostsPath": "/run/user/1000/containers/vfs-containers/606e6a197505aceb29e4ececd6ebf73942619ce9559ebc04a507a00e3db8b4a2/userdata/hosts",
"StaticDir": "/home/user/.local/share/containers/storage/vfs-containers/606e6a197505aceb29e4ececd6ebf73942619ce9559ebc04a507a00e3db8b4a2/userdata",
"OCIConfigPath": "/home/user/.local/share/containers/storage/vfs-containers/606e6a197505aceb29e4ececd6ebf73942619ce9559ebc04a507a00e3db8b4a2/userdata/config.json",
"OCIRuntime": "crun",
"ConmonPidFile": "/run/user/1000/containers/vfs-containers/606e6a197505aceb29e4ececd6ebf73942619ce9559ebc04a507a00e3db8b4a2/userdata/conmon.pid",
"PidFile": "/run/user/1000/containers/vfs-containers/606e6a197505aceb29e4ececd6ebf73942619ce9559ebc04a507a00e3db8b4a2/userdata/pidfile",
"Name": "minikube",
"RestartCount": 0,
"Driver": "vfs",
"MountLabel": "",
"ProcessLabel": "",
"AppArmorProfile": "",
"EffectiveCaps": [
"CAP_AUDIT_CONTROL",
"CAP_AUDIT_READ",
"CAP_AUDIT_WRITE",
"CAP_BLOCK_SUSPEND",
"CAP_BPF",
"CAP_CHECKPOINT_RESTORE",
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_KILL",
"CAP_LEASE",
"CAP_LINUX_IMMUTABLE",
"CAP_MAC_ADMIN",
"CAP_MAC_OVERRIDE",
"CAP_MKNOD",
"CAP_NET_ADMIN",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_RAW",
"CAP_PERFMON",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYSLOG",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_CHROOT",
"CAP_SYS_MODULE",
"CAP_SYS_NICE",
"CAP_SYS_PACCT",
"CAP_SYS_PTRACE",
"CAP_SYS_RAWIO",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_WAKE_ALARM"
],
"BoundingCaps": [
"CAP_AUDIT_CONTROL",
"CAP_AUDIT_READ",
"CAP_AUDIT_WRITE",
"CAP_BLOCK_SUSPEND",
"CAP_BPF",
"CAP_CHECKPOINT_RESTORE",
"CAP_CHOWN",
"CAP_DAC_OVERRIDE",
"CAP_DAC_READ_SEARCH",
"CAP_FOWNER",
"CAP_FSETID",
"CAP_IPC_LOCK",
"CAP_IPC_OWNER",
"CAP_KILL",
"CAP_LEASE",
"CAP_LINUX_IMMUTABLE",
"CAP_MAC_ADMIN",
"CAP_MAC_OVERRIDE",
"CAP_MKNOD",
"CAP_NET_ADMIN",
"CAP_NET_BIND_SERVICE",
"CAP_NET_BROADCAST",
"CAP_NET_RAW",
"CAP_PERFMON",
"CAP_SETFCAP",
"CAP_SETGID",
"CAP_SETPCAP",
"CAP_SETUID",
"CAP_SYSLOG",
"CAP_SYS_ADMIN",
"CAP_SYS_BOOT",
"CAP_SYS_CHROOT",
"CAP_SYS_MODULE",
"CAP_SYS_NICE",
"CAP_SYS_PACCT",
"CAP_SYS_PTRACE",
"CAP_SYS_RAWIO",
"CAP_SYS_RESOURCE",
"CAP_SYS_TIME",
"CAP_SYS_TTY_CONFIG",
"CAP_WAKE_ALARM"
],
"ExecIDs": [],
"GraphDriver": {
"Name": "vfs",
"Data": null
},
"Mounts": [
{
"Type": "volume",
"Name": "minikube",
"Source": "/home/user/.local/share/containers/storage/volumes/minikube/_data",
"Destination": "/var",
"Driver": "local",
"Mode": "",
"Options": [
"exec",
"nosuid",
"nodev",
"rbind"
],
"RW": true,
"Propagation": "rprivate"
},
{
"Type": "bind",
"Source": "/lib/modules",
"Destination": "/lib/modules",
"Driver": "",
"Mode": "",
"Options": [
"rbind"
],
"RW": false,
"Propagation": "rprivate"
}
],
"Dependencies": [],
"NetworkSettings": {
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"Bridge": "",
"SandboxID": "",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "38079"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "41757"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "37569"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "41015"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "36917"
}
]
},
"SandboxKey": "",
"Networks": {
"minikube": {
"EndpointID": "",
"Gateway": "",
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "",
"NetworkID": "minikube",
"DriverOpts": null,
"IPAMConfig": null,
"Links": null,
"Aliases": [
"606e6a197505"
]
}
}
},
"Namespace": "",
"IsInfra": false,
"IsService": false,
"Config": {
"Hostname": "minikube",
"Domainname": "",
"User": "",
"AttachStdin": false,
"AttachStdout": false,
"AttachStderr": false,
"Tty": true,
"OpenStdin": false,
"StdinOnce": false,
"Env": [
"container=podman",
"HTTP_PROXY=http://internal.local:3130/",
"http_proxy=http://internal.local:3130/",
"HTTPS_PROXY=http://internal.local:3130/",
"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"ftp_proxy=http://internal.local:3130/",
"FTP_PROXY=http://internal.local:3130/",
"https_proxy=http://internal.local:3130/",
"TERM=xterm",
"HOSTNAME=minikube",
"HOME=/root"
],
"Cmd": null,
"Image": "gcr.io/k8s-minikube/kicbase:v0.0.40",
"Volumes": null,
"WorkingDir": "/",
"Entrypoint": "/usr/local/bin/entrypoint /sbin/init",
"OnBuild": null,
"Labels": {
"created_by.minikube.sigs.k8s.io": "true",
"mode.minikube.sigs.k8s.io": "minikube",
"name.minikube.sigs.k8s.io": "minikube",
"role.minikube.sigs.k8s.io": ""
},
"Annotations": {
"io.container.manager": "libpod",
"io.kubernetes.cri-o.Created": "2023-07-20T17:52:43.060528479+02:00",
"io.kubernetes.cri-o.TTY": "true",
"io.podman.annotations.autoremove": "FALSE",
"io.podman.annotations.init": "FALSE",
"io.podman.annotations.privileged": "TRUE",
"io.podman.annotations.publish-all": "FALSE",
"io.podman.annotations.seccomp": "unconfined",
"org.opencontainers.image.stopSignal": "37"
},
"StopSignal": 37,
"HealthcheckOnFailureAction": "none",
"CreateCommand": [
"podman",
"run",
"-d",
"-t",
"--privileged",
"--security-opt",
"seccomp=unconfined",
"--tmpfs",
"/tmp",
"--tmpfs",
"/run",
"-v",
"/lib/modules:/lib/modules:ro",
"--hostname",
"minikube",
"--name",
"minikube",
"--label",
"created_by.minikube.sigs.k8s.io=true",
"--label",
"name.minikube.sigs.k8s.io=minikube",
"--label",
"role.minikube.sigs.k8s.io=",
"--label",
"mode.minikube.sigs.k8s.io=minikube",
"--network",
"minikube",
"--ip",
"192.168.49.2",
"--volume",
"minikube:/var:exec",
"--memory=7900mb",
"-e",
"container=podman",
"--expose",
"8443",
"--publish=127.0.0.1::8443",
"--publish=127.0.0.1::22",
"--publish=127.0.0.1::2376",
"--publish=127.0.0.1::5000",
"--publish=127.0.0.1::32443",
"gcr.io/k8s-minikube/kicbase:v0.0.40"
],
"Umask": "0022",
"Timeout": 0,
"StopTimeout": 10,
"Passwd": true,
"sdNotifyMode": "container"
},
"HostConfig": {
"Binds": [
"minikube:/var:exec,rw,rprivate,nosuid,nodev,rbind",
"/lib/modules:/lib/modules:ro,rprivate,rbind"
],
"CgroupManager": "systemd",
"CgroupMode": "private",
"ContainerIDFile": "",
"LogConfig": {
"Type": "journald",
"Config": null,
"Path": "",
"Tag": "",
"Size": "0B"
},
"NetworkMode": "bridge",
"PortBindings": {
"22/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "38079"
}
],
"2376/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "41757"
}
],
"32443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "37569"
}
],
"5000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "41015"
}
],
"8443/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "36917"
}
]
},
"RestartPolicy": {
"Name": "",
"MaximumRetryCount": 0
},
"AutoRemove": false,
"VolumeDriver": "",
"VolumesFrom": null,
"CapAdd": [],
"CapDrop": [],
"Dns": [],
"DnsOptions": [],
"DnsSearch": [],
"ExtraHosts": [],
"GroupAdd": [],
"IpcMode": "shareable",
"Cgroup": "",
"Cgroups": "default",
"Links": null,
"OomScoreAdj": 0,
"PidMode": "private",
"Privileged": true,
"PublishAllPorts": false,
"ReadonlyRootfs": false,
"SecurityOpt": [
"seccomp=unconfined"
],
"Tmpfs": {
"/run": "rw,rprivate,nosuid,nodev,tmpcopyup",
"/tmp": "rw,rprivate,nosuid,nodev,tmpcopyup"
},
"UTSMode": "private",
"UsernsMode": "",
"ShmSize": 65536000,
"Runtime": "oci",
"ConsoleSize": [
0,
0
],
"Isolation": "",
"CpuShares": 0,
"Memory": 8283750400,
"NanoCpus": 0,
"CgroupParent": "user.slice",
"BlkioWeight": 0,
"BlkioWeightDevice": null,
"BlkioDeviceReadBps": null,
"BlkioDeviceWriteBps": null,
"BlkioDeviceReadIOps": null,
"BlkioDeviceWriteIOps": null,
"CpuPeriod": 0,
"CpuQuota": 0,
"CpuRealtimePeriod": 0,
"CpuRealtimeRuntime": 0,
"CpusetCpus": "",
"CpusetMems": "",
"Devices": [],
"DiskQuota": 0,
"KernelMemory": 0,
"MemoryReservation": 0,
"MemorySwap": 16567500800,
"MemorySwappiness": -1,
"OomKillDisable": false,
"PidsLimit": 2048,
"Ulimits": [],
"CpuCount": 0,
"CpuPercent": 0,
"IOMaximumIOps": 0,
"IOMaximumBandwidth": 0,
"CgroupConf": null
}
}
]

-- /stdout --
I0720 17:53:01.574423 156176 cli_runner.go:164] Run: podman logs --timestamps minikube
I0720 17:53:01.725014 156176 errors.go:91] Postmortem logs ("podman logs --timestamps minikube"):
I0720 17:53:01.725089 156176 cli_runner.go:164] Run: podman system info --format json
I0720 17:53:04.953456 156176 cli_runner.go:217] Completed: podman system info --format json: (3.22834532s)
I0720 17:53:04.953567 156176 info.go:288] podman info: {Host:{BuildahVersion:1.28.2 CgroupVersion:v2 Conmon:{Package:conmon_2.1.6+ds1-1_amd64 Path:/usr/bin/conmon Version:conmon version 2.1.6, commit: unknown} Distribution:{Distribution:debian Version:12} MemFree:699437056 MemTotal:33273450496 OCIRuntime:{Name:crun Package:crun_1.8.1-1+b1_amd64 Path:/usr/bin/crun Version:crun version 1.8.1
commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL} SwapFree:16575623168 SwapTotal:16651382784 Arch:amd64 Cpus:8 Eventlogger:journald Hostname:FRSIEL11142 Kernel:6.1.0-9-amd64 Os:linux Security:{Rootless:true} Uptime:7h 59m 49.00s (Approximately 0.29 days)} Registries:{Search:[]} Store:{ConfigFile:/home/user/.config/containers/storage.conf ContainerStore:{Number:1} GraphDriverName:vfs GraphOptions:{} GraphRoot:/home/user/.local/share/containers/storage GraphStatus:{BackingFilesystem: NativeOverlayDiff: SupportsDType: UsingMetacopy:} ImageStore:{Number:1} RunRoot:/run/user/1000/containers VolumePath:/home/user/.local/share/containers/storage/volumes}}
I0720 17:53:04.953591 156176 errors.go:106] postmortem podman info: {Host:{BuildahVersion:1.28.2 CgroupVersion:v2 Conmon:{Package:conmon_2.1.6+ds1-1_amd64 Path:/usr/bin/conmon Version:conmon version 2.1.6, commit: unknown} Distribution:{Distribution:debian Version:12} MemFree:699437056 MemTotal:33273450496 OCIRuntime:{Name:crun Package:crun_1.8.1-1+b1_amd64 Path:/usr/bin/crun Version:crun version 1.8.1
commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL} SwapFree:16575623168 SwapTotal:16651382784 Arch:amd64 Cpus:8 Eventlogger:journald Hostname:FRSIEL11142 Kernel:6.1.0-9-amd64 Os:linux Security:{Rootless:true} Uptime:7h 59m 49.00s (Approximately 0.29 days)} Registries:{Search:[]} Store:{ConfigFile:/home/user/.config/containers/storage.conf ContainerStore:{Number:1} GraphDriverName:vfs GraphOptions:{} GraphRoot:/home/user/.local/share/containers/storage GraphStatus:{BackingFilesystem: NativeOverlayDiff: SupportsDType: UsingMetacopy:} ImageStore:{Number:1} RunRoot:/run/user/1000/containers VolumePath:/home/user/.local/share/containers/storage/volumes}}
I0720 17:53:04.953626 156176 network_create.go:281] running [podman network inspect minikube] to gather additional debugging logs...
I0720 17:53:04.953640 156176 cli_runner.go:164] Run: podman network inspect minikube
I0720 17:53:05.015723 156176 network_create.go:286] output of [podman network inspect minikube]: -- stdout --
[
{
"name": "minikube",
"id": "7a38d9133e293225d3e21fa37439d5062a3399221571bcb7ddf30f0a80cc5d75",
"driver": "bridge",
"network_interface": "podman1",
"created": "2023-07-20T17:52:24.761261983+02:00",
"subnets": [
{
"subnet": "192.168.49.0/24",
"gateway": "192.168.49.1"
}
],
"ipv6_enabled": false,
"internal": false,
"dns_enabled": true,
"labels": {
"created_by.minikube.sigs.k8s.io": "true",
"name.minikube.sigs.k8s.io": "minikube"
},
"ipam_options": {
"driver": "host-local"
}
}
]

-- /stdout --
I0720 17:53:05.015778 156176 cli_runner.go:164] Run: podman system info --format json
I0720 17:53:08.215273 156176 cli_runner.go:217] Completed: podman system info --format json: (3.199474444s)
I0720 17:53:08.215377 156176 info.go:288] podman info: {Host:{BuildahVersion:1.28.2 CgroupVersion:v2 Conmon:{Package:conmon_2.1.6+ds1-1_amd64 Path:/usr/bin/conmon Version:conmon version 2.1.6, commit: unknown} Distribution:{Distribution:debian Version:12} MemFree:865320960 MemTotal:33273450496 OCIRuntime:{Name:crun Package:crun_1.8.1-1+b1_amd64 Path:/usr/bin/crun Version:crun version 1.8.1
commit: f8a096be060b22ccd3d5f3ebe44108517fbf6c30
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL} SwapFree:16575623168 SwapTotal:16651382784 Arch:amd64 Cpus:8 Eventlogger:journald Hostname:FRSIEL11142 Kernel:6.1.0-9-amd64 Os:linux Security:{Rootless:true} Uptime:7h 59m 52.00s (Approximately 0.29 days)} Registries:{Search:[]} Store:{ConfigFile:/home/user/.config/containers/storage.conf ContainerStore:{Number:1} GraphDriverName:vfs GraphOptions:{} GraphRoot:/home/user/.local/share/containers/storage GraphStatus:{BackingFilesystem: NativeOverlayDiff: SupportsDType: UsingMetacopy:} ImageStore:{Number:1} RunRoot:/run/user/1000/containers VolumePath:/home/user/.local/share/containers/storage/volumes}}
I0720 17:53:08.215457 156176 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0720 17:53:08.215482 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:53:08.298213 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:53:08.420061 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:53:08.420276 156176 retry.go:31] will retry after 225.530645ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:53:08.646827 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:53:08.724052 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:53:08.818672 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:53:08.818775 156176 retry.go:31] will retry after 376.992331ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:53:09.196651 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:53:09.299874 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:53:09.344280 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:53:09.344347 156176 retry.go:31] will retry after 456.954268ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:53:09.802200 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:53:09.926292 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:53:10.026608 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
W0720 17:53:10.026822 156176 start.go:275] error running df -h /var: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
W0720 17:53:10.026848 156176 start.go:242] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:53:10.026911 156176 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0720 17:53:10.026969 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:53:10.118862 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:53:10.208754 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:53:10.208814 156176 retry.go:31] will retry after 281.478285ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:53:10.491460 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:53:10.612094 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:53:10.698805 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
I0720 17:53:10.698878 156176 retry.go:31] will retry after 256.111449ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:53:10.955507 156176 cli_runner.go:164] Run: podman version --format {{.Version}}
I0720 17:53:11.099117 156176 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0720 17:53:11.237857 156176 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:38079 SSHKeyPath:/home/user/.minikube/machines/minikube/id_rsa Username:docker}
W0720 17:53:11.238028 156176 start.go:290] error running df -BG /var: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
W0720 17:53:11.238057 156176 start.go:247] error getting GiB of /var that is available: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/user/.minikube/machines/minikube/id_rsa]}: open /home/user/.minikube/machines/minikube/id_rsa: no such file or directory
I0720 17:53:11.238071 156176 fix.go:56] fixHost completed within 10.586684215s
I0720 17:53:11.238082 156176 start.go:83] releasing machines lock for "minikube", held for 10.586726087s
W0720 17:53:11.238197 156176 out.go:239] ๐Ÿ˜ฟ Failed to start podman container. Running "minikube delete" may fix it: driver start: start: podman start minikube: exit status 125
stdout:

stderr:
Error: unable to start container "606e6a197505aceb29e4ececd6ebf73942619ce9559ebc04a507a00e3db8b4a2": crun: error stat'ing file /dev/vboxusb/001/003: Permission denied: OCI permission denied

I0720 17:53:11.245120 156176 out.go:177]
W0720 17:53:11.248844 156176 out.go:239] โŒ Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: driver start: start: podman start minikube: exit status 125
stdout:

stderr:
Error: unable to start container "606e6a197505aceb29e4ececd6ebf73942619ce9559ebc04a507a00e3db8b4a2": crun: error stat'ing file /dev/vboxusb/001/003: Permission denied: OCI permission denied
`

Operating System

Other

Driver

Podman

@kundan2707
Copy link
Contributor

/kind support

@k8s-ci-robot k8s-ci-robot added the kind/support Categorizes issue or PR as a support question. label Jul 23, 2023
@inuyasha82
Copy link

I"m having the same identical error on fedora 38, with podman version 4.5.1.
Same output .

Trying to create a minikube profile with the following command:

minikube start --driver=podman --cpus=4 --memory=8192 --kubernetes-version 1.17.1 --disk-size=40g --profile podmanenv --container-runtime=cri-o

@afbjorklund
Copy link
Collaborator

afbjorklund commented Aug 29, 2023

It might work better with --container-runtime=containerd (haven't tested)

https://minikube.sigs.k8s.io/docs/drivers/podman/

@afbjorklund afbjorklund added the co/podman-driver podman driver issues label Aug 29, 2023
@osiler
Copy link

osiler commented Dec 12, 2023

Also reporting similar issue using rootless podman and the command ~ Podman 4.6.2

minikube start --driver=podman --container-runtime=containerd

@smegeath
Copy link

Having the same issue with v1.32.0 of minikube and podman 4.6.1 on Rocky 8.9.

@adrian-moisa
Copy link

adrian-moisa commented Feb 21, 2024

Got it working. Thanks @osiler! I'm usually keeping deep notes whenever I'm stuck on a critical problem. I'll share here what worked and what not. There's the TLDR and the long read. Hopefully both help you in the future.

TLDR - In short summary, due to unexpected latencies stemming from cache invalidation I mistakenly thought my podman/minikube setup is broken again. I reinstalled everything. this time minikube did not work. I thought it's a versions issues, tried many combos, nothing worked. I tried rolling back podman, qemu, kubectl and got no progress. Eventually I tried without podman driver and it works using qemu driver. It also works if I specify podman driver and containerd runtime. Wasted 12 hours to find this. Over 25 reinstalls. Thanks to @osiler for sharing the --container-runtime=containerd flag

Full description of diagnosis attempts - Hint: this is the long read.

For some unknown reason my minikube stopped working reliably after I had a laptop sleep cycle (went out a bit). The minikube pods were not able to start (I had an odd feeling seeing them struggle so much). I also noticed podman stats not responding, another sign of trouble. I tried rebuilding one of them but I kept noticing what felt to me strange behaviors: Podman builds hanging (I misinterpreted podman behavior). It felt as if the podman builds do not respond. Then I tried to rebuild the base image of flutter in case there's something wrong there. I had the same odd behavior, lots of hanging. I tried running verbose and again I saw little useful info besides the list of docker ignore files.

After many tries I had a feeling there's an issue with the podman setup. So I decided to do a complete reinstall. This turned out later to be a completely undesired choice. After reinstalling podman I tried again building the docker images and I saw the same behavior of hanging a long time. It occurred to me to check the network traffic and I noticed there's some significant amount. So I deduced that there's unreported activity even if not visible in the terminal. Decided to wait it out and eventually it finished and moved on to the building steps.

This means that podman was basically waiting to download the base image. For some reason it does not report in terminal this download stage. After some more review I realized that podman invalidated the original caches of the go and flutter base images because I changed in many places the dockerignore files. Lesson learned. Being used to see fast and cached builds I've mistaken this delay with problems in podman caused potentially by the stand by period. Knowing that in the past similar events were really hard to debug I decided to cut the chase and to a full reinstall hoping to get back to work quickly. This is what led to the actual problem of not being able to start minikube.

As I explained, I thought I'm doing smth good by skipping the long effort of debugging minikube. Turns out that after the new install minikube no longer worked. Everything seemed to work fine up to the point of starting the control pane. This is where it hangs a lot and fails. Renders some diagnosis with hints to what errors could be the root cause. I searched them on google and GPT did not find anything useful. After checking my podman and minikube versions I remember about the issues I had in the past with a broken version of qemu. So I decided to rollback. After some more hassle there I finally managed to revert podman and qemu to what I knew as the last stable versions. Sadly no luck still no change. Minikube still fails. Despite attempting to completely erasing podman and minikube including settings files.

This was what stood out from the logs. Searching on we I couldn't find anything useful. Logs can be reviewed with minikube logs.

[WARNING SystemVerification]: missing optional cgroups: hugetlb
[WARNING SystemVerification]: missing required cgroups: cpuset
[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
[WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster
To see the stack trace of this error execute with --v=5 or higher

Reinstall Prior Version Of Podman Using podman.rb - No Help
Prior to this event I had an issue with latest qemu breaking podman machine. The solution was to rollback to a prior version of qemu. A few days later the qemu team fixed it. Having this experience I thought this is the same thing again. So I tried to rollback various combinations of podman, qemu, kubectl and minikube. Nothing helped in the end.

  • Installing older brew versions - How To Install Older Versions of Homebrew Packages
  • Last Stable Versions 28 Feb - These are the versions that are currently working, basically all latest. In the end I learned that it's not a version rollback issue. It was a container runtime issue.
    • podman 4.9.3
    • podman-desktop 1.7.1
    • qemu 8.2.1
    • kubectl 1.22.2
    • minikube 1.32.0
    • MacOS Sonoma 14.2

Full uninstall procedure
Eventually I figured out that all I need to remove is minikube. That's why I sorted by categories the cleanup procedure.

# Minikube
minikube stop --all
minikube delete --all --purge
brew uninstall --force minikube
rm -rf ~/.minikube

# Kubernetes CLI
brew uninstall --force kubectl
rm -rf ~/.kube

# Podman
podman stop -a
podman rm -a
brew uninstall --force podman-desktop
brew uninstall --force podman
rm -rf ~/.local/share/containers
rm -rf ~/.config/containers/
rm ~/.ssh/podman*

# Qemu
brew uninstall --force qemu
rm -rf ~/.qemu
rm -rf ~/.config/qemu

Restart procedure
Note: You have to prepare your .rb files as instructed in the tutorial if you want to rollback.

brew install podman.rb
brew reinstall qemu.rb
brew install podman-desktop
brew install kubernetes-cli.rb
brew install minikube.rb

# Double check versions
brew list podman
brew list qemu
brew list podman-desktop
brew list kubectl
brew list minikube

# Init podman
podman machine init
podman machine stop
podman machine set --cpus 2 --memory=8192
podman machine start

# Init minikube
minikube start -p myspace --driver=podman --memory=6g
kubectl create namespace myspace
minikube -p myspace addons enable metrics-server
minikube -p myspace addons enable dashboard
minikube -p myspace dashboard &
minikube tunnel -p myspace

Sometimes brew links are lost

brew link podman
brew link qemu
  • Linking older brew projects - Sometimes they need to be linked manually
  • Installing older podman and qemu made no difference, seems to be minikube issue - Still no progress, all combos of libs fail. Even with complete "factory reset" and system restart.

Attempting Verbose Logs

  • minikube start -p visualspace --driver=podman --memory=6g --alsologtostderr -v=7 - There's a super verbose flag that can be used when starting minikube. I kept getting this error on repeat many times.
E0220 22:52:00.889626   36203 memcache.go:265] couldn't get current server API group list: Get "https://localhost:8443/api?timeout=32s": dial tcp [::1]:8443: connect: connection refused

Trying Older Kubectl - Did the trick - Tried Again, Could Not Reproduce
After trying an older kubectl it finally worked again. I noticed that kubectl release many weekly updates, surely a recent one broke things. The problem started since I downloaded 1.29.2. I tried a second time the older version. Still no luck

Investigating CNI issues
I found a tutorial on the web that suggested to check CNI Troubleshooting the โ€œfailed to create pod sandboxโ€ error. I looked in the issues-journal.md for the previous problems we had with CNI and found these commands to read the state of the network.

  • Missing 1-k8s.conflist - Won't fix it myself GPT: The file /etc/cni/net.d/1-k8s.conflist does not exist. This file is typically responsible for defining how Kubernetes pods should be networked together. The absence of this file suggests that the necessary CNI network configuration for Kubernetes has not been successfully applied or is missing. I'm not going down the route of trying to patch the system after install. I'd rather have it install correctly from the first try.
minikube ssh
docker@minikube:~$ cat /etc/cni/net.d/1-k8s.conflist
cat /etc/cni/net.d/200-loopback.conf

# Results
cat: /etc/cni/net.d/1-k8s.conflist: No such file or directory
{
    "cniVersion": "1.0.0",
    "name": "loopback",
    "type": "loopback"
}
  • Enable CNI param - No help - GPT: Ensure that Minikube is started with the correct network plugin options. Sometimes specifying --network-plugin=cni and --enable-default-cni during Minikube start-up can help ensure the correct CNI configurations are applied. I tried it, makes no difference.

Fix1: Qemu Driver instead of podman - WORKS
I tried running without podman driver. It seems that the entire problem can be fixed by removing the --driver=podman param. When doing so, minikube seems to be using qemu as driver. Podman uses qemu as driver as well. Then why bother at all with the podman driver?

  • Do we need local rootless? - GTP keeps saying that rootless is important. As long as VPS Kubernetes uses smth else to run why does this matter at all? I'm not the main target as a local developer. So rootless local is not such a big win. I guess.
  • No Docker Desktop, No license issue - Also regarding avoiding docker licenses. That is for docker desktop. We use podman for the local development stuff. Therefore there's no strong reason to use podman driver when running minikube (that I know of).

Fix2: Containerd instead of CRI-O - WORKS

  • minikube start --driver=podman --container-runtime=containerd - Found here: Fail to start minikube with root-less podman #16919 - A day ago I I managed to make it work while attempting rollbacks. At one point minikube asked if I want containerd or cri-o. I know that I chose containerd because it is the oldest format on the market since docker. Meaning it's more reliable. The setup worked and I thought I found the right combo of rollbacks. However this was the only time minikube asked which runtime to use. The second day I tried to reinstall all of it again to prove my previous theory of the versions rollback. Again it stopped working. I learned just recently of this param to select container runtime. I don't understand on what basis minikube chooses to ask you or not. It is certain that there is this param by which you can say that you necessarily want containerd. And then it runs ok.

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 21, 2024
@waja
Copy link

waja commented May 21, 2024

/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label May 21, 2024
@kuanyui
Copy link

kuanyui commented Jul 10, 2024

quoted from @adrian-moisa 's reply:

Fix2: Containerd instead of CRI-O - WORKS

  • minikube start --driver=podman --container-runtime=containerd - Found here: Fail to start minikube with root-less podman #16919 - A day ago I I managed to make it work while attempting rollbacks. At one point minikube asked if I want containerd or cri-o. I know that I chose containerd because it is the oldest format on the market since docker. Meaning it's more reliable. The setup worked and I thought I found the right combo of rollbacks. However this was the only time minikube asked which runtime to use. The second day I tried to reinstall all of it again to prove my previous theory of the versions rollback. Again it stopped working. I learned just recently of this param to select container runtime. I don't understand on what basis minikube chooses to ask you or not. It is certain that there is this param by which you can say that you necessarily want containerd. And then it runs ok.

Minikube's official document also said that though

For Rootless Podman, it is recommended to set --container-runtime to containerd:

minikube start --driver=podman --container-runtime=containerd

But I still get same error and failed to start up:

minikube start --driver=podman --container-runtime=containerd --cpus=2 --memory=3000MB
 ~/tmp ๎‚ฐ minikube stop --all  
๐Ÿ›‘  0 node stopped.
 ~/tmp ๎‚ฐ minikube delete --all
๐Ÿ”ฅ  Successfully deleted all profiles
 ~/tmp ๎‚ฐ minikube start --driver=podman --container-runtime=containerd --cpus=2 --memory=3000MB                    
๐Ÿ˜„  minikube v1.33.1 on Opensuse-Tumbleweed 20240703
    โ–ช MINIKUBE_ROOTLESS=true
โœจ  Using the podman driver based on user configuration
๐Ÿ“Œ  Using rootless Podman driver
๐Ÿ‘  Starting "minikube" primary control-plane node in "minikube" cluster
๐Ÿšœ  Pulling base image v0.0.44 ...
E0710 18:59:44.518863 2053124 cache.go:189] Error downloading kic artifacts:  not yet implemented, see issue #8426
๐Ÿ”ฅ  Creating podman container (CPUs=2, Memory=3000MB) ...
๐Ÿคฆ  StartHost failed, but will try again: creating host: create: creating: create kic node: create container: podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=3000mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.44: exit status 127
stdout:

stderr:
Error: crun: opening file `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found

๐Ÿ”„  Restarting existing podman container for "minikube" ...
๐Ÿ˜ฟ  Failed to start podman container. Running "minikube delete" may fix it: driver start: start: podman start minikube: exit status 125
stdout:

stderr:
Error: unable to start container "860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5": container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...


โŒ  Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: driver start: start: podman start minikube: exit status 125
stdout:

stderr:
Error: unable to start container "860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5": container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found , error found in #0 byte of ...||..., bigger context ...||...
minikube logs --file=logs.txt
==> Audit <==
|--------------|--------------------------------|----------|------|---------|---------------------|---------------------|
|   Command    |              Args              | Profile  | User | Version |     Start Time      |      End Time       |
|--------------|--------------------------------|----------|------|---------|---------------------|---------------------|
| config       | set driver podman              | minikube | foo  | v1.33.1 | 19 Jun 24 18:04 CST | 19 Jun 24 18:04 CST |
| start        |                                | minikube | foo  | v1.33.1 | 19 Jun 24 18:04 CST |                     |
| config       | set rootless true              | minikube | foo  | v1.33.1 | 19 Jun 24 18:05 CST | 19 Jun 24 18:05 CST |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 19 Jun 24 18:05 CST |                     |
|              | --container-runtime=containerd |          |      |         |                     |                     |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 19 Jun 24 18:09 CST |                     |
|              | --container-runtime=containerd |          |      |         |                     |                     |
| start        | --driver=kvm2                  | minikube | foo  | v1.33.1 | 19 Jun 24 18:13 CST |                     |
| delete       |                                | minikube | foo  | v1.33.1 | 19 Jun 24 18:13 CST | 19 Jun 24 18:13 CST |
| start        | --driver=kvm2                  | minikube | foo  | v1.33.1 | 19 Jun 24 18:13 CST |                     |
| start        | --driver=kvm2                  | minikube | foo  | v1.33.1 | 19 Jun 24 18:24 CST |                     |
| update-check |                                | minikube | foo  | v1.33.1 | 24 Jun 24 10:02 CST | 24 Jun 24 10:02 CST |
| update-check |                                | minikube | foo  | v1.33.1 | 27 Jun 24 13:11 CST | 27 Jun 24 13:11 CST |
| update-check |                                | minikube | foo  | v1.33.1 | 28 Jun 24 13:35 CST | 28 Jun 24 13:35 CST |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 05 Jul 24 12:53 CST |                     |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 05 Jul 24 12:55 CST |                     |
| delete       |                                | minikube | foo  | v1.33.1 | 05 Jul 24 12:55 CST | 05 Jul 24 12:55 CST |
| delete       |                                | minikube | foo  | v1.33.1 | 05 Jul 24 12:56 CST | 05 Jul 24 12:56 CST |
| start        | --driver=podman --cpus=2       | minikube | foo  | v1.33.1 | 05 Jul 24 12:56 CST |                     |
|              | --memory=2072mb                |          |      |         |                     |                     |
| config       | set rootless true              | minikube | foo  | v1.33.1 | 05 Jul 24 12:58 CST | 05 Jul 24 12:58 CST |
| delete       |                                | minikube | foo  | v1.33.1 | 05 Jul 24 12:58 CST | 05 Jul 24 12:58 CST |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 05 Jul 24 12:58 CST |                     |
|              | --container-runtime=containerd |          |      |         |                     |                     |
|              | --cpus=2 --memory=3000MB       |          |      |         |                     |                     |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 10 Jul 24 18:15 CST |                     |
|              | --container-runtime=cri-o      |          |      |         |                     |                     |
| config       | set rootless true              | minikube | foo  | v1.33.1 | 10 Jul 24 18:27 CST | 10 Jul 24 18:27 CST |
| config       | set rootless true              | minikube | foo  | v1.33.1 | 10 Jul 24 18:27 CST | 10 Jul 24 18:27 CST |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 10 Jul 24 18:27 CST |                     |
|              | --container-runtime=containerd |          |      |         |                     |                     |
| config       | set rootless true              | minikube | foo  | v1.33.1 | 10 Jul 24 18:30 CST | 10 Jul 24 18:30 CST |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 10 Jul 24 18:30 CST |                     |
|              | --container-runtime=containerd |          |      |         |                     |                     |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 10 Jul 24 18:44 CST |                     |
|              | --container-runtime=containerd |          |      |         |                     |                     |
| stop         |                                | minikube | foo  | v1.33.1 | 10 Jul 24 18:44 CST |                     |
| delete       |                                | minikube | foo  | v1.33.1 | 10 Jul 24 18:45 CST | 10 Jul 24 18:45 CST |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 10 Jul 24 18:45 CST |                     |
|              | --container-runtime=containerd |          |      |         |                     |                     |
| stop         | --all                          | minikube | foo  | v1.33.1 | 10 Jul 24 18:52 CST |                     |
| delete       | --all                          | minikube | foo  | v1.33.1 | 10 Jul 24 18:53 CST | 10 Jul 24 18:53 CST |
| stop         | --all                          | minikube | foo  | v1.33.1 | 10 Jul 24 18:53 CST | 10 Jul 24 18:53 CST |
| delete       | --all                          | minikube | foo  | v1.33.1 | 10 Jul 24 18:53 CST | 10 Jul 24 18:54 CST |
| start        | --driver=podman                | minikube | foo  | v1.33.1 | 10 Jul 24 18:59 CST |                     |
|              | --container-runtime=containerd |          |      |         |                     |                     |
|              | --cpus=2 --memory=3000MB       |          |      |         |                     |                     |
|--------------|--------------------------------|----------|------|---------|---------------------|---------------------|


==> Last Start <==
Log file created at: 2024/07/10 18:59:44
Running on machine: foo-laptop
Binary: Built with gc go1.22.3 for linux/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0710 18:59:44.299915 2053124 out.go:291] Setting OutFile to fd 1 ...
I0710 18:59:44.299992 2053124 out.go:343] isatty.IsTerminal(1) = true
I0710 18:59:44.299994 2053124 out.go:304] Setting ErrFile to fd 2...
I0710 18:59:44.299997 2053124 out.go:343] isatty.IsTerminal(2) = true
I0710 18:59:44.300099 2053124 root.go:338] Updating PATH: /home/foo/.minikube/bin
I0710 18:59:44.300522 2053124 out.go:298] Setting JSON to false
I0710 18:59:44.304502 2053124 start.go:129] hostinfo: {"hostname":"foo-laptop","uptime":460814,"bootTime":1720148370,"procs":682,"os":"linux","platform":"opensuse-tumbleweed","platformFamily":"suse","platformVersion":"20240703","kernelVersion":"6.9.7-1-default","kernelArch":"x86_64","virtualizationSystem":"kvm","virtualizationRole":"host","hostId":"76a1bdb4-8bbf-44a6-a612-263bb9579cf8"}
I0710 18:59:44.304532 2053124 start.go:139] virtualization: kvm host
I0710 18:59:44.314028 2053124 out.go:177] ๐Ÿ˜„  minikube v1.33.1 on Opensuse-Tumbleweed 20240703
I0710 18:59:44.315866 2053124 notify.go:220] Checking for updates...
I0710 18:59:44.317310 2053124 out.go:177]     โ–ช MINIKUBE_ROOTLESS=true
I0710 18:59:44.319092 2053124 driver.go:392] Setting default libvirt URI to qemu:///system
I0710 18:59:44.362870 2053124 podman.go:123] podman version: 5.1.1
I0710 18:59:44.364973 2053124 out.go:177] โœจ  Using the podman driver based on user configuration
I0710 18:59:44.366272 2053124 start.go:297] selected driver: podman
I0710 18:59:44.366277 2053124 start.go:901] validating driver "podman" against <nil>
I0710 18:59:44.366284 2053124 start.go:912] status for podman: {Installed:true Healthy:true Running:false NeedsImprovement:false Error:<nil> Reason: Fix: Doc: Version:}
I0710 18:59:44.366337 2053124 cli_runner.go:164] Run: podman system info --format json
I0710 18:59:44.502414 2053124 info.go:288] podman info: {Host:{BuildahVersion:1.36.0 CgroupVersion:v2 Conmon:{Package:conmon-2.1.12-1.1.x86_64 Path:/usr/bin/conmon Version:conmon version 2.1.12, commit: unknown} Distribution:{Distribution:opensuse-tumbleweed Version:20240703} MemFree:3987447808 MemTotal:41729761280 OCIRuntime:{Name:crun Package:crun-1.15-1.1.x86_64 Path:/usr/bin/crun Version:crun version 1.15
commit: e6eacaf4034e84185fd8780ac9262bbf57082278
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:32 Eventlogger:journald Hostname:foo-laptop Kernel:6.9.7-1-default Os:linux Security:{Rootless:true} Uptime:128h 0m 14.00s (Approximately 5.33 days)} Registries:{Search:[registry.opensuse.org registry.suse.com docker.io]} Store:{ConfigFile:/home/foo/.config/containers/storage.conf ContainerStore:{Number:3} GraphDriverName:overlay GraphOptions:{} GraphRoot:/home/foo/.local/share/containers/storage GraphStatus:{BackingFilesystem:extfs NativeOverlayDiff:true SupportsDType:true UsingMetacopy:false} ImageStore:{Number:35} RunRoot:/run/user/1000/containers VolumePath:/home/foo/.local/share/containers/storage/volumes}}
I0710 18:59:44.503555 2053124 start_flags.go:310] no existing cluster config was found, will generate one from the flags
I0710 18:59:44.503659 2053124 start_flags.go:929] Wait components to verify : map[apiserver:true system_pods:true]
I0710 18:59:44.512411 2053124 out.go:177] ๐Ÿ“Œ  Using rootless Podman driver
I0710 18:59:44.513669 2053124 cni.go:84] Creating CNI manager for ""
I0710 18:59:44.513691 2053124 cni.go:143] "podman" driver + "containerd" runtime found, recommending kindnet
I0710 18:59:44.513708 2053124 start_flags.go:319] Found "CNI" CNI - setting NetworkPlugin=cni
I0710 18:59:44.513758 2053124 start.go:340] cluster config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e Memory:3000 CPUs:2 DiskSize:20000 Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.30.0 ClusterName:minikube Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates:KubeletInUserNamespace=true ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.30.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/foo:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s}
I0710 18:59:44.514982 2053124 out.go:177] ๐Ÿ‘  Starting "minikube" primary control-plane node in "minikube" cluster
I0710 18:59:44.516085 2053124 cache.go:121] Beginning downloading kic base image for podman with containerd
I0710 18:59:44.517296 2053124 out.go:177] ๐Ÿšœ  Pulling base image v0.0.44 ...
I0710 18:59:44.518363 2053124 preload.go:132] Checking if preload exists for k8s version v1.30.0 and runtime containerd
I0710 18:59:44.518383 2053124 preload.go:147] Found local preload: /home/foo/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-containerd-overlay2-amd64.tar.lz4
I0710 18:59:44.518388 2053124 cache.go:56] Caching tarball of preloaded images
I0710 18:59:44.518413 2053124 cache.go:149] Downloading gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e to local cache
I0710 18:59:44.518434 2053124 preload.go:173] Found /home/foo/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-containerd-overlay2-amd64.tar.lz4 in cache, skipping download
I0710 18:59:44.518439 2053124 cache.go:59] Finished verifying existence of preloaded tar for v1.30.0 on containerd
I0710 18:59:44.518494 2053124 image.go:63] Checking for gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e in local cache directory
I0710 18:59:44.518500 2053124 image.go:66] Found gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e in local cache directory, skipping pull
I0710 18:59:44.518502 2053124 image.go:105] gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e exists in cache, skipping pull
I0710 18:59:44.518506 2053124 cache.go:152] successfully saved gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e as a tarball
I0710 18:59:44.518793 2053124 profile.go:143] Saving config to /home/foo/.minikube/profiles/minikube/config.json ...
I0710 18:59:44.518803 2053124 lock.go:35] WriteFile acquiring /home/foo/.minikube/profiles/minikube/config.json: {Name:mk393008f67e129369cbf6432e0200e24ffb79cb Clock:{} Delay:500ms Timeout:1m0s Cancel:<nil>}
E0710 18:59:44.518863 2053124 cache.go:189] Error downloading kic artifacts:  not yet implemented, see issue #8426
I0710 18:59:44.518870 2053124 cache.go:194] Successfully downloaded all kic artifacts
I0710 18:59:44.518883 2053124 start.go:360] acquireMachinesLock for minikube: {Name:mk9c1ba480d385ef3dfcd96f457b2280829621b8 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0710 18:59:44.518894 2053124 start.go:364] duration metric: took 6.349ยตs to acquireMachinesLock for "minikube"
I0710 18:59:44.518899 2053124 start.go:93] Provisioning new machine with config: &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO: KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.44@sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e Memory:3000 CPUs:2 DiskSize:20000 Driver:podman HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.59.1/24 HypervVirtualSwitch: HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 APIServerPort:8443 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.30.0 ClusterName:minikube Namespace:default APIServerHAVIP: APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:containerd CRISocket: NetworkPlugin:cni FeatureGates:KubeletInUserNamespace=true ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: RegistryAliases: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI:} Nodes:[{Name: IP: Port:8443 KubernetesVersion:v1.30.0 ContainerRuntime:containerd ControlPlane:true Worker:true}] Addons:map[] CustomAddonImages:map[] CustomAddonRegistries:map[] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop:<nil> ExposedPorts:[] ListenAddress: Network: Subnet: MultiNodeRequested:false ExtraDisks:0 CertExpiration:26280h0m0s Mount:false MountString:/home/foo:/minikube-host Mount9PVersion:9p2000.L MountGID:docker MountIP: MountMSize:262144 MountOptions:[] MountPort:0 MountType:9p MountUID:docker BinaryMirror: DisableOptimizations:false DisableMetrics:false CustomQemuFirmwarePath: SocketVMnetClientPath: SocketVMnetPath: StaticIP: SSHAuthSock: SSHAgentPID:0 GPUs: AutoPauseInterval:1m0s} &{Name: IP: Port:8443 KubernetesVersion:v1.30.0 ContainerRuntime:containerd ControlPlane:true Worker:true}
I0710 18:59:44.518929 2053124 start.go:125] createHost starting for "" (driver="podman")
I0710 18:59:44.521105 2053124 out.go:204] ๐Ÿ”ฅ  Creating podman container (CPUs=2, Memory=3000MB) ...
I0710 18:59:44.521251 2053124 start.go:159] libmachine.API.Create for "minikube" (driver="podman")
I0710 18:59:44.521263 2053124 client.go:168] LocalClient.Create starting
I0710 18:59:44.521291 2053124 main.go:141] libmachine: Reading certificate data from /home/foo/.minikube/certs/ca.pem
I0710 18:59:44.521308 2053124 main.go:141] libmachine: Decoding PEM data...
I0710 18:59:44.521316 2053124 main.go:141] libmachine: Parsing certificate...
I0710 18:59:44.521345 2053124 main.go:141] libmachine: Reading certificate data from /home/foo/.minikube/certs/cert.pem
I0710 18:59:44.521353 2053124 main.go:141] libmachine: Decoding PEM data...
I0710 18:59:44.521358 2053124 main.go:141] libmachine: Parsing certificate...
I0710 18:59:44.521561 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:44.548157 2053124 cli_runner.go:164] Run: podman network inspect minikube --format "{{range .}}{{if eq .Driver "bridge"}}{{(index .Subnets 0).Subnet}},{{(index .Subnets 0).Gateway}}{{end}}{{end}}"
W0710 18:59:44.599865 2053124 cli_runner.go:211] podman network inspect minikube --format "{{range .}}{{if eq .Driver "bridge"}}{{(index .Subnets 0).Subnet}},{{(index .Subnets 0).Gateway}}{{end}}{{end}}" returned with exit code 125
I0710 18:59:44.599962 2053124 network_create.go:281] running [podman network inspect minikube] to gather additional debugging logs...
I0710 18:59:44.599980 2053124 cli_runner.go:164] Run: podman network inspect minikube
W0710 18:59:44.646590 2053124 cli_runner.go:211] podman network inspect minikube returned with exit code 125
I0710 18:59:44.646610 2053124 network_create.go:284] error running [podman network inspect minikube]: podman network inspect minikube: exit status 125
stdout:
[]

stderr:
Error: network minikube: unable to find network with name or ID minikube: network not found
I0710 18:59:44.646623 2053124 network_create.go:286] output of [podman network inspect minikube]: -- stdout --
[]

-- /stdout --
** stderr **
Error: network minikube: unable to find network with name or ID minikube: network not found

** /stderr **
I0710 18:59:44.646686 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:44.677961 2053124 cli_runner.go:164] Run: podman network inspect podman --format "{{range .}}{{if eq .Driver "bridge"}}{{(index .Subnets 0).Subnet}},{{(index .Subnets 0).Gateway}}{{end}}{{end}}"
I0710 18:59:44.724201 2053124 network.go:206] using free private subnet 192.168.49.0/24: &{IP:192.168.49.0 Netmask:255.255.255.0 Prefix:24 CIDR:192.168.49.0/24 Gateway:192.168.49.1 ClientMin:192.168.49.2 ClientMax:192.168.49.254 Broadcast:192.168.49.255 IsPrivate:true Interface:{IfaceName: IfaceIPv4: IfaceMTU:0 IfaceMAC:} reservation:0xc001703fd0}
I0710 18:59:44.724241 2053124 network_create.go:124] attempt to create podman network minikube 192.168.49.0/24 with gateway 192.168.49.1 and MTU of 0 ...
I0710 18:59:44.724304 2053124 cli_runner.go:164] Run: podman network create --driver=bridge --subnet=192.168.49.0/24 --gateway=192.168.49.1 --label=created_by.minikube.sigs.k8s.io=true --label=name.minikube.sigs.k8s.io=minikube minikube
I0710 18:59:44.763916 2053124 network_create.go:108] podman network minikube 192.168.49.0/24 created
I0710 18:59:44.763934 2053124 kic.go:121] calculated static IP "192.168.49.2" for the "minikube" container
I0710 18:59:44.764001 2053124 cli_runner.go:164] Run: podman ps -a --format {{.Names}}
I0710 18:59:44.799485 2053124 cli_runner.go:164] Run: podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true
I0710 18:59:44.851396 2053124 oci.go:103] Successfully created a podman volume minikube
I0710 18:59:44.851500 2053124 cli_runner.go:164] Run: podman run --rm --name minikube-preload-sidecar --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --entrypoint /usr/bin/test -v minikube:/var gcr.io/k8s-minikube/kicbase:v0.0.44 -d /var/lib
I0710 18:59:45.175360 2053124 oci.go:107] Successfully prepared a podman volume minikube
I0710 18:59:45.175439 2053124 preload.go:132] Checking if preload exists for k8s version v1.30.0 and runtime containerd
I0710 18:59:45.175469 2053124 kic.go:194] Starting extracting preloaded images to volume ...
I0710 18:59:45.175577 2053124 cli_runner.go:164] Run: podman run --rm --entrypoint /usr/bin/tar --security-opt label=disable -v /home/foo/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-containerd-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.44 -I lz4 -xf /preloaded.tar -C /extractDir
I0710 18:59:46.315427 2053124 cli_runner.go:217] Completed: podman run --rm --entrypoint /usr/bin/tar --security-opt label=disable -v /home/foo/.minikube/cache/preloaded-tarball/preloaded-images-k8s-v18-v1.30.0-containerd-overlay2-amd64.tar.lz4:/preloaded.tar:ro -v minikube:/extractDir gcr.io/k8s-minikube/kicbase:v0.0.44 -I lz4 -xf /preloaded.tar -C /extractDir: (1.139803758s)
I0710 18:59:46.315455 2053124 kic.go:203] duration metric: took 1.139982415s to extract preloaded images to volume ...
W0710 18:59:46.315561 2053124 cgroups_linux.go:77] Your kernel does not support swap limit capabilities or the cgroup is not mounted.
W0710 18:59:46.315606 2053124 oci.go:243] Your kernel does not support CPU cfs period/quota or the cgroup is not mounted.
I0710 18:59:46.315676 2053124 cli_runner.go:164] Run: podman info --format "'{{json .SecurityOptions}}'"
W0710 18:59:46.437806 2053124 cli_runner.go:211] podman info --format "'{{json .SecurityOptions}}'" returned with exit code 125
I0710 18:59:46.437885 2053124 cli_runner.go:164] Run: podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=3000mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.44
W0710 18:59:47.436338 2053124 cli_runner.go:211] podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=3000mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.44 returned with exit code 127
I0710 18:59:47.436375 2053124 client.go:171] duration metric: took 2.915108399s to LocalClient.Create
I0710 18:59:49.437608 2053124 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0710 18:59:49.437652 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:49.455716 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:49.480144 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 18:59:49.480206 2053124 retry.go:31] will retry after 284.341444ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 18:59:49.764701 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:49.798943 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:49.841346 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 18:59:49.841481 2053124 retry.go:31] will retry after 534.289803ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 18:59:50.376703 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:50.417271 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:50.445561 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 18:59:50.445611 2053124 retry.go:31] will retry after 521.23067ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 18:59:50.967731 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:50.987744 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:51.018288 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
W0710 18:59:51.018342 2053124 start.go:268] error running df -h /var: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
W0710 18:59:51.018350 2053124 start.go:235] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 18:59:51.018371 2053124 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0710 18:59:51.018407 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:51.042502 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:51.085647 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 18:59:51.085763 2053124 retry.go:31] will retry after 281.951475ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 18:59:51.368687 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:51.408004 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:51.440981 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 18:59:51.441092 2053124 retry.go:31] will retry after 530.958428ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 18:59:51.972750 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:51.997409 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:52.022734 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 18:59:52.022784 2053124 retry.go:31] will retry after 509.280416ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 18:59:52.533179 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:52.569029 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:52.613353 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
W0710 18:59:52.613433 2053124 start.go:283] error running df -BG /var: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
W0710 18:59:52.613442 2053124 start.go:240] error getting GiB of /var that is available: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 18:59:52.613447 2053124 start.go:128] duration metric: took 8.094514437s to createHost
I0710 18:59:52.613453 2053124 start.go:83] releasing machines lock for "minikube", held for 8.094555109s
W0710 18:59:52.613463 2053124 start.go:713] error starting host: creating host: create: creating: create kic node: create container: podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=3000mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.44: exit status 127
stdout:

stderr:
Error: crun: opening file `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found
I0710 18:59:52.613783 2053124 cli_runner.go:164] Run: podman container inspect minikube --format={{.State.Status}}
W0710 18:59:52.679624 2053124 start.go:718] delete host: Docker machine "minikube" does not exist. Use "docker-machine ls" to list machines. Use "docker-machine create" to add a new one.
W0710 18:59:52.679743 2053124 out.go:239] ๐Ÿคฆ  StartHost failed, but will try again: creating host: create: creating: create kic node: create container: podman run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname minikube --name minikube --label created_by.minikube.sigs.k8s.io=true --label name.minikube.sigs.k8s.io=minikube --label role.minikube.sigs.k8s.io= --label mode.minikube.sigs.k8s.io=minikube --network minikube --ip 192.168.49.2 --volume minikube:/var:exec --memory=3000mb -e container=podman --expose 8443 --publish=127.0.0.1::8443 --publish=127.0.0.1::22 --publish=127.0.0.1::2376 --publish=127.0.0.1::5000 --publish=127.0.0.1::32443 gcr.io/k8s-minikube/kicbase:v0.0.44: exit status 127
stdout:

stderr:
Error: crun: opening file `memory.max` for writing: No such file or directory: OCI runtime attempted to invoke a command that was not found

I0710 18:59:52.679793 2053124 start.go:728] Will try again in 5 seconds ...
I0710 18:59:57.681524 2053124 start.go:360] acquireMachinesLock for minikube: {Name:mk9c1ba480d385ef3dfcd96f457b2280829621b8 Clock:{} Delay:500ms Timeout:10m0s Cancel:<nil>}
I0710 18:59:57.681635 2053124 start.go:364] duration metric: took 69.33ยตs to acquireMachinesLock for "minikube"
I0710 18:59:57.681669 2053124 start.go:96] Skipping create...Using existing machine configuration
I0710 18:59:57.681675 2053124 fix.go:54] fixHost starting:
I0710 18:59:57.682019 2053124 cli_runner.go:164] Run: podman container inspect minikube --format={{.State.Status}}
I0710 18:59:57.721403 2053124 fix.go:112] recreateIfNeeded on minikube: state= err=<nil>
W0710 18:59:57.721422 2053124 fix.go:138] unexpected machine state, will restart: <nil>
I0710 18:59:57.730278 2053124 out.go:177] ๐Ÿ”„  Restarting existing podman container for "minikube" ...
I0710 18:59:57.731684 2053124 cli_runner.go:164] Run: podman start minikube
W0710 18:59:58.509916 2053124 cli_runner.go:211] podman start minikube returned with exit code 125
I0710 18:59:58.509964 2053124 cli_runner.go:164] Run: podman inspect minikube
I0710 18:59:58.548628 2053124 errors.go:84] Postmortem inspect ("podman inspect minikube"): -- stdout --
[
     {
          "Id": "860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5",
          "Created": "2024-07-10T18:59:46.465418686+08:00",
          "Path": "/usr/local/bin/entrypoint",
          "Args": [
               "/sbin/init"
          ],
          "State": {
               "OciVersion": "1.2.0",
               "Status": "created",
               "Running": false,
               "Paused": false,
               "Restarting": false,
               "OOMKilled": false,
               "Dead": false,
               "Pid": 0,
               "ExitCode": 0,
               "Error": "container create failed (no logs from conmon): conmon bytes \"\": readObjectStart: expect { or n, but found \u0000, error found in #0 byte of ...||..., bigger context ...||...",
               "StartedAt": "0001-01-01T00:00:00Z",
               "FinishedAt": "0001-01-01T00:00:00Z",
               "CheckpointedAt": "0001-01-01T00:00:00Z",
               "RestoredAt": "0001-01-01T00:00:00Z"
          },
          "Image": "5a6e59a9bdc0d32876fd51e3702c6cb16f38b145ed5528e5f0bfb1de21e70803",
          "ImageDigest": "sha256:eb04641328b06c5c4a14f4348470e1046bbcf9c2cbc551486e343d3a49db557e",
          "ImageName": "gcr.io/k8s-minikube/kicbase:v0.0.44",
          "Rootfs": "",
          "Pod": "",
          "ResolvConfPath": "/run/user/1000/containers/overlay-containers/860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5/userdata/resolv.conf",
          "HostnamePath": "/run/user/1000/containers/overlay-containers/860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5/userdata/hostname",
          "HostsPath": "/run/user/1000/containers/overlay-containers/860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5/userdata/hosts",
          "StaticDir": "/home/foo/.local/share/containers/storage/overlay-containers/860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5/userdata",
          "OCIConfigPath": "/home/foo/.local/share/containers/storage/overlay-containers/860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5/userdata/config.json",
          "OCIRuntime": "crun",
          "ConmonPidFile": "/run/user/1000/containers/overlay-containers/860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5/userdata/conmon.pid",
          "PidFile": "/run/user/1000/containers/overlay-containers/860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5/userdata/pidfile",
          "Name": "minikube",
          "RestartCount": 0,
          "Driver": "overlay",
          "MountLabel": "",
          "ProcessLabel": "",
          "AppArmorProfile": "",
          "EffectiveCaps": [
               "CAP_AUDIT_CONTROL",
               "CAP_AUDIT_READ",
               "CAP_AUDIT_WRITE",
               "CAP_BLOCK_SUSPEND",
               "CAP_BPF",
               "CAP_CHECKPOINT_RESTORE",
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_DAC_READ_SEARCH",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_IPC_LOCK",
               "CAP_IPC_OWNER",
               "CAP_KILL",
               "CAP_LEASE",
               "CAP_LINUX_IMMUTABLE",
               "CAP_MAC_ADMIN",
               "CAP_MAC_OVERRIDE",
               "CAP_MKNOD",
               "CAP_NET_ADMIN",
               "CAP_NET_BIND_SERVICE",
               "CAP_NET_BROADCAST",
               "CAP_NET_RAW",
               "CAP_PERFMON",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYSLOG",
               "CAP_SYS_ADMIN",
               "CAP_SYS_BOOT",
               "CAP_SYS_CHROOT",
               "CAP_SYS_MODULE",
               "CAP_SYS_NICE",
               "CAP_SYS_PACCT",
               "CAP_SYS_PTRACE",
               "CAP_SYS_RAWIO",
               "CAP_SYS_RESOURCE",
               "CAP_SYS_TIME",
               "CAP_SYS_TTY_CONFIG",
               "CAP_WAKE_ALARM"
          ],
          "BoundingCaps": [
               "CAP_AUDIT_CONTROL",
               "CAP_AUDIT_READ",
               "CAP_AUDIT_WRITE",
               "CAP_BLOCK_SUSPEND",
               "CAP_BPF",
               "CAP_CHECKPOINT_RESTORE",
               "CAP_CHOWN",
               "CAP_DAC_OVERRIDE",
               "CAP_DAC_READ_SEARCH",
               "CAP_FOWNER",
               "CAP_FSETID",
               "CAP_IPC_LOCK",
               "CAP_IPC_OWNER",
               "CAP_KILL",
               "CAP_LEASE",
               "CAP_LINUX_IMMUTABLE",
               "CAP_MAC_ADMIN",
               "CAP_MAC_OVERRIDE",
               "CAP_MKNOD",
               "CAP_NET_ADMIN",
               "CAP_NET_BIND_SERVICE",
               "CAP_NET_BROADCAST",
               "CAP_NET_RAW",
               "CAP_PERFMON",
               "CAP_SETFCAP",
               "CAP_SETGID",
               "CAP_SETPCAP",
               "CAP_SETUID",
               "CAP_SYSLOG",
               "CAP_SYS_ADMIN",
               "CAP_SYS_BOOT",
               "CAP_SYS_CHROOT",
               "CAP_SYS_MODULE",
               "CAP_SYS_NICE",
               "CAP_SYS_PACCT",
               "CAP_SYS_PTRACE",
               "CAP_SYS_RAWIO",
               "CAP_SYS_RESOURCE",
               "CAP_SYS_TIME",
               "CAP_SYS_TTY_CONFIG",
               "CAP_WAKE_ALARM"
          ],
          "ExecIDs": [],
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/home/foo/.local/share/containers/storage/overlay/8c5d4477ed8983e0ca9543208f7ee26d96aa4c3def1c73b20a8ba13dcbccd61a/diff",
                    "UpperDir": "/home/foo/.local/share/containers/storage/overlay/9bdfb2a7ae8519e8efebd2afb1cd619e5f753077c2b31d5b095eca49ff29495e/diff",
                    "WorkDir": "/home/foo/.local/share/containers/storage/overlay/9bdfb2a7ae8519e8efebd2afb1cd619e5f753077c2b31d5b095eca49ff29495e/work"
               }
          },
          "Mounts": [
               {
                    "Type": "volume",
                    "Name": "minikube",
                    "Source": "/home/foo/.local/share/containers/storage/volumes/minikube/_data",
                    "Destination": "/var",
                    "Driver": "local",
                    "Mode": "",
                    "Options": [
                         "exec",
                         "nosuid",
                         "nodev",
                         "rbind"
                    ],
                    "RW": true,
                    "Propagation": "rprivate"
               },
               {
                    "Type": "bind",
                    "Source": "/lib/modules",
                    "Destination": "/lib/modules",
                    "Driver": "",
                    "Mode": "",
                    "Options": [
                         "rbind"
                    ],
                    "RW": false,
                    "Propagation": "rprivate"
               }
          ],
          "Dependencies": [],
          "NetworkSettings": {
               "EndpointID": "",
               "Gateway": "",
               "IPAddress": "",
               "IPPrefixLen": 0,
               "IPv6Gateway": "",
               "GlobalIPv6Address": "",
               "GlobalIPv6PrefixLen": 0,
               "MacAddress": "",
               "Bridge": "",
               "SandboxID": "",
               "HairpinMode": false,
               "LinkLocalIPv6Address": "",
               "LinkLocalIPv6PrefixLen": 0,
               "Ports": {
                    "22/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "39251"
                         }
                    ],
                    "2376/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "39663"
                         }
                    ],
                    "32443/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "35655"
                         }
                    ],
                    "5000/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "36489"
                         }
                    ],
                    "8443/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "38735"
                         }
                    ]
               },
               "SandboxKey": "",
               "Networks": {
                    "minikube": {
                         "EndpointID": "",
                         "Gateway": "",
                         "IPAddress": "",
                         "IPPrefixLen": 0,
                         "IPv6Gateway": "",
                         "GlobalIPv6Address": "",
                         "GlobalIPv6PrefixLen": 0,
                         "MacAddress": "",
                         "NetworkID": "minikube",
                         "DriverOpts": null,
                         "IPAMConfig": null,
                         "Links": null,
                         "Aliases": [
                              "860174a13a4a",
                              "minikube"
                         ]
                    }
               }
          },
          "Namespace": "",
          "IsInfra": false,
          "IsService": false,
          "KubeExitCodePropagation": "invalid",
          "lockNumber": 1,
          "Config": {
               "Hostname": "minikube",
               "Domainname": "",
               "User": "",
               "AttachStdin": false,
               "AttachStdout": false,
               "AttachStderr": false,
               "Tty": true,
               "OpenStdin": false,
               "StdinOnce": false,
               "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                    "container=podman",
                    "TERM=xterm",
                    "HOSTNAME=minikube",
                    "HOME=/root"
               ],
               "Cmd": null,
               "Image": "gcr.io/k8s-minikube/kicbase:v0.0.44",
               "Volumes": null,
               "WorkingDir": "/",
               "Entrypoint": [
                    "/usr/local/bin/entrypoint",
                    "/sbin/init"
               ],
               "OnBuild": null,
               "Labels": {
                    "created_by.minikube.sigs.k8s.io": "true",
                    "mode.minikube.sigs.k8s.io": "minikube",
                    "name.minikube.sigs.k8s.io": "minikube",
                    "role.minikube.sigs.k8s.io": ""
               },
               "Annotations": {
                    "io.container.manager": "libpod",
                    "io.podman.annotations.privileged": "TRUE",
                    "io.podman.annotations.seccomp": "unconfined",
                    "org.opencontainers.image.stopSignal": "37"
               },
               "StopSignal": "SIGRTMIN+3",
               "HealthcheckOnFailureAction": "none",
               "CreateCommand": [
                    "podman",
                    "run",
                    "-d",
                    "-t",
                    "--privileged",
                    "--security-opt",
                    "seccomp=unconfined",
                    "--tmpfs",
                    "/tmp",
                    "--tmpfs",
                    "/run",
                    "-v",
                    "/lib/modules:/lib/modules:ro",
                    "--hostname",
                    "minikube",
                    "--name",
                    "minikube",
                    "--label",
                    "created_by.minikube.sigs.k8s.io=true",
                    "--label",
                    "name.minikube.sigs.k8s.io=minikube",
                    "--label",
                    "role.minikube.sigs.k8s.io=",
                    "--label",
                    "mode.minikube.sigs.k8s.io=minikube",
                    "--network",
                    "minikube",
                    "--ip",
                    "192.168.49.2",
                    "--volume",
                    "minikube:/var:exec",
                    "--memory=3000mb",
                    "-e",
                    "container=podman",
                    "--expose",
                    "8443",
                    "--publish=127.0.0.1::8443",
                    "--publish=127.0.0.1::22",
                    "--publish=127.0.0.1::2376",
                    "--publish=127.0.0.1::5000",
                    "--publish=127.0.0.1::32443",
                    "gcr.io/k8s-minikube/kicbase:v0.0.44"
               ],
               "Umask": "0022",
               "Timeout": 0,
               "StopTimeout": 10,
               "Passwd": true,
               "sdNotifyMode": "container"
          },
          "HostConfig": {
               "Binds": [
                    "minikube:/var:exec,rw,rprivate,nosuid,nodev,rbind",
                    "/lib/modules:/lib/modules:ro,rprivate,rbind"
               ],
               "CgroupManager": "systemd",
               "CgroupMode": "private",
               "ContainerIDFile": "",
               "LogConfig": {
                    "Type": "journald",
                    "Config": null,
                    "Path": "",
                    "Tag": "",
                    "Size": "0B"
               },
               "NetworkMode": "bridge",
               "PortBindings": {
                    "22/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "39251"
                         }
                    ],
                    "2376/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "39663"
                         }
                    ],
                    "32443/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "35655"
                         }
                    ],
                    "5000/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "36489"
                         }
                    ],
                    "8443/tcp": [
                         {
                              "HostIp": "127.0.0.1",
                              "HostPort": "38735"
                         }
                    ]
               },
               "RestartPolicy": {
                    "Name": "no",
                    "MaximumRetryCount": 0
               },
               "AutoRemove": false,
               "Annotations": {
                    "io.container.manager": "libpod",
                    "io.podman.annotations.privileged": "TRUE",
                    "io.podman.annotations.seccomp": "unconfined",
                    "org.opencontainers.image.stopSignal": "37"
               },
               "VolumeDriver": "",
               "VolumesFrom": null,
               "CapAdd": [],
               "CapDrop": [],
               "Dns": [],
               "DnsOptions": [],
               "DnsSearch": [],
               "ExtraHosts": [],
               "GroupAdd": [],
               "IpcMode": "shareable",
               "Cgroup": "",
               "Cgroups": "default",
               "Links": null,
               "OomScoreAdj": 0,
               "PidMode": "private",
               "Privileged": true,
               "PublishAllPorts": false,
               "ReadonlyRootfs": false,
               "SecurityOpt": [
                    "seccomp=unconfined",
                    "unmask=all"
               ],
               "Tmpfs": {
                    "/run": "rw,rprivate,nosuid,nodev,tmpcopyup",
                    "/tmp": "rw,rprivate,nosuid,nodev,tmpcopyup"
               },
               "UTSMode": "private",
               "UsernsMode": "",
               "ShmSize": 65536000,
               "Runtime": "oci",
               "ConsoleSize": [
                    0,
                    0
               ],
               "Isolation": "",
               "CpuShares": 0,
               "Memory": 3145728000,
               "NanoCpus": 0,
               "CgroupParent": "user.slice",
               "BlkioWeight": 0,
               "BlkioWeightDevice": null,
               "BlkioDeviceReadBps": null,
               "BlkioDeviceWriteBps": null,
               "BlkioDeviceReadIOps": null,
               "BlkioDeviceWriteIOps": null,
               "CpuPeriod": 0,
               "CpuQuota": 0,
               "CpuRealtimePeriod": 0,
               "CpuRealtimeRuntime": 0,
               "CpusetCpus": "",
               "CpusetMems": "",
               "Devices": [],
               "DiskQuota": 0,
               "KernelMemory": 0,
               "MemoryReservation": 0,
               "MemorySwap": 6291456000,
               "MemorySwappiness": -1,
               "OomKillDisable": false,
               "PidsLimit": 2048,
               "Ulimits": [
                    {
                         "Name": "RLIMIT_NOFILE",
                         "Soft": 524288,
                         "Hard": 524288
                    },
                    {
                         "Name": "RLIMIT_NPROC",
                         "Soft": 158611,
                         "Hard": 158611
                    }
               ],
               "CpuCount": 0,
               "CpuPercent": 0,
               "IOMaximumIOps": 0,
               "IOMaximumBandwidth": 0,
               "CgroupConf": null
          }
     }
]

-- /stdout --
I0710 18:59:58.548689 2053124 cli_runner.go:164] Run: podman logs --timestamps minikube
I0710 18:59:58.661828 2053124 errors.go:91] Postmortem logs ("podman logs --timestamps minikube"):
I0710 18:59:58.661896 2053124 cli_runner.go:164] Run: podman system info --format json
I0710 18:59:58.841772 2053124 info.go:288] podman info: {Host:{BuildahVersion:1.36.0 CgroupVersion:v2 Conmon:{Package:conmon-2.1.12-1.1.x86_64 Path:/usr/bin/conmon Version:conmon version 2.1.12, commit: unknown} Distribution:{Distribution:opensuse-tumbleweed Version:20240703} MemFree:2747793408 MemTotal:41729761280 OCIRuntime:{Name:crun Package:crun-1.15-1.1.x86_64 Path:/usr/bin/crun Version:crun version 1.15
commit: e6eacaf4034e84185fd8780ac9262bbf57082278
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:32 Eventlogger:journald Hostname:foo-laptop Kernel:6.9.7-1-default Os:linux Security:{Rootless:true} Uptime:128h 0m 28.00s (Approximately 5.33 days)} Registries:{Search:[registry.opensuse.org registry.suse.com docker.io]} Store:{ConfigFile:/home/foo/.config/containers/storage.conf ContainerStore:{Number:4} GraphDriverName:overlay GraphOptions:{} GraphRoot:/home/foo/.local/share/containers/storage GraphStatus:{BackingFilesystem:extfs NativeOverlayDiff:true SupportsDType:true UsingMetacopy:false} ImageStore:{Number:35} RunRoot:/run/user/1000/containers VolumePath:/home/foo/.local/share/containers/storage/volumes}}
I0710 18:59:58.841810 2053124 errors.go:106] postmortem podman info: {Host:{BuildahVersion:1.36.0 CgroupVersion:v2 Conmon:{Package:conmon-2.1.12-1.1.x86_64 Path:/usr/bin/conmon Version:conmon version 2.1.12, commit: unknown} Distribution:{Distribution:opensuse-tumbleweed Version:20240703} MemFree:2747793408 MemTotal:41729761280 OCIRuntime:{Name:crun Package:crun-1.15-1.1.x86_64 Path:/usr/bin/crun Version:crun version 1.15
commit: e6eacaf4034e84185fd8780ac9262bbf57082278
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:32 Eventlogger:journald Hostname:foo-laptop Kernel:6.9.7-1-default Os:linux Security:{Rootless:true} Uptime:128h 0m 28.00s (Approximately 5.33 days)} Registries:{Search:[registry.opensuse.org registry.suse.com docker.io]} Store:{ConfigFile:/home/foo/.config/containers/storage.conf ContainerStore:{Number:4} GraphDriverName:overlay GraphOptions:{} GraphRoot:/home/foo/.local/share/containers/storage GraphStatus:{BackingFilesystem:extfs NativeOverlayDiff:true SupportsDType:true UsingMetacopy:false} ImageStore:{Number:35} RunRoot:/run/user/1000/containers VolumePath:/home/foo/.local/share/containers/storage/volumes}}
I0710 18:59:58.841868 2053124 network_create.go:281] running [podman network inspect minikube] to gather additional debugging logs...
I0710 18:59:58.841886 2053124 cli_runner.go:164] Run: podman network inspect minikube
I0710 18:59:58.876812 2053124 network_create.go:286] output of [podman network inspect minikube]: -- stdout --
[
     {
          "name": "minikube",
          "id": "df2b61078ce050abebe11edfb4e5bdfd8138fc3d875b00cd779ce26961e7260a",
          "driver": "bridge",
          "network_interface": "podman5",
          "created": "2024-07-10T18:59:44.760785293+08:00",
          "subnets": [
               {
                    "subnet": "192.168.49.0/24",
                    "gateway": "192.168.49.1"
               }
          ],
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "labels": {
               "created_by.minikube.sigs.k8s.io": "true",
               "name.minikube.sigs.k8s.io": "minikube"
          },
          "ipam_options": {
               "driver": "host-local"
          },
          "containers": {}
     }
]

-- /stdout --
I0710 18:59:58.876887 2053124 cli_runner.go:164] Run: podman system info --format json
I0710 18:59:58.985572 2053124 info.go:288] podman info: {Host:{BuildahVersion:1.36.0 CgroupVersion:v2 Conmon:{Package:conmon-2.1.12-1.1.x86_64 Path:/usr/bin/conmon Version:conmon version 2.1.12, commit: unknown} Distribution:{Distribution:opensuse-tumbleweed Version:20240703} MemFree:2765737984 MemTotal:41729761280 OCIRuntime:{Name:crun Package:crun-1.15-1.1.x86_64 Path:/usr/bin/crun Version:crun version 1.15
commit: e6eacaf4034e84185fd8780ac9262bbf57082278
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL} SwapFree:0 SwapTotal:0 Arch:amd64 Cpus:32 Eventlogger:journald Hostname:foo-laptop Kernel:6.9.7-1-default Os:linux Security:{Rootless:true} Uptime:128h 0m 28.00s (Approximately 5.33 days)} Registries:{Search:[registry.opensuse.org registry.suse.com docker.io]} Store:{ConfigFile:/home/foo/.config/containers/storage.conf ContainerStore:{Number:4} GraphDriverName:overlay GraphOptions:{} GraphRoot:/home/foo/.local/share/containers/storage GraphStatus:{BackingFilesystem:extfs NativeOverlayDiff:true SupportsDType:true UsingMetacopy:false} ImageStore:{Number:35} RunRoot:/run/user/1000/containers VolumePath:/home/foo/.local/share/containers/storage/volumes}}
I0710 18:59:58.985653 2053124 ssh_runner.go:195] Run: sh -c "df -h /var | awk 'NR==2{print $5}'"
I0710 18:59:58.985678 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:59.005493 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:59.036899 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 18:59:59.036952 2053124 retry.go:31] will retry after 360.910864ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 18:59:59.398653 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:59.429963 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:59.457480 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 18:59:59.457570 2053124 retry.go:31] will retry after 386.332857ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 18:59:59.844291 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 18:59:59.883095 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 18:59:59.926183 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 18:59:59.926233 2053124 retry.go:31] will retry after 311.33551ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 19:00:00.237848 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 19:00:00.277369 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 19:00:00.316043 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 19:00:00.316168 2053124 retry.go:31] will retry after 452.542637ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 19:00:00.769899 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 19:00:00.797882 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 19:00:00.834746 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
W0710 19:00:00.834911 2053124 start.go:268] error running df -h /var: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
W0710 19:00:00.834938 2053124 start.go:235] error getting percentage of /var that is free: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 19:00:00.834994 2053124 ssh_runner.go:195] Run: sh -c "df -BG /var | awk 'NR==2{print $4}'"
I0710 19:00:00.835043 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 19:00:00.866532 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 19:00:00.909911 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 19:00:00.910035 2053124 retry.go:31] will retry after 185.049647ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 19:00:01.095532 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 19:00:01.132918 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 19:00:01.182820 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 19:00:01.182935 2053124 retry.go:31] will retry after 205.33058ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 19:00:01.389582 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 19:00:01.424900 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 19:00:01.454432 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 19:00:01.454488 2053124 retry.go:31] will retry after 384.633774ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 19:00:01.840257 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 19:00:01.875857 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 19:00:01.919236 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
I0710 19:00:01.919391 2053124 retry.go:31] will retry after 839.554061ms: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 19:00:02.759552 2053124 cli_runner.go:164] Run: podman version --format {{.Version}}
I0710 19:00:02.793003 2053124 cli_runner.go:164] Run: podman container inspect -f "'{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}'" minikube
I0710 19:00:02.821354 2053124 sshutil.go:53] new ssh client: &{IP:127.0.0.1 Port:39251 SSHKeyPath:/home/foo/.minikube/machines/minikube/id_rsa Username:docker}
W0710 19:00:02.821477 2053124 start.go:283] error running df -BG /var: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
W0710 19:00:02.821488 2053124 start.go:240] error getting GiB of /var that is available: NewSession: new client: new client: Error creating new native config from ssh using: docker, &{[] [/home/foo/.minikube/machines/minikube/id_rsa]}: open /home/foo/.minikube/machines/minikube/id_rsa: no such file or directory
I0710 19:00:02.821494 2053124 fix.go:56] duration metric: took 5.13981992s for fixHost
I0710 19:00:02.821501 2053124 start.go:83] releasing machines lock for "minikube", held for 5.139858163s
W0710 19:00:02.821564 2053124 out.go:239] ๐Ÿ˜ฟ  Failed to start podman container. Running "minikube delete" may fix it: driver start: start: podman start minikube: exit status 125
stdout:

stderr:
Error: unable to start container "860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5": container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found \0, error found in #0 byte of ...||..., bigger context ...||...

I0710 19:00:02.831333 2053124 out.go:177]
W0710 19:00:02.832600 2053124 out.go:239] โŒ  Exiting due to GUEST_PROVISION: error provisioning guest: Failed to start host: driver start: start: podman start minikube: exit status 125
stdout:

stderr:
Error: unable to start container "860174a13a4a1e292a051863dcc7181bd9323d57b3a1d0d382e24b3e4c430ef5": container create failed (no logs from conmon): conmon bytes "": readObjectStart: expect { or n, but found \0, error found in #0 byte of ...||..., bigger context ...||...

W0710 19:00:02.832626 2053124 out.go:239]
W0710 19:00:02.834155 2053124 out.go:239] ๏ฟฝ[31mโ•ญโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฎ๏ฟฝ[0m
๏ฟฝ[31mโ”‚๏ฟฝ[0m                                                                                           ๏ฟฝ[31mโ”‚๏ฟฝ[0m
๏ฟฝ[31mโ”‚๏ฟฝ[0m    ๐Ÿ˜ฟ  If the above advice does not help, please let us know:                             ๏ฟฝ[31mโ”‚๏ฟฝ[0m
๏ฟฝ[31mโ”‚๏ฟฝ[0m    ๐Ÿ‘‰  https://github.com/kubernetes/minikube/issues/new/choose                           ๏ฟฝ[31mโ”‚๏ฟฝ[0m
๏ฟฝ[31mโ”‚๏ฟฝ[0m                                                                                           ๏ฟฝ[31mโ”‚๏ฟฝ[0m
๏ฟฝ[31mโ”‚๏ฟฝ[0m    Please run `minikube logs --file=logs.txt` and attach logs.txt to the GitHub issue.    ๏ฟฝ[31mโ”‚๏ฟฝ[0m
๏ฟฝ[31mโ”‚๏ฟฝ[0m                                                                                           ๏ฟฝ[31mโ”‚๏ฟฝ[0m
๏ฟฝ[31mโ•ฐโ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ”€โ•ฏ๏ฟฝ[0m
I0710 19:00:02.835534 2053124 out.go:177]

@Business-Wizard
Copy link

Business-Wizard commented Jul 30, 2024

Got it working. Thanks @osiler! I'm usually keeping deep notes whenever I'm stuck on a critical problem. I'll share here what worked and what not. There's the TLDR and the long read. Hopefully both help you in the future.

TLDR2,
Allowed me to skip reinstalling or convoluted bughunts:
Details: arm64 macos, nix-shell env

minikube delete
minikube start --driver podman --container-runtime containerd

@k8s-triage-robot
Copy link

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 90d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
co/podman-driver podman driver issues kind/support Categorizes issue or PR as a support question. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale.
Projects
None yet
Development

No branches or pull requests