data not restored after restart

[iompo]

What Happened?

Hi all, I have a MacOS 12.2.1, Apple M1 Pro. Docker 20.10.12 and minikube 1.25.1.
I create a new cluster with the default settings, add a namespace and then stop the cluster. When I restart it the namespace is lost. Same thing happens for any kind of resource I try to create.
I've seen other issues similar to this, but they were all closed. Docker is the only option for arm, so I'm not sure what to try.

Attach the log file

minikube-restart-error.txt

Operating System

MacOS

Driver

Docker

[marwatk]

We're seeing the same behavior using --driver=none on Centos 7 and Docker 20.10.12.

Here's the errant behavior (1.25.1 and 1.25.0 both have this behavior):

[mwatkins@pd-ci ~]$ minikube-1.25.1 start --driver=none
😄  minikube v1.25.1 on Centos 7.9.2009
✨  Using the none driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=4, Memory=64247MB, Disk=14326MB) ...
ℹ️  OS release is CentOS Linux 7 (Core)
🐳  Preparing Kubernetes v1.23.1 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    > kubeadm.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubelet.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubectl.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubectl: 44.43 MiB / 44.43 MiB [------------] 100.00% 92.65 MiB p/s 700ms
    > kubeadm: 43.12 MiB / 43.12 MiB [-------------] 100.00% 54.64 MiB p/s 1.0s
    > kubelet: 118.75 MiB / 118.75 MiB [-----------] 100.00% 67.49 MiB p/s 2.0s
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🤹  Configuring local host environment ...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❗  kubectl and minikube configuration will be stored in /home/mwatkins
❗  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /home/mwatkins/.kube /home/mwatkins/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
[mwatkins@pd-ci ~]$ minikube-1.25.1 kubectl create namespace test
namespace/test created
[mwatkins@pd-ci ~]$ minikube-1.25.1 kubectl get namespace/test
NAME   STATUS   AGE
test   Active   11s
[mwatkins@pd-ci ~]$ minikube-1.25.1 stop
✋  Stopping node "minikube"  ...
🛑  1 node stopped.
[mwatkins@pd-ci ~]$ minikube-1.25.1 start
😄  minikube v1.25.1 on Centos 7.9.2009
✨  Using the none driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing none bare metal machine for "minikube" ...
ℹ️  OS release is CentOS Linux 7 (Core)
🐳  Preparing Kubernetes v1.23.1 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🤹  Configuring local host environment ...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❗  kubectl and minikube configuration will be stored in /home/mwatkins
❗  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /home/mwatkins/.kube /home/mwatkins/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
[mwatkins@pd-ci ~]$ minikube-1.25.1 kubectl get namespace/test
Error from server (NotFound): namespaces "test" not found

And here's the correct behavior (1.24.0):

[mwatkins@pd-ci ~]$ minikube-1.24.0 start --driver=none
😄  minikube v1.24.0 on Centos 7.9.2009
✨  Using the none driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=4, Memory=64247MB, Disk=14326MB) ...
ℹ️  OS release is CentOS Linux 7 (Core)
    > kubeadm.sha256: 64 B / 64 B [--------------------------] 100.00% ? p/s 0s
    > kubelet: 115.57 MiB / 115.57 MiB [-----------] 100.00% 50.92 MiB p/s 2.5s
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🤹  Configuring local host environment ...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❗  kubectl and minikube configuration will be stored in /home/mwatkins
❗  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /home/mwatkins/.kube /home/mwatkins/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
[mwatkins@pd-ci ~]$ minikube-1.24.0 kubectl create namespace test
namespace/test created
[mwatkins@pd-ci ~]$ minikube-1.24.0 kubectl get namespace/test
NAME   STATUS   AGE
test   Active   9s
[mwatkins@pd-ci ~]$ minikube-1.24.0 stop
✋  Stopping node "minikube"  ...
🛑  1 node stopped.
[mwatkins@pd-ci ~]$ minikube-1.24.0 start
😄  minikube v1.24.0 on Centos 7.9.2009
✨  Using the none driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🔄  Restarting existing none bare metal machine for "minikube" ...
ℹ️  OS release is CentOS Linux 7 (Core)
🐳  Preparing Kubernetes v1.22.3 on Docker 20.10.12 ...
🤹  Configuring local host environment ...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❗  kubectl and minikube configuration will be stored in /home/mwatkins
❗  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /home/mwatkins/.kube /home/mwatkins/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
[mwatkins@pd-ci ~]$ minikube-1.24.0 kubectl get namespace/test
NAME   STATUS   AGE
test   Active   53s

[marwatk]

It also happens with --driver=docker, which has me wondering if I'm doing something fundamentally wrong because I would expect enough people to be hitting it that an issue would have been filed sooner than 28 days after the release.

[mwatkins@pd-ci ~]$ minikube-1.25.1 start --driver=docker
😄  minikube v1.25.1 on Centos 7.9.2009
✨  Using the docker driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
💾  Downloading Kubernetes v1.23.1 preload ...
    > gcr.io/k8s-minikube/kicbase: 378.97 MiB / 378.98 MiB  100.00% 58.25 MiB p
    > preloaded-images-k8s-v16-v1...: 504.42 MiB / 504.42 MiB  100.00% 76.54 Mi
🔥  Creating docker container (CPUs=2, Memory=16000MB) ...
❗  This container is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.23.1 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
[mwatkins@pd-ci ~]$ minikube-1.25.1 kubectl create namespace test
namespace/test created
[mwatkins@pd-ci ~]$ minikube-1.25.1 stop
✋  Stopping node "minikube"  ...
🛑  Powering off "minikube" via SSH ...
🛑  1 node stopped.
[mwatkins@pd-ci ~]$ minikube-1.25.1 start
😄  minikube v1.25.1 on Centos 7.9.2009
✨  Using the docker driver based on existing profile
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔄  Restarting existing docker container for "minikube" ...
❗  This container is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.23.1 on Docker 20.10.12 ...
    ▪ kubelet.housekeeping-interval=5m
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
[mwatkins@pd-ci ~]$ minikube-1.25.1 kubectl get namespace/test
Error from server (NotFound): namespaces "test" not found

[marwatk]

Nevermind, this is a duplicate of #13503 and was resolved here

[spowelljr]

Hi @iompo, thanks for reporting your issue with minikube!

As @marwatk pointed out this is fixed by #13506 but it currently isn't included with a release yet. However, you can download the binary from this comment that has the fix included #13503 (comment).

I'm going to close this issue, but feel free to respond if you have any further questions, thanks!