Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cenots 7: minikube v1.33.1 faces coredns CrashLoopBackOff. #18950

Closed
charleech opened this issue May 23, 2024 · 13 comments
Closed

cenots 7: minikube v1.33.1 faces coredns CrashLoopBackOff. #18950

charleech opened this issue May 23, 2024 · 13 comments
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@charleech
Copy link

What Happened?

minikube on CentOS 7.7 does not work.

Software information

$ minikube version
minikube version: v1.33.1
commit: 5883c09216182566a63dff4c326a6fc9ed2982ff

$ docker version
Client: Docker Engine - Community
 Version:           26.1.2
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        211e74b
 Built:             Wed May  8 14:01:02 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.2
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       ef1912d
  Built:            Wed May  8 13:59:55 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Steps

$ export MINIKUBE_HOME=/tmp/minikube_home

$ minikube start \
   --driver=docker \
   --network minikube

😄  minikube v1.33.1 on Centos 7.7.1908
    ▪ MINIKUBE_IN_STYLE=true
    ▪ MINIKUBE_HOME=/tmp/minikube_home
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.44 ...
🔥  Creating docker container (CPUs=2, Memory=8000MB) ...
❗  This container is having trouble accessing https://registry.k8s.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring Calico (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

The coredns status is CrashLoopBackOff

$  kubectl get pod -A -o wide

NAMESPACE     NAME                               READY   STATUS             RESTARTS     AGE   IP             NODE       NOMINATED NODE   READINESS GATES
kube-system   coredns-7db6d8ff4d-mp9pd           0/1     CrashLoopBackOff   2 (3s ago)   26s   10.244.0.2     minikube   <none>           <none>
kube-system   etcd-minikube                      1/1     Running            0            39s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-apiserver-minikube            1/1     Running            0            39s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-controller-manager-minikube   1/1     Running            0            39s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-proxy-cknfg                   1/1     Running            0            26s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-scheduler-minikube            1/1     Running            0            39s   192.188.49.2   minikube   <none>           <none>
kube-system   storage-provisioner                1/1     Running            0            38s   192.188.49.2   minikube   <none>           <none>

describe pod

Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  17m                   default-scheduler  Successfully assigned kube-system/coredns-7db6d8ff4d-mp9pd to minikube
  Normal   Created    16m (x4 over 17m)     kubelet            Created container coredns
  Normal   Started    16m (x4 over 17m)     kubelet            Started container coredns
  Normal   Pulled     16m (x5 over 17m)     kubelet            Container image "registry.k8s.io/coredns/coredns:v1.11.1" already present on machine
  Warning  BackOff    2m21s (x82 over 17m)  kubelet            Back-off restarting failed container coredns in pod coredns-7db6d8ff4d-mp9pd_kube-system(fbae1fb2-d70f-4683-ae4b-2b11dd0e7d14)

Try to print the log

$ kubectl -n kube-system logs -f coredns-7db6d8ff4d-mp9pd
Listen: listen tcp :53: bind: permission denied

Attach the log file

log.txt

Operating System

Redhat/Fedora

Driver

Docker
@charleech
Copy link
Author

charleech commented May 23, 2024 •
edited
Loading

I can confirm that after rolling back to minikube version 1.32.0, this issue does not occur.

$ minikube version
minikube version: v1.32.0
commit: 8220a6eb95f0a4d75f7f2d7b14cef975f050512d

$ docker version
Client: Docker Engine - Community
 Version:           26.1.2
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        211e74b
 Built:             Wed May  8 14:01:02 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.2
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       ef1912d
  Built:            Wed May  8 13:59:55 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ export MINIKUBE_HOME=/tmp/minikube_home

$ minikube start \
   --driver=docker \
   --network minikube

😄  minikube v1.32.0 on Centos 7.7.1908
    ▪ MINIKUBE_IN_STYLE=true
    ▪ MINIKUBE_HOME=/tmp/minikube_home
✨  Using the docker driver based on user configuration
📌  Using Docker driver with root privileges
👍  Starting control plane node minikube in cluster minikube
🚜  Pulling base image ...
🔥  Creating docker container (CPUs=2, Memory=8000MB) ...
❗  This container is having trouble accessing https://registry.k8s.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
🐳  Preparing Kubernetes v1.28.3 on Docker 24.0.7 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎  Verifying Kubernetes components...
🌟  Enabled addons: default-storageclass, storage-provisioner

❗  /usr/local/bin/kubectl is version 1.30.0, which may have incompatibilities with Kubernetes 1.28.3.
    ▪ Want kubectl v1.28.3? Try 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$  kubectl get pods -A -o wide
NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE   IP             NODE       NOMINATED NODE   READINESS GATES
kube-system   coredns-5dd5756b68-lt5m5           1/1     Running   0          15s   10.244.0.2     minikube   <none>           <none>
kube-system   etcd-minikube                      1/1     Running   0          27s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-apiserver-minikube            1/1     Running   0          27s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-controller-manager-minikube   1/1     Running   0          27s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-proxy-b8dn2                   1/1     Running   0          15s   192.188.49.2   minikube   <none>           <none>
kube-system   kube-scheduler-minikube            1/1     Running   0          27s   192.188.49.2   minikube   <none>           <none>
kube-system   storage-provisioner                1/1     Running   0          26s   192.188.49.2   minikube   <none>           <none>

log.txt

@Dranaxel
Copy link

Not totally sure it's related but I'm trying to install minikube with cilium in the same version and got a coredns error too:

╰─$ minikube version
minikube version: v1.33.0
commit: 86fc9d54fca63f295d8737c8eacdbb7987e89c67

╰─$ docker version                                                                                                                 127 ↵
Client: Docker Engine - Community
 Version:           26.1.3
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        b72abbb
 Built:             Thu May 16 08:33:48 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:48 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.32
  GitCommit:        8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I think cilium is crashing bc of the coredns errors

╰─$ k get pods -A
NAMESPACE     NAME                               READY   STATUS               RESTARTS         AGE
kube-system   cilium-l9prq                       0/1     CrashLoopBackOff     11 (76s ago)     39m
kube-system   cilium-mhl56                       0/1     CrashLoopBackOff     10 (3m14s ago)   32m
kube-system   cilium-nmrfz                       0/1     PostStartHookError   11 (8s ago)      36m
kube-system   cilium-operator-86f4c5579c-q6s75   1/1     Running              5 (31m ago)      39m
kube-system   coredns-7db6d8ff4d-wt9s6           0/1     CrashLoopBackOff     17 (9s ago)      39m
kube-system   etcd-minikube                      1/1     Running              0                40m
kube-system   kube-apiserver-minikube            1/1     Running              1 (38m ago)      40m
kube-system   kube-controller-manager-minikube   1/1     Running              0                40m
kube-system   kube-proxy-ffnbj                   1/1     Running              0                36m
kube-system   kube-proxy-jpr6w                   1/1     Running              0                32m
kube-system   kube-proxy-kk54f                   1/1     Running              0                39m
kube-system   kube-scheduler-minikube            1/1     Running              0                40m
kube-system   storage-provisioner                1/1     Running              6 (30m ago)      40m

@Dranaxel
Copy link

Yes, downgrading to 1.32.0 fixed the issue

@HuXinjing
Copy link

Same question.

@medyagh
Copy link
Member

medyagh commented Jun 4, 2024

HuXinjing are you using Centos 7 as well ? I am suspecting this is a centos only issue, can everybody confirm they are all on centos

@HuXinjing
Copy link

HuXinjing commented Jun 4, 2024 via email

@medyagh medyagh added kind/bug Categorizes issue or PR as related to a bug. help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Jun 4, 2024
@medyagh medyagh changed the title minikube v1.33.1 faces coredns CrashLoopBackOff. cenots 7: minikube v1.33.1 faces coredns CrashLoopBackOff. Jun 4, 2024
@medyagh medyagh mentioned this issue Jun 4, 2024
Closed
@prezha
Copy link
Contributor

prezha commented Jun 4, 2024 •
edited
Loading

hi @charleech, i think that the issue is not minikube v1.33.1 vs v1.32.0 but it's about the centos 7 and the default kubernetes version that comes with the respective minikube version

i think you'd be able to confirm that with the (latest) minikube v1.33.1 whilst specifying the older (ie, <v1.29.0) kubernetes version (eg, k8s v1.28.3 that's used by minikube v1.32.0 - as shown in your comment above as well):

minikube start --container-runtime=docker --kubernetes-version=1.28.3 -p k8s-1.28.3

i expect this to work for @Dranaxel and @HuXinjing too

more details about this issue is in the comment of #18421

can you please run this and let us know if it works for you?

@prezha
Copy link
Contributor

prezha commented Jun 4, 2024 •
edited
Loading

btw, i was able to replicate the issue with the latest minikube (v1.33.1) running on centos 7 (vm), but on the other hand, it works fine on centos (stream) 8 (vm), so i think it's specifically related to the centos 7 that's going end-of-life by the end of this months (june 2024) and therefore probably will not get the necessary fixes backported (eg, setting net.ipv4.ip_unprivileged_port_start)

$ minikube start
😄  minikube v1.33.1 on Centos 8 (kvm/amd64)
✨  Automatically selected the docker driver
📌  Using Docker driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.44 ...
💾  Downloading Kubernetes v1.30.0 preload ...
    > preloaded-images-k8s-v18-v1...:  342.90 MiB / 342.90 MiB  100.00% 23.57 M
    > gcr.io/k8s-minikube/kicbase...:  481.58 MiB / 481.58 MiB  100.00% 16.31 M
🔥  Creating docker container (CPUs=2, Memory=2200MB) ...
🐳  Preparing Kubernetes v1.30.0 on Docker 26.1.1 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring bridge CNI (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
💡  kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

$ minikube kubectl -- get pods -A -owide
    > kubectl.sha256:  64 B / 64 B [-------------------------] 100.00% ? p/s 0s
    > kubectl:  49.07 MiB / 49.07 MiB [--------------] 100.00% 1.73 MiB p/s 29s
NAMESPACE     NAME                               READY   STATUS    RESTARTS      AGE   IP             NODE       NOMINATED NODE   READINESS GATES
kube-system   coredns-7db6d8ff4d-j62tq           1/1     Running   0             81s   10.244.0.2     minikube   <none>           <none>
kube-system   etcd-minikube                      1/1     Running   0             97s   192.168.49.2   minikube   <none>           <none>
kube-system   kube-apiserver-minikube            1/1     Running   0             97s   192.168.49.2   minikube   <none>           <none>
kube-system   kube-controller-manager-minikube   1/1     Running   0             96s   192.168.49.2   minikube   <none>           <none>
kube-system   kube-proxy-9d4dg                   1/1     Running   0             82s   192.168.49.2   minikube   <none>           <none>
kube-system   kube-scheduler-minikube            1/1     Running   0             95s   192.168.49.2   minikube   <none>           <none>
kube-system   storage-provisioner                1/1     Running   1 (50s ago)   94s   192.168.49.2   minikube   <none>           <none>

@charleech
Copy link
Author

@prezha

Thank you for your advice.

I'm requesting my engineer to set up a new VM with the latest version of Oracle Linux 9.x. Once it's ready, I'll test it and update you on the results. This process should be completed by Friday.

@charleech
Copy link
Author

charleech commented Jun 7, 2024 •
edited
Loading

I can confirm that after upgrading the OS to newer/latest version (for me it is an Oracle Linux 9.4), the coredns is started properly.

@prezha
Copy link
Contributor

prezha commented Jun 8, 2024

excellent, thanks for checking and confirming @charleech !

i'm going to close this issue as it's related to centos 7 that's going end-of-life in a couple of weeks

@prezha prezha closed this as completed Jun 8, 2024
@raphaelpoumarede raphaelpoumarede mentioned this issue Jun 14, 2024
Closed
@PrasadP744
Copy link

hi @charleech, i think that the issue is not minikube v1.33.1 vs v1.32.0 but it's about the centos 7 and the default kubernetes version that comes with the respective minikube version

i think you'd be able to confirm that with the (latest) minikube v1.33.1 whilst specifying the older (ie, <v1.29.0) kubernetes version (eg, k8s v1.28.3 that's used by minikube v1.32.0 - as shown in your comment above as well):

minikube start --container-runtime=docker --kubernetes-version=1.28.3 -p k8s-1.28.3

i expect this to work for @Dranaxel and @HuXinjing too

more details about this issue is in the comment of #18421

can you please run this and let us know if it works for you?

This worked with kvm virtualization
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.10.0-1160.114.2.el7.x86_64
Architecture: x86-64

  Thanks @prezha

@cdxiaodong
Copy link

I have encountered this problem too. Thank you for your solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
help wanted Denotes an issue that needs help from a contributor. Must meet "help wanted" guidelines. kind/bug Categorizes issue or PR as related to a bug. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@charleech @medyagh @prezha @Dranaxel @HuXinjing @cdxiaodong @PrasadP744