firewalld: dashboard CrashLoopBackoff: dial tcp 10.96.0.1:443: connect: no route to host

[ivuk]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
BUG REPORT

Environment:

minikube version: v0.33.1

OS:
NAME="Ubuntu"
VERSION="16.04.5 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.5 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

VM driver:
    "DriverName": "none",

What happened:
When running minikube addons enable dashboard, the dashboard pod ends in CrashLoopBackOff state.

What you expected to happen:
The kubernetes-dashboard pod should be in Running state, with proper iptables rules added.

How to reproduce it (as minimally and precisely as possible):
Install the firewalld daemon package on Ubuntu 16.04, ensure it's running (systemctl start firewalld), attempt to initialize a minikube environment by running minikube start --vm-driver none. Enable the dashboard with minikube addons enable dashboard. Observe the kubernetes-dashboard-* pod crashing.

Anything else do we need to know:
The logs for the dashboard container show:

2019/01/21 15:47:17 Starting overwatch
2019/01/21 15:47:17 Using in-cluster config to connect to apiserver
2019/01/21 15:47:17 Using service account token for csrf signing
2019/01/21 15:47:18 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service account's configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: connect: no route to host
Refer to our FAQ and wiki pages for more information: https://github.com/kubernetes/dashboard/wiki/FAQ

The iptables rules contain:

-A KUBE-SERVICES -d 10.108.32.98/32 -p tcp -m comment --comment "kube-system/kubernetes-dashboard: has no endpoints" -m tcp --dport 80 -j REJECT --reject-with icmp-port-unreachable

[tstromberg]

I'm not super familiar with Kubernetes and firewalls, but here is what I understand so far:

• You've installed a firewall (firewalld)
• There appears to be an unrelated rule for packets to 10.108.32.98:80 - did you define this rule?
• You expect there to be an auto-generated iptables rule that allows packets to 10.96.0.1:443?
• Dashboard crashes because it can't send packets to 10.96.0.1:443

I suspect that kubeadm/k8s doesn't configure firewalld in this environment. Is your request then to add some firewalld integration support to minikube?

[ivuk]

Hi @tstromberg, thank you for the feedback!

With the "default" iptables state on Ubuntu 16.04 (with just Docker running, no additional rules aside from those that Docker adds by default), these are the rules that are added regarding the dashboard (when enabled):

root@ubuntu-xenial:~# iptables-save | grep KUBE-SVC-XGLOHA7QRQ3V22RZ
:KUBE-SVC-XGLOHA7QRQ3V22RZ - [0:0]
-A KUBE-SERVICES -d 10.104.15.165/32 -p tcp -m comment --comment "kube-system/kubernetes-dashboard: cluster IP" -m tcp --dport 80 -j KUBE-SVC-XGLOHA7QRQ3V22RZ
-A KUBE-SVC-XGLOHA7QRQ3V22RZ -j KUBE-SEP-F3JA2MOXYYHCYKC5

With firewalld running, there's only the rule pasted in the original comment. I think something (kubeadm/k8s/docker) falls over with firewalld running, and fails to add all the iptables rules that are needed.

I'm not adding any iptables rules manually, that's just what gets created automatically in the process of creating/starting pods.

Please let me know if there's any additional information you need. :)

[dash042]

I'm having the same issue on CentOS 7.5

[princepaulson1]

I'm facing the same issue as well on CentOS Linux release 7.5.1804 (Core)

[devguo]

I have the same issue too on centos 7.6. When stop firewalld, minikube can work and the issue resolved. Can anybody tell me how to setting the firewalld so I don't need to stop firewalld?

[tstromberg]

I'm closing this issue as it hasn't seen activity in awhile, and it's unclear if this issue still exists. If this issue does continue to exist in the most recent release of minikube, please feel free to re-open it.

Thank you for opening the issue!

FYI - for others discussing firewall rules: https://github.com/kubernetes/minikube/blob/master/docs/networking.md#firewalls-vpns-and-proxies now has some documentation for how to configure them. This issue appears to be due to automatic firewall rules being added by Kubernetes, however.

[tester-rep]

mark