Minikube to host communication not working on Fedora 37

[mnk]

What Happened?

There seems to be a difference in minikube iptables rules when comparing a fully updated Fedora 36 and Fedora 37 system.
On Fedora 36:

$ sudo iptables -t nat -S|grep -e '--to-destination 127.0.0.11'
-A DOCKER_OUTPUT -d 192.168.49.1/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 127.0.0.11:39397
-A DOCKER_OUTPUT -d 192.168.49.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:34196

On Fedora 37:

$ sudo iptables -t nat -S|grep -e '--to-destination 127.0.0.11'
-A DOCKER_OUTPUT -d 192.168.49.1/32 -p tcp -j DNAT --to-destination 127.0.0.11:46739
-A DOCKER_OUTPUT -d 192.168.49.1/32 -p udp -j DNAT --to-destination 127.0.0.11:37392

The missing --dport 53 condition on the destination NAT breaks all non-DNS communication between host and minikube.
What might be causing this difference?

Attach the log file

log.txt

Operating System

Redhat/Fedora

Driver

Docker

[mnk]

Full output of minikube ssh sudo iptables-save:
iptables-save-f37.txt
iptables-save-f36.txt

[aojea]

hmm, do we know if was docker the one to drop the port?

I've seen a recent report in kubernetes about some weird iptables rules being mutated in Centos

[prezha]

@mnk thanks for reporting this issue

there are a couple of improvements i'm currently working on in draft pr #15463

could you please try with https://storage.googleapis.com/minikube-builds/15463/minikube-linux-amd64 and let us know if it works for you

if not, could you pull that pr and run:

make integration -e TEST_ARGS="-minikube-start-args='--driver=docker --container-runtime=docker --alsologtostderr -v=7' -test.run TestNetworkPlugins --cleanup=true"

then share the whole output you get

i've tried to replicate your setup (ie, fresh fedora 37 install [in kvm] + docker) and all the tests above passed for me, so i'm curious to know if it would work for you

[mnk]

@prezha , I tried the minikube build you linked to, but I still get the same result - no --dport 53 condition on the DNAT rule. Do I need to specify a base image or just do a minikube-linux-amd64 start?

Do you get the --dport 53 condition when testing with you branch?

My use-case can be tested by starting sshd on the host and then doing minikube ssh ssh $(id -un)@192.168.49.1. This works fine on Fedora 36 but not on Fedora 37.

[BenTheElder]

@aojea as mentioned on the PR I also see no dport currently, on gLinux (~debian):

$ docker run -d --entrypoint=sleep --network=kind --privileged --name=aaa kindest/node:v1.25.3 infinity
07cc1460e1bf62a936e33775efdda0fbce577634eb06b07dcdd267bd855f9248

$ docker exec --privileged aaa iptables-save
# Generated by iptables-save v1.8.7 on Wed Jan  4 22:07:38 2023
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER_OUTPUT - [0:0]
:DOCKER_POSTROUTING - [0:0]
-A OUTPUT -d 127.0.0.11/32 -j DOCKER_OUTPUT
-A POSTROUTING -d 127.0.0.11/32 -j DOCKER_POSTROUTING
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p tcp -j DNAT --to-destination 127.0.0.11:44029
-A DOCKER_OUTPUT -d 127.0.0.11/32 -p udp -j DNAT --to-destination 127.0.0.11:33690
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p tcp -j SNAT --to-source :53
-A DOCKER_POSTROUTING -s 127.0.0.11/32 -p udp -j SNAT --to-source :53
COMMIT

(NOTE: this skips the entrypoint logic etc, the point is to debug purely what docker is doing on its own with the embedded DNS rules)

[BenTheElder]

Docker might have actually dropped the dport themselves at some point, given that they probably don't expect other traffic on 127.0.0.11 typically, but in that case I'd argue this is a bug on their end and we should fix it there.

I did a bit of digging and haven't turned up anything though, and in #15578 (comment) it appears that docker, containerd are identical but iptables versions are different.

[prezha]

@mnk i was wrong - haven't read your initial problem statement carefully, so was jumping to a conclusion; sorry about that!

i've looked at it a bit and i think that the problem is in using iptables-nft (which i think is default for fedora37) instead of iptables-legacy (i also remember reading in one of the kubernetes issues recently that nft is not [yet] supported, but i don't have a reference at hand atm)

i suggest you try with iptables-legacy instead - here's what i did:

$ sudo dnf install iptables-legacy

$ sudo update-alternatives --config iptables => select '/usr/sbin/iptables-legacy'

$ iptables --version
iptables v1.8.8 (legacy)

$ sudo reboot

$ minikube start
...

$ minikube ssh -- sudo iptables-save
# Generated by iptables-save v1.8.4 on Fri Jan  6 22:37:48 2023
*nat
...
-A DOCKER_OUTPUT -d 192.168.49.1/32 -p tcp -m tcp --dport 53 -j DNAT --to-destination 127.0.0.11:41653
-A DOCKER_OUTPUT -d 192.168.49.1/32 -p udp -m udp --dport 53 -j DNAT --to-destination 127.0.0.11:53598
...

$ minikube ssh ssh $(id -un)@192.168.49.1
The authenticity of host '192.168.49.1 (192.168.49.1)' can't be established.
ECDSA key fingerprint is SHA256:Y8jQ23KJ8Oy+H2e9eXDpttirqcg42g7HVtg4ZFjVgHM.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.49.1' (ECDSA) to the list of known hosts.
prezha@192.168.49.1's password: 
Web console: https://localhost:9090/ or https://192.168.122.198:9090/

Last login: Fri Jan  6 22:35:54 2023 from 192.168.122.1

[prezha]

btw, i've also compared docker versions that are installed on a fresh ubuntu 20.04.5 (where iptables-legacy is default and this is working) and fedora 37 (where iptables-nft are default and apparently is not working), and they use identical versions/commits:

ubuntu 20.04.5:

prezha@minikube-test:~$ docker version
Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:08 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:58 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

fedora37:

[prezha@localhost ~]$ docker version
Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.18.9
 Git commit:        3a2c30b
 Built:             Thu Dec 15 22:28:45 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:26:25 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.14
  GitCommit:        9ba4b250366a5ddde94bb7c9d1def331423aa323
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[prezha]

looks like there were some efforts to support automatic iptables "mode" detection, but it seems that it's not working correctly in this case and this comment from Tim Hockin is a bit old but perhaps still relevant

[prezha]

looks like minikube user(s) reported identical problem earlier: #14631 (comment) whereas original problem refers to iptables-nft

so, if this switch to iptables-legacy is a working solution for @mnk, perhaps we add detection in minikube so if we see iptables-nft, we warn the user that "things might not work as expected"

[BenTheElder]

to kubernetes-sigs/iptables-wrappers#3, but it seems that it's not working correctly in this case and this comment from Tim Hockin is a bit old but perhaps still relevant

That's not quite related. That change in Kubernetes is related to automatic mode detection in the kube-proxy image, in a "normal" environment by looking for rules setup by kubelet on the host. In our case we instead detect based on docker injecting rules for the embedded DNS resolver using the host iptables legacy or nf_tables.

That tweet predates the detection logic entirely, which is itself a workaround for the problem of there not being a stable interface and distros switching between the two binaries / backends.

The upstream KIND entrypoint has the same detection logic as kube-proxy (prior to the trick looking at kubelet generated rules specifically).

My host really is using nf_tables 1.8.8 and I don't see --dport using either in the "node". So the issue is not mismatched nf_tables vs legacy, the problem is 1.8.7 vs 1.8.8 nf_tables.

And now that I've typed that, kubernetes/kubernetes#112477 is paging back into memory 🙃

$ minikube ssh -- sudo iptables-save
# Generated by iptables-save v1.8.4 on Fri Jan  6 22:37:48 2023

1.8.4 is really old, so that's a different problem for minikube specifically.

In both cases, 1.8.8 on the host is currently a problem. kube-proxy in Kubernetes 1.26 is also on 1.8.7 so updating to 1.8.8 in kind/minikube is probably not sufficient yet.

Downgrading to 1.8.7 on the host is one workaround. Switching to legacy mode is another. Both work around the mismatched versions rather than legacy vs nf_tables detection.

[BenTheElder]

This is also just one incompatibility between 1.8.8 and 1.8.7, we're going to have more problems when we do upgrade kubernetes/kind/... to > 1.8.7 (see above issue with --mark)