Inter host container communication is not working for Publish ports with Flannel VXLAN in Docker V28.0

[pavanasamr]

Description

When using Flannel VXLAN for overlay network, for ingress traffic interface is flannel.1 not docker0.

Docker V28.0 creates below rule for published port, which is blocking the traffic since the ingress traffic interface is flannel.1 for Flannel VXLAN overlay network.

# Warning: iptables-legacy tables present, use iptables-legacy to see them
   33  1980 DROP       tcp  --  !docker0 *       0.0.0.0/0            10.1.215.5           tcp dpt:2181
VM1#

VM2# nc -zv 10.1.215.5 2181
nc: connect to 10.1.215.5 port 2181 (tcp) failed: Connection timed out

if we change the interface from docker0 to flannel.1, then Inter host container communication is working fine.

VM1# iptables -L -t raw -v -n | grep 2181
# Warning: iptables-legacy tables present, use iptables-legacy to see them
    0     0 DROP       tcp  --  !flannel.1 *       0.0.0.0/0            10.1.215.5           tcp dpt:2181
VM1#

VM2# nc -zv 10.1.215.5 2181
Connection to 10.1.215.5 2181 port [tcp/*] succeeded!
VM2#

Below is the complete details:

VM1# docker -v
Docker version 28.0.4, build b8034c0

VM1# ./flanneld.bin -version
v0.24.2

VM1# docker run -d \
  --name zookeeper \
  -p 2181:2181 \
  zookeeper:latest

VM1# docker exec -it b4e41c31b0a3 bash
root@b4e41c31b0a3:/apache-zookeeper-3.9.3-bin# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:37105           0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:2181            0.0.0.0:*               LISTEN      -
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      -
root@b4e41c31b0a3:/apache-zookeeper-3.9.3-bin# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1410
        inet 10.1.215.5  netmask 255.255.255.0  broadcast 10.1.215.255
        ether 3a:6d:76:38:ed:64  txqueuelen 0  (Ethernet)
        RX packets 3157  bytes 39176309 (39.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2776  bytes 189334 (189.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


VM1# iptables -t raw -L -v -n |grep 2181
# Warning: iptables-legacy tables present, use iptables-legacy to see them
33  1980 DROP       tcp  --  !docker0 *       0.0.0.0/0            10.1.215.5           tcp dpt:2181
VM1#


VM2# nc -zv 10.1.215.5 2181
nc: connect to 10.1.215.5 port 2181 (tcp) failed: Connection timed out


VM1# iptables -t raw -L PREROUTING --line-numbers
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    DROP       tcp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  tcp dpt:8983
2    DROP       udp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  udp dpt:8983
3    DROP       tcp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  tcp dpt:17777
4    DROP       udp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  udp dpt:17777
5    DROP       tcp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  tcp dpt:18983
6    DROP       udp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  udp dpt:18983
7    DROP       tcp  --  anywhere             10.1.215.5           tcp dpt:2181
VM1# iptables -t raw -D PREROUTING 7
VM1# iptables -t raw -L PREROUTING --line-numbers
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain PREROUTING (policy ACCEPT)
num  target     prot opt source               destination
1    DROP       tcp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  tcp dpt:8983
2    DROP       udp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  udp dpt:8983
3    DROP       tcp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  tcp dpt:17777
4    DROP       udp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  udp dpt:17777
5    DROP       tcp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  tcp dpt:18983
6    DROP       udp  --  anywhere             search-vurnl-solrcloud-solrcloud.service.default.devfarm.mu.cobalt.ariba.com  udp dpt:18983
VM1#

 
VM1# sudo iptables -t raw -I PREROUTING ! -i flannel.1 -d 10.1.215.5 -p tcp --dport 2181 -j DROP
VM1# iptables -L -t raw -v -n | grep 2181
# Warning: iptables-legacy tables present, use iptables-legacy to see them
    0     0 DROP       tcp  --  !flannel.1 *       0.0.0.0/0            10.1.215.5           tcp dpt:2181
VM1#

VM2# nc -zv 10.1.215.5 2181
Connection to 10.1.215.5 2181 port [tcp/*] succeeded!
VM2#


VM1# grep -ir vxlan *
grep: flanneld.bin: binary file matches
flannel-service.sh:local/cobalt/etcdctl.bin put /coreos.com/network/config "{ \"Network\": \"${CONTAINER_SUBNET}\", \"Backend\": {\"Type\": \"vxlan\"} }"

VM1# cat /etc/docker/daemon.json
{
  "dns": ["10.178.186.45"],
  "dns-opts": ["ndots:3"],
  "bip": "10.1.215.1/24",
  "selinux-enabled": true,
  "mtu": 1410,
  "ip-masq": false
}

VM1# ifconfig
docker0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1410
        inet 10.1.215.1  netmask 255.255.255.0  broadcast 10.1.215.255
        ether 9e:0f:5e:01:1e:20  txqueuelen 0  (Ethernet)
        RX packets 5391236  bytes 1859152760 (1.8 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3921247  bytes 3539225416 (3.5 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens4: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1460
        inet 10.178.186.45  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 42:01:0a:b2:ba:2d  txqueuelen 1000  (Ethernet)
        RX packets 86457869  bytes 27792601309 (27.7 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 64012273  bytes 204858975727 (204.8 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1410
        inet 10.1.215.0  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 02:a6:fd:1f:3e:9c  txqueuelen 0  (Ethernet)
        RX packets 211250  bytes 1721911041 (1.7 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 242377  bytes 36963774 (36.9 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

VM1# netstat -nr
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.178.184.1    0.0.0.0         UG        0 0          0 ens4
10.1.1.0        10.1.1.0        255.255.255.0   UG        0 0          0 flannel.1
10.1.2.0        10.1.2.0        255.255.255.0   UG        0 0          0 flannel.1
10.1.3.0        10.1.3.0        255.255.255.0   UG        0 0          0 flannel.1
10.1.5.0        10.1.5.0        255.255.255.0   UG        0 0          0 flannel.1
10.1.7.0        10.1.7.0        255.255.255.0   UG        0 0          0 flannel.1
10.1.9.0        10.1.9.0        255.255.255.0   UG        0 0          0 flannel.1
10.1.13.0       10.1.13.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.14.0       10.1.14.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.17.0       10.1.17.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.18.0       10.1.18.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.22.0       10.1.22.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.24.0       10.1.24.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.26.0       10.1.26.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.27.0       10.1.27.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.30.0       10.1.30.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.32.0       10.1.32.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.35.0       10.1.35.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.37.0       10.1.37.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.39.0       10.1.39.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.40.0       10.1.40.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.46.0       10.1.46.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.47.0       10.1.47.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.49.0       10.1.49.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.52.0       10.1.52.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.53.0       10.1.53.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.57.0       10.1.57.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.60.0       10.1.60.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.63.0       10.1.63.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.66.0       10.1.66.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.70.0       10.1.70.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.76.0       10.1.76.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.79.0       10.1.79.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.86.0       10.1.86.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.90.0       10.1.90.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.91.0       10.1.91.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.93.0       10.1.93.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.96.0       10.1.96.0       255.255.255.0   UG        0 0          0 flannel.1
10.1.102.0      10.1.102.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.103.0      10.1.103.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.104.0      10.1.104.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.105.0      10.1.105.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.110.0      10.1.110.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.114.0      10.1.114.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.122.0      10.1.122.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.123.0      10.1.123.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.126.0      10.1.126.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.128.0      10.1.128.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.131.0      10.1.131.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.132.0      10.1.132.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.136.0      10.1.136.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.137.0      10.1.137.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.138.0      10.1.138.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.140.0      10.1.140.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.141.0      10.1.141.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.144.0      10.1.144.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.146.0      10.1.146.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.147.0      10.1.147.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.151.0      10.1.151.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.152.0      10.1.152.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.155.0      10.1.155.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.158.0      10.1.158.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.164.0      10.1.164.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.168.0      10.1.168.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.170.0      10.1.170.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.171.0      10.1.171.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.180.0      10.1.180.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.181.0      10.1.181.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.184.0      10.1.184.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.189.0      10.1.189.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.190.0      10.1.190.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.192.0      10.1.192.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.193.0      10.1.193.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.194.0      10.1.194.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.198.0      10.1.198.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.199.0      10.1.199.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.200.0      10.1.200.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.203.0      10.1.203.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.204.0      10.1.204.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.208.0      10.1.208.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.211.0      10.1.211.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.215.0      0.0.0.0         255.255.255.0   U         0 0          0 docker0
10.1.216.0      10.1.216.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.217.0      10.1.217.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.220.0      10.1.220.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.226.0      10.1.226.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.231.0      10.1.231.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.232.0      10.1.232.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.233.0      10.1.233.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.240.0      10.1.240.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.246.0      10.1.246.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.249.0      10.1.249.0      255.255.255.0   UG        0 0          0 flannel.1
10.1.250.0      10.1.250.0      255.255.255.0   UG        0 0          0 flannel.1
10.178.184.1    0.0.0.0         255.255.255.255 UH        0 0          0 ens4
169.254.169.254 10.178.184.1    255.255.255.255 UGH       0 0          0 ens4

Reproduce

1. install Flannel vxlan

2. docker run zookeeper with published port

  --name zookeeper \
  -p 2181:2181 \
  zookeeper:latest```

3. docker v28 creates below iptable rule.
```VM1# iptables -t raw -L -v -n |grep 2181
# Warning: iptables-legacy tables present, use iptables-legacy to see them
33  1980 DROP       tcp  --  !docker0 *       0.0.0.0/0            10.1.215.5           tcp dpt:2181
VM1#```

4. login to different host & test zookeeper container ip on port 2181 reachability, it will fail.
```VM2# nc -zv 10.1.215.5 2181
nc: connect to 10.1.215.5 port 2181 (tcp) failed: Connection timed out

Expected behavior

No response

docker version

# docker version
Client: Docker Engine - Community
 Version:           28.0.4
 API version:       1.48
 Go version:        go1.23.7
 Git commit:        b8034c0
 Built:             Tue Mar 25 15:07:11 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.0.4
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.7
  Git commit:       6430e49
  Built:            Tue Mar 25 15:07:11 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.27
  GitCommit:        05044ec0a9a75232cad458027ca83437aae3f4da
 runc:
  Version:          1.2.5
  GitCommit:        v1.2.5-0-g59923ef
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

# docker info
Client: Docker Engine - Community
 Version:    28.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.22.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.34.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 5
  Running: 5
  Paused: 0
  Stopped: 0
 Images: 5
 Server Version: 28.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.8.0-1025-gcp
 Operating System: Ubuntu 22.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 251.9GiB
 Name: search3-v2-devfarm-ch-jcr3
 ID: 2d4e91b6-f09e-4d40-bde4-695729d5ee3f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

No response

[robmry]

Hi @pavanasamr,

Thank you for letting us know, and including all the info.

That raw-PREROUTING rule was introduced in 28.0.0 by #49325 - and is intended to drop packets addressed directly to the container that originate outside the host ... the thinking being that most people didn't realise containers were directly accessible from outside their host machine, and we wanted to improve the default behaviour.

So, the docker0 in the rule is intentional. Normally, traffic originating on the host for the container's published port will come from that interface so, it's allowed, and other traffic is dropped.

But, that direct access is what traffic arriving on your vxlan's flannel.1 interface needs. So ...

It's possible to disable firewalling for a user-defined network by setting its "gateway mode" to "nat-unprotected" ... but the default bridge network (docker0) doesn't have that option.

As a workaround - you can set environment variable DOCKER_INSECURE_NO_IPTABLES_RAW=1 before starting dockerd. That'll stop it from creating the new rule - there's an example of a systemd config file for doing that in the PR description at #49621 - along with the caveats. (I think you'll need to manually delete the rule if it's already there.)

But, we plan to remove that workaround in the medium term.

To create a user-defined network that doesn't block access to unpublished ports, it's ...
docker network create -o com.docker.network.bridge.gateway_mode_ipv4=nat-unprotected <other options> mynet (and the equivalent for IPv6 if needed). Then, you'd add --network mynet to the docker run command for zookeeper. However - that will open up access to all ports on your container, not just published ones ... which probably isn't what you want.

We'll need to give it some more thought (cc @akerouanton).

(Just to note, docker0's subnet is 10.0.215.0/24 and flannel.1 has an address in that subnet, 10.1.215.0/32 - the network address, but with a "/32", which is a bit surprising. I wonder if we can use that somehow, perhaps not without allowing-in packets with a spoofed source address, but at least those wouldn't get replies. Or, perhaps better, we generalise an exception like the one for wsl2's loopback0, making it an interface allow-list.)

[robmry]

The best plan seems to be to add a way to list a set of "trusted interfaces" in daemon config (/etc/docker/daemon.json). Access to published ports on a container's own IP address would be allowed from these interfaces. Probably with a wildcard value that says allow access from any interface.

(@pavanasamr - does that sound like it'd work for you? Does the env-var help in the meantime?)

[pavanasamr]

@robmry, Using the environment variable DOCKER_INSECURE_NO_IPTABLES_RAW=1, its working. After setting this variable, we dont see iptables rules for RAW table is not getting added by Docker daemon.