TCP rejection can't work on Kind when the traffic mode is noEncap

[GraysonWu]

Describe the problem/challenge you have

When exec E2E test, the test case TestAntreaPolicy/TestGroupNoK8sNP/Case=ACNPRejectIngress which is testing if Reject can work on TCP traffic always fails when enabling AntreaPolicy in noEncap mode. Some connections that should be rejected are observed as dropped. But it doesn't fail when testing it on the local vagrant testbed.

When manually test Reject on TCP traffic in a local Kind cluster in noEncap mode with AntreaPolicy enabling:

• For the intra-node case, everything is good.
• For the inter-node case, the requesting client will keep retrying and waiting until timed out with no reject response received.

According to the investigation, on Kind cluster, we use the OVS netdev datapath, which requires a bridge setup on each node. For the inter-node traffic case in noEncap mode, there will be two conntrack lookups for the same connection. When a TCP RST packet is sent out by the Reject action, the conntrack entry will be destroyed by the first lookup. Then the second lookup will tag the packet as INVALID and drop it. Since the response packet has been dropped by the Kernal, the Pod won't receive it. Thus the Pod will keep retrying and waiting until timed out with no reject response received.

So skip this test for now when the provider is Kind the traffic mode is noEncap.

Thanks for @antoninbas 's help during the whole process.

Reference: https://ask.openstack.org/en/question/28300/iptables-invalid-rule-preventing-rst-packets-on-closed-ports-between-vms/

[antoninbas]

To be clear, it's a similar situation as https://ask.openstack.org/en/question/28300/iptables-invalid-rule-preventing-rst-packets-on-closed-ports-between-vms/, but not quite the same.

The TCP RST packet does go twice through conntrack for some reason, which seems related to the fact that we use an extra netdev bridge (br-phy) attached to a physical interface (eth0). I have confirmed this by adding the following iptables rule: iptables -t raw -A PREROUTING -p tcp --tcp-flags RST RST -j ACCEPT

# check counters
Chain PREROUTING (policy ACCEPT 87 packets, 35539 bytes)
 pkts bytes target     prot opt in     out     source               destination
 403K  367M ANTREA-PREROUTING  all  --  any    any     anywhere             anywhere             /* Antrea: jump to Antrea prerouting rules */
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp flags:RST/RST
# try a TCP connection to a Pod on another Node, which is rejected
# check counters again
Chain PREROUTING (policy ACCEPT 232 packets, 79185 bytes)
 pkts bytes target     prot opt in     out     source               destination
 403K  367M ANTREA-PREROUTING  all  --  any    any     anywhere             anywhere             /* Antrea: jump to Antrea prerouting rules */
    2    80 ACCEPT     tcp  --  any    any     anywhere             anywhere             tcp flags:RST/RST

Notice how that rule was hit twice despite the fact that there was a single TCP RST packet. So that packet must be going twice through PREROUTING and conntrack. The conntrack entry is destroyed the first time, which causes the packet to be dropped the second time because it is now invalid.

I read something related in the OVS documentation:

Firewall Rules
On Linux, when a physical interface is in use by the userspace datapath, packets received on the interface still also pass into the kernel TCP/IP stack. This can cause surprising and incorrect behavior. You can use "iptables" to avoid this behavior, by using it to drop received packets. For example, to drop packets received on eth0:

iptables -A INPUT -i eth0 -j DROP
iptables -A FORWARD -i eth0 -j DROP

These rules don't help with our situation (the packet still goes through conntrack twice). However the following rule did help:

iptables -t raw -A PREROUTING -i eth0 -j DROP

After that the TCP reset can make its way back to the source Pod as expected.

@GraysonWu It may be interesting to add this rule as part of https://github.com/vmware-tanzu/antrea/blob/main/build/images/scripts/start_ovs_netdev. Then we should be able to run the test instead of skipping it. It may help with other networking issues in Kind, who knows... We can look into this after #2001 is merged, there is no rush.

[GraysonWu]

Thanks @antoninbas for adding these helpful details. Yeah, we could try that later.