TCP liveness/readiness probes are blocked by network policies

[aojea]

          > I added TCP liveness/readiness probes, and if I apply my "deny ingress" rule, the probes fail. This make sense to me, a deny ingress should block the probes. @aojea Do you know if there is some "don't do that" for this situation documented somewhere. Please, don't search for it, I can do that myself, but if you happen to know...

@danwinship is the source of truth for network policies 😄 and has also revisited the problem of probes recently, it also mentions this in the KEP kubernetes/enhancements#4558

Changing the definition of NetworkPolicy v1 to remove the "kubelet
  can access all local pods" hole, even in the case where the cluster
  is configured in a way to forbid the use of probes that require the
  hole. Even if _kubelet_ no longer needs to access pods, some users
  likely depend on that rule to allow access to some pods from other
  host-level processes, and there is no other way in NetworkPolicy v1
  to write such a rule if we remove the built-in one

I think that traffic originated from the node should not be subject for network policies.

I checked with nftables trace and all trafic coming from the node has the iif lo independently that the destination IP belongs to another interface

trace id 2da4b9ff ip filter INPUT packet: iif "lo" @ll,0,112 0x800 ip saddr 192.168.8.2 ip daddr 192.168.8.2 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 30324 ip length 60 tcp sport 41872 tcp dport 10250 tcp flags == syn tcp window 33280

Originally posted by @aojea in #54 (comment)

[uablrek]

I will probably not work with the implementation, but I am happy to help with testing and review

[aojea]

The trace is in INPUT and the traffic directed to the Pods will not come from that hook, the rule has to be in POSTROUTING

This is an example trace

trace id a14a3f32 ip filter trace_chain2 packet: oif "veth33dfd85a" ip saddr 10.244.2.1 ip daddr 10.244.2.5 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 18830 ip length 60 tcp sport 55132 tcp dport 80 tcp flags == syn tcp window 32120
trace id a14a3f32 ip filter trace_chain2 rule tcp dport 80 meta nftrace set 1 (verdict continue)
trace id a14a3f32 ip filter trace_chain2 verdict continue
trace id a14a3f32 ip filter trace_chain2 policy accept
trace id a14a3f32 ip nat POSTROUTING packet: oif "veth33dfd85a" ip saddr 10.244.2.1 ip daddr 10.244.2.5 ip dscp cs0 ip ecn not-ect ip ttl 64 ip id 18830 ip length 60 tcp sport 55132 tcp dport 80 tcp flags == syn tcp window 32120
trace id a14a3f32 ip nat POSTROUTING rule  counter packets 480 bytes 33091 jump KUBE-POSTROUTING (verdict jump KUBE-POSTROUTING)
trace id a14a3f32 ip nat KUBE-POSTROUTING verdict return
trace id a14a3f32 ip nat POSTROUTING rule fib daddr type != local  counter packets 162 bytes 9720 jump KIND-MASQ-AGENT (verdict jump KIND-MASQ-AGENT)
trace id a14a3f32 ip nat KIND-MASQ-AGENT verdict return
trace id a14a3f32 ip nat POSTROUTING verdict continue
trace id a14a3f32 ip nat POSTROUTING policy accept

I think a rule like fib saddr type local accept will solve this problem

[vaskozl]

I have built 5d52862 and can confirm that this solves the issue. Everything works as expected with the addition meta skuid 0 accept. :)