VPC egress ACL chain (ACL_OUTBOUND) rules are misordered: 225.0.0.50 accept and a private gateway's first rule land behind the deny #14354

problem

On a VPC virtual router, rules in the egress ACL chain ACL_OUTBOUND_ethX (mangle table) are not kept in the order the ACL list gives them. The chain's own rules can end up behind the ACL's deny.

CsNetfilters.compare() inserts each ACL rule ahead of the last rule already in its ACL chain. That is right for ACL_INBOUND_ethX, whose last rule is always its -j DROP. But ACL_OUTBOUND_ethX has no terminal rule, so whichever rule happens to be last is pushed behind every ACL rule:

• Tier with a public network. The chain starts with two accepts, for 224.0.0.18 (VRRP) and 225.0.0.50. The 225.0.0.50 accept ends up last, after an egress deny:

  -A ACL_OUTBOUND_eth3 -d 224.0.0.18/32 -j ACCEPT
  -A ACL_OUTBOUND_eth3 -d 10.9.9.9/32 -p tcp -m tcp --dport 443 -j ACCEPT
  -A ACL_OUTBOUND_eth3 -d 8.8.8.8/32 -p udp -m udp --dport 53 -j ACCEPT
  -A ACL_OUTBOUND_eth3 -j DROP
  -A ACL_OUTBOUND_eth3 -d 225.0.0.50/32 -j ACCEPT
  

  225.0.0.50 is the multicast group conntrackd uses to sync connection state between redundant VPC routers, over the guest interface (CsRedundant.py). The peer router's sync packets come from inside the tier's CIDR, so the PREROUTING ... -j ACL_OUTBOUND_ethX jump sends them through this chain. With an egress deny on that tier (the default_deny list has one), they are dropped. This part is from reading the code, not observed: redundant routers would stop syncing state, and connections would not survive a failover.
• Private gateway. The chain starts empty, so the ACL's first rule is the one pushed to the end, behind the deny, and never matches:

  -A ACL_OUTBOUND_eth3 -d 8.8.8.8/32 -p udp -m udp --dport 53 -j ACCEPT
  -A ACL_OUTBOUND_eth3 -j DROP
  -A ACL_OUTBOUND_eth3 -d 10.9.9.9/32 -p tcp -m tcp --dport 443 -j ACCEPT
  

  (ACL list: 1 allow tcp 443 to 10.9.9.9/32, 2 allow udp 53 to 8.8.8.8/32, 3 deny all.)

Ingress is not affected on a tier with a public network or on a private gateway, because ACL_INBOUND_ethX ends with its DROP there. On a VPC without a public network, a tier with a static route through it gets no DROP and has the same problem on ingress.

versions

Present on 4.22 (the insert logic and the chain layout are older than that).

The steps to reproduce the bug

1. In a VPC, create an ACL list with egress rules: 1 allow TCP 443 to 10.9.9.9/32, 2 allow UDP 53 to 8.8.8.8/32, 3 deny all to 0.0.0.0/0.
2. Attach it to a tier, and (separately) to a private gateway.
3. On the VR, run iptables -t mangle -S ACL_OUTBOUND_ethX for each interface. You get the output above: on the tier the 225.0.0.50 accept is after the DROP, and on the private gateway rule 1 is after the DROP.

What to do about it?

Close ACL_OUTBOUND_ethX with a terminal rule, -j RETURN, the way ACL_INBOUND_ethX is closed with its DROP. RETURN at the end of a user chain does what falling off its end does, so it doesn't change how traffic is treated, but it gives the insert logic the last rule it assumes.