Add L2 networks to Zones with SG #7719

benj-n · 2023-07-04T23:49:49Z

Description

Currently, it is not possible to create a raw L2 network within an advanced zone with security groups enabled.
It is expected that Cloudstack does not have any control over the security in an L2 network, by the definition itself of such network.

The fact that security groups are enabled or disabled should not be a blocker for L2 networks. Especially as these L2 can be useful when used as secondary networks (with no services) in a zone with existing shared networks protected by security groups.

This PR doesn't add a new feature per se, but rather prevent cloudstack from blocking the creation of L2 networks.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

DaanHoogland · 2023-07-05T09:21:35Z

@blueorangutan package

blueorangutan · 2023-07-05T09:22:03Z

@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-07-05T10:23:38Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6414

DaanHoogland · 2023-07-05T12:32:06Z

@blueorangutan test

blueorangutan · 2023-07-05T12:34:06Z

@DaanHoogland a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2023-07-06T00:58:53Z

[SF] Trillian test result (tid-6994)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 43012 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7719-t6994-kvm-centos7.zip
Smoke tests completed. 111 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	`Error`	80.63	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	52.24	test_vm_life_cycle.py
test_05_vmschedule_test_e2e	`Failure`	170.58	test_vm_schedule.py

DaanHoogland · 2023-07-06T07:31:36Z

@benj-n can you have a quick look at the test_vm_schedule failure? This is in a recent merge.
The test_vm_life_cycle failures are addressed in #7718

DaanHoogland · 2023-07-06T08:20:00Z

@benj-n can you have a quick look at the test_vm_schedule failure? This is in a recent merge. The test_vm_life_cycle failures are addressed in #7718

never mind @benj-n , I think we have already found a cause outside of your PR. #7344 also shows the error.

weizhouapache · 2023-07-06T09:11:22Z

@benj-n
have you tested it ? if so , can you share the related iptables/ebtables rules and output of ipset list command ?

benj-n · 2023-07-07T00:11:43Z

@benj-n have you tested it ? if so , can you share the related iptables/ebtables rules and output of ipset list command ?

In the test, a VM has a first NIC with shared network (and security groups) on vlan100/vnet16 and a second NIC with an L2 network (vlan2150/vnet20):

root@node01:~# virsh domiflist i-2-596-VM
 Interface   Type     Source         Model    MAC
-----------------------------------------------------------------
 vnet16      bridge   brbond0-100    virtio   1e:00:06:00:20:29
 vnet20      bridge   brbond9-2150   virtio   02:00:1d:52:00:01

The vnet20 and the L2 network are completely absent from iptables :

Chain BF-brbond0-100-IN (1 references)
 pkts bytes target     prot opt in     out     source               destination
(...)
  240 24611 i-2-596-def  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged

Chain BF-brbond0-100-OUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
(...)
  876 43779 i-2-596-def  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet16 --physdev-is-bridged


Chain i-2-596-VM (1 references)
 pkts bytes target     prot opt in     out     source               destination
   14   973 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 255
  417 21629 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW
  150  6004 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain i-2-596-VM-eg (1 references)
 pkts bytes target     prot opt in     out     source               destination
  136  9456 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain i-2-596-def (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
   32 10556 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged udp spt:68 dpt:67
    2   667 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet16 --physdev-is-bridged udp spt:67 dpt:68
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged udp spt:67
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ! match-set i-2-596-VM src
  293 14506 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ! match-set i-2-596-VM dst
   72  4599 RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged match-set i-2-596-VM src udp dpt:53
    0     0 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged match-set i-2-596-VM src tcp dpt:53
  136  9456 i-2-596-VM-eg  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged match-set i-2-596-VM src
  581 28606 i-2-596-VM  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet16 --physdev-is-bridged

The L2 is also completely absent from ip6tables:

Chain BF-brbond0-100-IN (1 references)
 pkts bytes target     prot opt in     out     source               destination
(...)
   90  6768 i-2-596-def  all      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged

Chain BF-brbond0-100-OUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
(...)
   21  2184 i-2-596-def  all      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged

Chain i-2-596-VM (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0
    0     0 ACCEPT     all      *      *       ::/0                 ::/0                 state NEW
    0     0 DROP       all      *      *       ::/0                 ::/0

Chain i-2-596-VM-eg (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all      *      *       ::/0                 ::/0                 state NEW
    0     0 DROP       all      *      *       ::/0                 ::/0

Chain i-2-596-def (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all      *      *       ::/0                 ::/0                 state RELATED,ESTABLISHED
   21  2184 ACCEPT     icmpv6    *      *       fe80::/64            ff02::1              PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 134 HL match HL == 255
    0     0 RETURN     icmpv6    *      *       ::/0                 ff02::2              PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 133 HL match HL == 255
    0     0 DROP       icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 134
   18  1296 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 135 HL match HL == 255
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 135 HL match HL == 255
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 136 match-set i-2-596-VM-6 src HL match HL == 255
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 136 HL match HL == 255
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 2 match-set i-2-596-VM-6 src
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 2
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 1 match-set i-2-596-VM-6 src
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 1
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 3 match-set i-2-596-VM-6 src
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 3
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 4 match-set i-2-596-VM-6 src
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 4
   72  5472 RETURN     icmpv6    *      *       ::/0                 ff02::16             PHYSDEV match --physdev-in vnet16 --physdev-is-bridged
    0     0 RETURN     udp      *      *       fe80::1c00:6ff:fe00:2029  ff02::1:2            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged udp spt:546
    0     0 ACCEPT     udp      *      *       fe80::/64            fe80::1c00:6ff:fe00:2029  PHYSDEV match --physdev-out vnet16 --physdev-is-bridged udp dpt:546
    0     0 DROP       udp      *      *       ::/0                !fe80::/64            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged udp spt:547
    0     0 RETURN     udp      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged udp dpt:53 match-set i-2-596-VM-6 src
    0     0 RETURN     tcp      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged tcp dpt:53 match-set i-2-596-VM-6 src
    0     0 DROP       all      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ! match-set i-2-596-VM-6 src
    0     0 i-2-596-VM-eg  all      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged match-set i-2-596-VM-6 src
    0     0 i-2-596-VM  all      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged

And the ipset only references IPs from the shared network:


Name: i-2-596-VM
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 248
References: 5
Number of entries: 1
Members:
85.(redacted-public-ipv4).6

Name: i-2-596-VM-6
Type: hash:net
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536
Size in memory: 1456
References: 9
Number of entries: 2
Members:
fe80::1c00:6ff:fe00:2029
2001:(redacted-ipv6):2029

It's the same for ebtables, as expected, the L2 network is totally ignored there too.

Bridge chain: PREROUTING, entries: 6, policy: ACCEPT
(...)
-i vnet16 -j i-2-596-VM-in

Bridge chain: POSTROUTING, entries: 6, policy: ACCEPT
(...)
-o vnet16 -j i-2-596-VM-out

Bridge chain: i-2-596-VM-in, entries: 5, policy: ACCEPT
-j i-2-596-VM-in-src
-p ARP -j i-2-596-VM-in-ips
-p ARP --arp-op Request -j ACCEPT
-p ARP --arp-op Reply -j ACCEPT
-p ARP -j DROP

Bridge chain: i-2-596-VM-out, entries: 5, policy: ACCEPT
-p ARP --arp-op Reply -j i-2-596-VM-out-dst
-p ARP -j i-2-596-VM-out-ips
-p ARP --arp-op Request -j ACCEPT
-p ARP --arp-op Reply -j ACCEPT
-p ARP -j DROP

Bridge chain: i-2-596-VM-in-ips, entries: 2, policy: ACCEPT
-p ARP -s 1e:00:06:00:20:29 --arp-ip-src 85.(redacted-public-ipv4).6 --arp-mac-src 1e:00:06:00:20:29 -j RETURN
-j DROP

Bridge chain: i-2-596-VM-out-ips, entries: 2, policy: ACCEPT
-p ARP --arp-ip-dst 85.(redacted-public-ipv4).6 -j RETURN
-j DROP

Bridge chain: i-2-596-VM-in-src, entries: 2, policy: ACCEPT
-s 1e:00:06:00:20:29 -j RETURN
-j DROP

Bridge chain: i-2-596-VM-out-dst, entries: 2, policy: ACCEPT
-p ARP --arp-op Reply --arp-mac-dst 1e:00:06:00:20:29 -j RETURN
-p ARP --arp-op Reply -j DROP

DaanHoogland

code looks good, not sure about the side effects. needs extensive testing

weizhouapache · 2023-07-07T07:51:32Z

@benj-n have you tested it ? if so , can you share the related iptables/ebtables rules and output of ipset list command ?

In the test, a VM has a first NIC with shared network (and security groups) on vlan100/vnet16 and a second NIC with an L2 network (vlan2150/vnet20):

root@node01:~# virsh domiflist i-2-596-VM
 Interface   Type     Source         Model    MAC
-----------------------------------------------------------------
 vnet16      bridge   brbond0-100    virtio   1e:00:06:00:20:29
 vnet20      bridge   brbond9-2150   virtio   02:00:1d:52:00:01

The vnet20 and the L2 network are completely absent from iptables :

Chain BF-brbond0-100-IN (1 references)
 pkts bytes target     prot opt in     out     source               destination
(...)
  240 24611 i-2-596-def  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged

Chain BF-brbond0-100-OUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
(...)
  876 43779 i-2-596-def  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet16 --physdev-is-bridged


Chain i-2-596-VM (1 references)
 pkts bytes target     prot opt in     out     source               destination
   14   973 ACCEPT     icmp --  *      *       0.0.0.0/0            0.0.0.0/0            icmptype 255
  417 21629 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW
  150  6004 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain i-2-596-VM-eg (1 references)
 pkts bytes target     prot opt in     out     source               destination
  136  9456 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state NEW
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain i-2-596-def (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
   32 10556 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged udp spt:68 dpt:67
    2   667 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet16 --physdev-is-bridged udp spt:67 dpt:68
    0     0 DROP       udp  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged udp spt:67
    0     0 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ! match-set i-2-596-VM src
  293 14506 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ! match-set i-2-596-VM dst
   72  4599 RETURN     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged match-set i-2-596-VM src udp dpt:53
    0     0 RETURN     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged match-set i-2-596-VM src tcp dpt:53
  136  9456 i-2-596-VM-eg  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged match-set i-2-596-VM src
  581 28606 i-2-596-VM  all  --  *      *       0.0.0.0/0            0.0.0.0/0            PHYSDEV match --physdev-out vnet16 --physdev-is-bridged

The L2 is also completely absent from ip6tables:

Chain BF-brbond0-100-IN (1 references)
 pkts bytes target     prot opt in     out     source               destination
(...)
   90  6768 i-2-596-def  all      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged

Chain BF-brbond0-100-OUT (1 references)
 pkts bytes target     prot opt in     out     source               destination
(...)
   21  2184 i-2-596-def  all      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged

Chain i-2-596-VM (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0
    0     0 ACCEPT     all      *      *       ::/0                 ::/0                 state NEW
    0     0 DROP       all      *      *       ::/0                 ::/0

Chain i-2-596-VM-eg (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all      *      *       ::/0                 ::/0                 state NEW
    0     0 DROP       all      *      *       ::/0                 ::/0

Chain i-2-596-def (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all      *      *       ::/0                 ::/0                 state RELATED,ESTABLISHED
   21  2184 ACCEPT     icmpv6    *      *       fe80::/64            ff02::1              PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 134 HL match HL == 255
    0     0 RETURN     icmpv6    *      *       ::/0                 ff02::2              PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 133 HL match HL == 255
    0     0 DROP       icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 134
   18  1296 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 135 HL match HL == 255
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 135 HL match HL == 255
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 136 match-set i-2-596-VM-6 src HL match HL == 255
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 136 HL match HL == 255
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 2 match-set i-2-596-VM-6 src
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 2
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 1 match-set i-2-596-VM-6 src
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 1
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 3 match-set i-2-596-VM-6 src
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 3
    0     0 RETURN     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ipv6-icmptype 4 match-set i-2-596-VM-6 src
    0     0 ACCEPT     icmpv6    *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged ipv6-icmptype 4
   72  5472 RETURN     icmpv6    *      *       ::/0                 ff02::16             PHYSDEV match --physdev-in vnet16 --physdev-is-bridged
    0     0 RETURN     udp      *      *       fe80::1c00:6ff:fe00:2029  ff02::1:2            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged udp spt:546
    0     0 ACCEPT     udp      *      *       fe80::/64            fe80::1c00:6ff:fe00:2029  PHYSDEV match --physdev-out vnet16 --physdev-is-bridged udp dpt:546
    0     0 DROP       udp      *      *       ::/0                !fe80::/64            PHYSDEV match --physdev-in vnet16 --physdev-is-bridged udp spt:547
    0     0 RETURN     udp      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged udp dpt:53 match-set i-2-596-VM-6 src
    0     0 RETURN     tcp      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged tcp dpt:53 match-set i-2-596-VM-6 src
    0     0 DROP       all      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged ! match-set i-2-596-VM-6 src
    0     0 i-2-596-VM-eg  all      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-in vnet16 --physdev-is-bridged match-set i-2-596-VM-6 src
    0     0 i-2-596-VM  all      *      *       ::/0                 ::/0                 PHYSDEV match --physdev-out vnet16 --physdev-is-bridged

And the ipset only references IPs from the shared network:


Name: i-2-596-VM
Type: hash:ip
Revision: 4
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 248
References: 5
Number of entries: 1
Members:
85.(redacted-public-ipv4).6

Name: i-2-596-VM-6
Type: hash:net
Revision: 6
Header: family inet6 hashsize 1024 maxelem 65536
Size in memory: 1456
References: 9
Number of entries: 2
Members:
fe80::1c00:6ff:fe00:2029
2001:(redacted-ipv6):2029

It's the same for ebtables, as expected, the L2 network is totally ignored there too.

Bridge chain: PREROUTING, entries: 6, policy: ACCEPT
(...)
-i vnet16 -j i-2-596-VM-in

Bridge chain: POSTROUTING, entries: 6, policy: ACCEPT
(...)
-o vnet16 -j i-2-596-VM-out

Bridge chain: i-2-596-VM-in, entries: 5, policy: ACCEPT
-j i-2-596-VM-in-src
-p ARP -j i-2-596-VM-in-ips
-p ARP --arp-op Request -j ACCEPT
-p ARP --arp-op Reply -j ACCEPT
-p ARP -j DROP

Bridge chain: i-2-596-VM-out, entries: 5, policy: ACCEPT
-p ARP --arp-op Reply -j i-2-596-VM-out-dst
-p ARP -j i-2-596-VM-out-ips
-p ARP --arp-op Request -j ACCEPT
-p ARP --arp-op Reply -j ACCEPT
-p ARP -j DROP

Bridge chain: i-2-596-VM-in-ips, entries: 2, policy: ACCEPT
-p ARP -s 1e:00:06:00:20:29 --arp-ip-src 85.(redacted-public-ipv4).6 --arp-mac-src 1e:00:06:00:20:29 -j RETURN
-j DROP

Bridge chain: i-2-596-VM-out-ips, entries: 2, policy: ACCEPT
-p ARP --arp-ip-dst 85.(redacted-public-ipv4).6 -j RETURN
-j DROP

Bridge chain: i-2-596-VM-in-src, entries: 2, policy: ACCEPT
-s 1e:00:06:00:20:29 -j RETURN
-j DROP

Bridge chain: i-2-596-VM-out-dst, entries: 2, policy: ACCEPT
-p ARP --arp-op Reply --arp-mac-dst 1e:00:06:00:20:29 -j RETURN
-p ARP --arp-op Reply -j DROP

@benj-n
looks ok.
can you create a vm with only L2 networks ?

btw: did you create the L2 network with specified vlan id ?

DaanHoogland · 2023-07-07T09:41:41Z

@weizhouapache @benj-n is this applicable for 4.18 as well?

weizhouapache · 2023-07-07T10:26:50Z

@weizhouapache @benj-n is this applicable for 4.18 as well?

@DaanHoogland yes, very nice feature with small changes. I'd like to add it to 4.18.1.0 , unless someone objects.
@benj-n can you rebase with 4.18 ?

NuxRo · 2023-07-07T16:54:33Z

Great feature. Did a bit of testing and largely my findings are similar to those of Wei, ie the VM gets connected in the appropriate L2 network, but no iptables or ebtables rules are set up for it, so far so good.

A few notes:

1 - network traffic between machines on separate HVs worked because iptables defaults to ACCEPT for INPUT, FORWARD and OUTPUT. If an operator makes changes there it will result in issues. This is just FYI, I am not saying we should do any action in this kind of scenarios.

2 - Creating a VM only on an L2 network is not possible currently, Cloudstack returns:
Can specify only Shared Guest networks when deploy vm in Advance Security Group enabled zone
It would be GREAT if the feature allowed L2-only!

3 - Network offering "Offering for L2 networks" did not produce a functional VM deployment for me, had to go for "Offering for L2 networks VLAN" and specify a VLAN. Not a major issue, but worth keeping in mind.

4 - Last but NOT least, unlike Shared networks, the scope of L2 networks is account, so there can't be an expectation for multiple accounts to be able to connect in the same network. Just making sure we're aware of this. :)

Looking forward to having this feature in Cloudstack!

benj-n · 2023-07-07T18:46:33Z

btw: did you create the L2 network with specified vlan id ?

Yes. Unfortunately, I can only have tagged traffic on the specific L2 network I have for testing.

benj-n · 2023-07-07T19:42:21Z

@DaanHoogland yes, very nice feature with small changes. I'd like to add it to 4.18.1.0 , unless someone objects. @benj-n can you rebase with 4.18 ?

It's now rebased on 4.18.

DaanHoogland · 2023-07-10T07:39:13Z

@blueorangutan package

blueorangutan · 2023-07-10T07:44:08Z

@DaanHoogland a [SF] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2023-07-10T08:50:23Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 6451

DaanHoogland · 2023-07-10T10:08:08Z

@blueorangutan test

blueorangutan · 2023-07-10T10:10:05Z

@DaanHoogland a [SF] Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

NuxRo · 2023-07-10T15:23:03Z

Good job rebasing. I have tested and my results are identical, the feature works as per my previous attempt on 4.19.
I think it looks good as an initial implementation.

In the future it will be nice if we were able to deploy on l2-net only as well as have users be able to create them themselves from a pool of vlans.

soreana · 2023-07-10T15:31:28Z

Good job rebasing. I have tested and my results are identical, the feature works as per my previous attempt on 4.19. I think it looks good as an initial implementation.

In the future it will be nice if we were able to deploy on l2-net only as well as have users be able to create them themselves from a pool of vlans.

@NuxRo Thanks for testing. 🙏

Those are awesome features, but let's keep the scope as it is for now. Those changes require a lot to be done, we can plan them later.

NuxRo

LGTM based on testing with KVM/EL8 deployments, both 4.18 and 4.19.

blueorangutan · 2023-07-11T00:12:36Z

[SF] Trillian test result (tid-7036)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 49116 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr7719-t7036-kvm-centos7.zip
Smoke tests completed. 107 look OK, 1 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_01_migrate_VM_and_root_volume	`Error`	78.76	test_vm_life_cycle.py
test_02_migrate_VM_with_two_data_disks	`Error`	55.39	test_vm_life_cycle.py

* 4.18: Storage and volumes statistics tasks for StorPool primary storage (#7404) proper storage construction (#6797) guarantee MAC uniqueness (#7634) server: allow migration of all VMs with local storage on KVM (#7656) Add L2 networks to Zones with SG (#7719)

weizhouapache · 2023-07-20T05:48:36Z

Good job rebasing. I have tested and my results are identical, the feature works as per my previous attempt on 4.19. I think it looks good as an initial implementation.
In the future it will be nice if we were able to deploy on l2-net only as well as have users be able to create them themselves from a pool of vlans.

@NuxRo Thanks for testing. pray

Those are awesome features, but let's keep the scope as it is for now. Those changes require a lot to be done, we can plan them later.

@soreana @NuxRo
can you create a github issue for tracking ?

boring-cyborg bot added component:api component:networking component:orchestration component:UI labels Jul 4, 2023

DaanHoogland requested a review from weizhouapache July 6, 2023 08:20

DaanHoogland added the component:security-group label Jul 6, 2023

weizhouapache modified the milestones: 4.18.1.0, 4.19.0.0 Jul 6, 2023

DaanHoogland approved these changes Jul 7, 2023

View reviewed changes

Add L2 networks to Zones with SG

a80b3c4

benj-n changed the base branch from main to 4.18 July 7, 2023 19:38

boring-cyborg bot added component:agent component:build component:integration labels Jul 7, 2023

benj-n force-pushed the L2SG branch from d623369 to a80b3c4 Compare July 7, 2023 19:38

soreana modified the milestones: 4.19.0.0, 4.18.1.0 Jul 10, 2023

soreana requested a review from NuxRo July 10, 2023 15:33

NuxRo approved these changes Jul 10, 2023

View reviewed changes

DaanHoogland closed this Jul 11, 2023

DaanHoogland reopened this Jul 11, 2023

DaanHoogland merged commit 981d748 into apache:4.18 Jul 11, 2023

weizhouapache mentioned this pull request Oct 20, 2023

Advanced SG: allow create/update physical networks with vlan range #8122

Merged

13 tasks

benj-n deleted the L2SG branch January 26, 2024 20:56

Add L2 networks to Zones with SG #7719

Add L2 networks to Zones with SG #7719

Uh oh!

Conversation

benj-n commented Jul 4, 2023

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Uh oh!

DaanHoogland commented Jul 5, 2023

Uh oh!

blueorangutan commented Jul 5, 2023

Uh oh!

blueorangutan commented Jul 5, 2023

Uh oh!

DaanHoogland commented Jul 5, 2023

Uh oh!

blueorangutan commented Jul 5, 2023

Uh oh!

blueorangutan commented Jul 6, 2023

Uh oh!

DaanHoogland commented Jul 6, 2023

Uh oh!

DaanHoogland commented Jul 6, 2023

Uh oh!

weizhouapache commented Jul 6, 2023

Uh oh!

benj-n commented Jul 7, 2023

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

weizhouapache commented Jul 7, 2023

Uh oh!

DaanHoogland commented Jul 7, 2023

Uh oh!

weizhouapache commented Jul 7, 2023

Uh oh!

NuxRo commented Jul 7, 2023

Uh oh!

benj-n commented Jul 7, 2023

Uh oh!

benj-n commented Jul 7, 2023

Uh oh!

DaanHoogland commented Jul 10, 2023

Uh oh!

blueorangutan commented Jul 10, 2023

Uh oh!

blueorangutan commented Jul 10, 2023

Uh oh!

DaanHoogland commented Jul 10, 2023

Uh oh!

blueorangutan commented Jul 10, 2023

Uh oh!

NuxRo commented Jul 10, 2023

Uh oh!

soreana commented Jul 10, 2023

Uh oh!

NuxRo left a comment

Choose a reason for hiding this comment

Uh oh!

blueorangutan commented Jul 11, 2023

Uh oh!

weizhouapache commented Jul 20, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants