Remove IPSec tunnel limitation #2489

luolanzone · 2021-07-29T08:54:29Z

I was trying to enable IPSec with VxLAN tunnel, after verify the traffic in real cluster, it turns out Antrea can actually support all tunnel modes now without much code changes.
I verified all tunnel modes with IPSec in a local Cluster, it all works fine now, so I remove the limitation on our docs and the check in the code.

resolves #2468

codecov-commenter · 2021-07-29T10:11:57Z

Codecov Report

Merging #2489 (1858113) into main (d785904) will increase coverage by 11.23%.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             main    #2489       +/-   ##
===========================================
+ Coverage   42.51%   53.75%   +11.23%     
===========================================
  Files         148      283      +135     
  Lines       18247    22455     +4208     
===========================================
+ Hits         7758    12070     +4312     
+ Misses       9789     9099      -690     
- Partials      700     1286      +586

Flag	Coverage Δ
kind-e2e-tests	`35.74% <ø> (?)`
unit-tests	`41.93% <ø> (-0.59%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/apiserver/handlers/endpoint/handler.go	`58.82% <0.00%> (-11.77%)`	⬇️
pkg/apiserver/handlers/webhook/mutation_labels.go	`24.71% <0.00%> (ø)`
pkg/legacyclient/listers/security/v1alpha1/tier.go	`0.00% <0.00%> (ø)`
pkg/apis/stats/v1alpha1/register.go	`90.90% <0.00%> (ø)`
...ed/typed/security/v1alpha1/clusternetworkpolicy.go	`0.00% <0.00%> (ø)`
pkg/apiserver/handlers/loglevel/handler.go	`0.00% <0.00%> (ø)`
pkg/legacyclient/clientset/versioned/clientset.go	`22.58% <0.00%> (ø)`
pkg/agent/proxy/metrics/metrics.go	`100.00% <0.00%> (ø)`
pkg/agent/proxy/types/types.go	`46.66% <0.00%> (ø)`
pkg/util/logdir/logdir.go	`0.00% <0.00%> (ø)`
... and 215 more

jianjuns

In my previous tests, only GRE worked. But my testbed was Ubuntu 16.04, not sure if it is too old.

Please change antrea-ipsec.yml to use Geneve tunnel type and make sure CI tests can pass before merge the change.

build/yamls/antrea-ipsec.yml

jianjuns · 2021-08-09T17:28:11Z

docs/ipsec-tunnel.md

-moment, IPsec encyption works only for GRE tunnel (but not Geneve, VXLAN, and
-STT tunnel types).
+Antrea supports encrypting tunnel traffic across Nodes with IPsec ESP.
+IPsec encyption works for all OVS supported tunnel including GRE, Geneve,


Then move Geneve to the be first, before GRE.

luolanzone · 2021-08-10T01:16:17Z

/test-all

luolanzone · 2021-08-10T02:09:20Z

/test-ipv6-all

luolanzone · 2021-08-10T02:09:27Z

/test-ipv6-only-all

luolanzone · 2021-08-11T02:01:32Z

@jianjuns e2e test failed on CI testbed with IPsec+Geneve mode , after double check in my local clusters, it turns out it's kernel version issue, I test in one cluster with ubuntu 18.04.3 which has kernel version 4.15.0-66 failed, but another one with 4.15.0-143 passed. I suppose we need to update docs to point out the kernel version to support different tunnel mode. any other suggestion for this kind of requirement?

jianjuns · 2021-08-11T04:15:20Z

I think we need to figure out from exact what kernel version or with what kernel change Geneve can pass. I am surprised it can change from 4.15.0-66 to 4.15.0-143.

luolanzone · 2021-08-11T05:55:22Z

@jianjuns yes, I will figure out what's changed or required, I was actually running 'apt-get upgrade', so I suspect maybe it's some software upgrades fix it. but I need time to check.

luolanzone · 2021-08-16T01:26:50Z

Hi @jianjuns the change or fix is introduced in this commit: https://kernel.ubuntu.com/git/ubuntu/ubuntu-bionic.git/commit/?id=be556894e8eaefd5d21690d56614d76e45786ecb, I verified it on my local cluster, for ubuntu 18.04, it has to be a version after 4.15.0-124. if we use Geneve as default tunnel for IPSec, we have to upgrade the e2e environment to pass it. do you think it's OK to still keep the old setting to use IPSec+GRE so we can run e2e without testbed change?

jianjuns · 2021-08-16T17:17:32Z

@luolanzone: thanks for the update! Given Geneve and others are not supported by earlier kernel version, I think still better to keep GRE to be the default for antrea-ipsec.yml, and test GRE in CI. I first thought we should update CI testbed to support Geneve + IPsec, but even we update testbed do we want to run tests for both GRE and Geneve? @antoninbas : thoughts?

antoninbas · 2021-08-16T17:36:48Z

@luolanzone: thanks for the update! Given Geneve and others are not supported by earlier kernel version, I think still better to keep GRE to be the default for antrea-ipsec.yml, and test GRE in CI. I first thought we should update CI testbed to support Geneve + IPsec, but even we update testbed do we want to run tests for both GRE and Geneve? @antoninbas : thoughts?

I feel like it's ok to test a single transport (GRE) since the code is the same on the Antrea side. Just like we run pretty much all of our testing with Geneve (there are some leftover tests for VXLAN, but it won't necessarily always be the case). We should document the kernel version requirements for using IPsec + Geneve though.

luolanzone · 2021-08-17T01:50:16Z

Hi @antoninbas @jianjuns , I change the manifest back to use GRE with IPSec, and update the doc ipsec-tunnel.md with kernel requirement. please help to review again, thanks!

jianjuns · 2021-08-17T20:32:29Z

docs/ipsec-tunnel.md


 ## Prerequisites

 IPsec requires a set of Linux kernel modules. Check the required kernel modules
 listed in the [strongSwan documentation](https://wiki.strongswan.org/projects/strongswan/wiki/KernelModules).
 Make sure the required kernel modules are loaded on the Kubernetes Nodes before
 deploying Antrea with IPsec encyption enabled.
-
+If you want to enable IPsec with Geneve, please make sure [this commit](https://github.com/torvalds/linux/commit/34beb21594519ce64a55a498c2fe7d567bc1ca20) is included


So do VXLAN and STT work with older 18.04 versions, without the commit?

Add a new line before this paragraph.

yes, VXLAN works, STT module is not in the Linux tree, so need extra step to install it manually from source, I have updated here last time, let me add it here as well.

I do not see you added a new line.

ah, misunderstood the new line, I added it before ## Installation, will add it.

jianjuns · 2021-08-17T20:33:09Z

docs/ipsec-tunnel.md


 ## Prerequisites

 IPsec requires a set of Linux kernel modules. Check the required kernel modules
 listed in the [strongSwan documentation](https://wiki.strongswan.org/projects/strongswan/wiki/KernelModules).
 Make sure the required kernel modules are loaded on the Kubernetes Nodes before
 deploying Antrea with IPsec encyption enabled.
-
+If you want to enable IPsec with Geneve, please make sure [this commit](https://github.com/torvalds/linux/commit/34beb21594519ce64a55a498c2fe7d567bc1ca20) is included
+in your kernel. for Ubuntu 18.04, it should be a version after 4.15.0-124. for Ubuntu 20.04, it should be a version after 5.4.67.


jianjuns · 2021-08-18T05:15:36Z

docs/ipsec-tunnel.md


 ## Prerequisites

 IPsec requires a set of Linux kernel modules. Check the required kernel modules
 listed in the [strongSwan documentation](https://wiki.strongswan.org/projects/strongswan/wiki/KernelModules).
 Make sure the required kernel modules are loaded on the Kubernetes Nodes before
 deploying Antrea with IPsec encyption enabled.
+If you want to enable IPsec with Geneve, please make sure [this commit](https://github.com/torvalds/linux/commit/34beb21594519ce64a55a498c2fe7d567bc1ca20) is included
+in your kernel. For Ubuntu 18.04, it should be a version after `4.15.0-124`. For Ubuntu 20.04, it should be a version after `5.4.67`.
+If you'd like to enable IPSec with STT, please be ware that the `vport-stt` module is not in the Linux tree and needs to be built from source, please build and load it manually before IPSec with STT tunneling is enabled.


I feel no need to mention the STT prerequisite here. It is documented in getting-started already.

jianjuns · 2021-08-18T05:15:55Z

docs/ipsec-tunnel.md


 ## Prerequisites

 IPsec requires a set of Linux kernel modules. Check the required kernel modules
 listed in the [strongSwan documentation](https://wiki.strongswan.org/projects/strongswan/wiki/KernelModules).
 Make sure the required kernel modules are loaded on the Kubernetes Nodes before
 deploying Antrea with IPsec encyption enabled.
-
+If you want to enable IPsec with Geneve, please make sure [this commit](https://github.com/torvalds/linux/commit/34beb21594519ce64a55a498c2fe7d567bc1ca20) is included


I do not see you added a new line.

jianjuns

LGTM.

luolanzone · 2021-08-18T06:05:21Z

/test-all /test-ipv6-all /test-ipv6-only-all

antoninbas · 2021-08-18T18:38:13Z

docs/ipsec-tunnel.md

-moment, IPsec encyption works only for GRE tunnel (but not Geneve, VXLAN, and
-STT tunnel types).
+Antrea supports encrypting tunnel traffic across Nodes with IPsec ESP.
+IPsec encyption works for all OVS supported tunnel including Geneve, GRE,


Suggested change

IPsec encyption works for all OVS supported tunnel including Geneve, GRE,

IPsec encyption works for all tunnel types supported by OVS including Geneve, GRE,

antoninbas · 2021-08-18T18:39:07Z

docs/ipsec-tunnel.md

@@ -11,6 +11,9 @@ listed in the [strongSwan documentation](https://wiki.strongswan.org/projects/st
 Make sure the required kernel modules are loaded on the Kubernetes Nodes before
 deploying Antrea with IPsec encyption enabled.

+If you want to enable IPsec with Geneve, please make sure [this commit](https://github.com/torvalds/linux/commit/34beb21594519ce64a55a498c2fe7d567bc1ca20) is included
+in your kernel. For Ubuntu 18.04, it should be a version after `4.15.0-124`. For Ubuntu 20.04, it should be a version after `5.4.67`.


wrap line at 80 chars

antoninbas · 2021-08-18T18:40:06Z

docs/ipsec-tunnel.md

@@ -11,6 +11,9 @@ listed in the [strongSwan documentation](https://wiki.strongswan.org/projects/st
 Make sure the required kernel modules are loaded on the Kubernetes Nodes before
 deploying Antrea with IPsec encyption enabled.

+If you want to enable IPsec with Geneve, please make sure [this commit](https://github.com/torvalds/linux/commit/34beb21594519ce64a55a498c2fe7d567bc1ca20) is included
+in your kernel. For Ubuntu 18.04, it should be a version after `4.15.0-124`. For Ubuntu 20.04, it should be a version after `5.4.67`.


s/it should be a version after/kernel version should be at least

(both occurrences)

(assuming that 4.15.0-124 itself includes the patch, but typically the provided version number is the first one which includes the required patch).

Ok, there are kernel version tags like 4.15.0-125 and 5.4.68, but I can't get them from apt list. so minimum working version should be 4.15.0-128 and 5.4.70. let me correct it.

this PR is based on antrea-io#2486 and I verified all tunnel modes with IPSec in K8s Cluster, it all works fine now, so I remove the limitation on our docs and the check in the code. Signed-off-by: Lan Luo <luola@vmware.com>

antoninbas · 2021-08-19T19:07:56Z

/skip-all

We have verified that all tunnel types can be used for IPsec. When using Geneve though, a recent enough kernel version is required. Signed-off-by: Lan Luo <luola@vmware.com>

luolanzone force-pushed the ipsec-vxlan branch 2 times, most recently from d145bfd to fa00c64 Compare July 29, 2021 09:20

luolanzone force-pushed the ipsec-vxlan branch from fa00c64 to fe9cac1 Compare August 9, 2021 02:26

luolanzone changed the title ~~[WIP]Remove IPSec tunnel limitation~~ Remove IPSec tunnel limitation Aug 9, 2021

luolanzone force-pushed the ipsec-vxlan branch from fe9cac1 to 7d50614 Compare August 9, 2021 02:29

luolanzone requested review from jianjuns, tnqn and antoninbas August 9, 2021 02:29

jianjuns reviewed Aug 9, 2021

View reviewed changes

luolanzone force-pushed the ipsec-vxlan branch from 7d50614 to 4d7473b Compare August 10, 2021 01:15

luolanzone force-pushed the ipsec-vxlan branch from 4d7473b to 8084d87 Compare August 17, 2021 01:46

jianjuns reviewed Aug 17, 2021

View reviewed changes

luolanzone force-pushed the ipsec-vxlan branch from 8084d87 to a306b94 Compare August 18, 2021 01:13

jianjuns reviewed Aug 18, 2021

View reviewed changes

luolanzone force-pushed the ipsec-vxlan branch from a306b94 to fdb5457 Compare August 18, 2021 05:42

jianjuns previously approved these changes Aug 18, 2021

View reviewed changes

antoninbas reviewed Aug 18, 2021

View reviewed changes

remove tunnel limitation for IPSec

1858113

this PR is based on antrea-io#2486 and I verified all tunnel modes with IPSec in K8s Cluster, it all works fine now, so I remove the limitation on our docs and the check in the code. Signed-off-by: Lan Luo <luola@vmware.com>

luolanzone dismissed jianjuns’s stale review via 1858113 August 19, 2021 01:15

luolanzone force-pushed the ipsec-vxlan branch from fdb5457 to 1858113 Compare August 19, 2021 01:15

antoninbas approved these changes Aug 19, 2021

View reviewed changes

antoninbas merged commit fdf3034 into antrea-io:main Aug 19, 2021

luolanzone mentioned this pull request Sep 13, 2021

Automated cherry pick of #2489: remove tunnel limitation for IPSec #2764

Merged

luolanzone deleted the ipsec-vxlan branch September 14, 2021 09:23

	IPsec encyption works for all OVS supported tunnel including Geneve, GRE,
	IPsec encyption works for all tunnel types supported by OVS including Geneve, GRE,

Remove IPSec tunnel limitation #2489

Remove IPSec tunnel limitation #2489

Conversation

luolanzone commented Jul 29, 2021 • edited Loading

codecov-commenter commented Jul 29, 2021 • edited Loading

Codecov Report

jianjuns left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

luolanzone commented Aug 10, 2021

luolanzone commented Aug 10, 2021

luolanzone commented Aug 10, 2021

luolanzone commented Aug 11, 2021

jianjuns commented Aug 11, 2021

luolanzone commented Aug 11, 2021

luolanzone commented Aug 16, 2021

jianjuns commented Aug 16, 2021

antoninbas commented Aug 16, 2021

luolanzone commented Aug 17, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jianjuns left a comment

Choose a reason for hiding this comment

luolanzone commented Aug 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antoninbas commented Aug 19, 2021

luolanzone commented Jul 29, 2021 •

edited

Loading

codecov-commenter commented Jul 29, 2021 •

edited

Loading