Antrea L7 NetworkPolicies broken with latest Suricata version (v6.0.11) #4921
Comments
|
cc @tnqn @hongliangl @hongliangl would you mind looking into this since you have worked with the Suricata team before, for a different bug? |
antoninbas
added
area/network-policy
|
I discovered that Suricata's suricatasc tool is encountering issues when executing commands that involve multiple tenants in Suricata 6.0.11. This causes the tool to become unresponsive and leads to the L7NetworkPolicy not syncing correctly in Antrea. I have raised an issue regarding this problem. Here is the link https://redmine.openinfosecfoundation.org/issues/6027. |
|
I believe we have two workarounds:
|
|
It seems that the best thing to do for now would be to wait for a patch and for the 6.0.12 release. This should be considered a blocker for Antrea v1.12.0.
|
|
I think Suricata 6.0.x is typically released every two months according to its release history. However, version 6.0.11 was released just three weeks ago. This means that the next version (6.0.12) is still around five weeks away. Unfortunately, this will be a blocker for Antrea v1.12.0 regardless. I have another idea - could we manually create a new Antrea base image (v1.12.0) based on the existing v1.11.0 image? This way, we could update all software components except for Suricata. |
|
$ docker run antrea/base-ubuntu:antrea-v1.11 suricata -V
This is Suricata version 6.0.10 RELEASEHowever, this solution is a bit flimsy, and it means we may not be getting the latest software updates for other dependencies. We also have to make changes here and there to prevent the base image from being rebuilt. Building from source may be the best option. At least we will have the flexibility to pick the exact Suricata version we want to use, if the same issue comes up in the future. BTW, Ubuntu comes with Suricata 6.0.4 by default (main PPA, not OISF PPA). Would that version work for us? |
The latest release from Suricata could suffer from a bug, impacting Antrea features. When installing from PPA, we don't have flexibility when it comes to the Suricata version we ship with Antrea. Instead, we can install Suricata from source as part of our Docker build. The advantages are: * full control over the version we install. * a smaller Antrea image, as we do not need to install all of Suricata's dependencies / enable all its features. The disadvantages of building from source are: * CVEs in Suricata itself won't be detected by scanners (CVEs in Suricata's dependencies will). * While we have more customization options, we also have to manually keep track of Suricata's dependencies (and use correct build options). * We have to remember to update the Suricata version we build & install. For antrea-io#4921 This can also be treated as a temporary fix until Suricata 6.0.12 is released, at which point we could revert this change. Signed-off-by: Antonin Bas <abas@vmware.com>
@antoninbas Sorry, I should have sent the message that the image was updated by me manually on April 28th, and should have validated the new version first.
We don't have dependency on 6.0.11 except the bugfix, so it should work for us, but does UBI has its own source which uses a previous patch release too? @hongliangl I see the issue has been fixed and backported to 6.x: OISF/suricata@fe45258, could you check with Suricata team when it's expected to release 6.0.12? If it's before our v1.12.0, perhaps we could use some workaround. I still have the previous base-ubuntu version which contains suricata 6.0.10, it should continue using the cache if I force update the image) |
If there is a change in the base ubuntu:22.04 image, it should invalidate any cached image. It seems that the |
To avoid a known issue with Suricata 6.0.11. The main Ubuntu PPA ships Suircata 6.0.4. With this change, e2e tests for L7NetworkPolicy will stop failing. We do not "fix" the UBI build at the moment, but it will be taken care of before the Antrea v1.12 release. For antrea-io#4921 Signed-off-by: Antonin Bas <abas@vmware.com>
To avoid a known issue with Suricata 6.0.11. The main Ubuntu PPA ships Suircata 6.0.4. With this change, e2e tests for L7NetworkPolicy will stop failing. We do not "fix" the UBI build at the moment, but it will be taken care of before the Antrea v1.12 release. For #4921 Signed-off-by: Antonin Bas <abas@vmware.com>
|
@xliuxu Suricata v6.0.12 has been released: https://suricata.io/2023/05/09/suricata-6-0-12-released/ |
|
@antoninbas Sure. I will test it. |
Seems the issue has been fixed in 6.0.12. I will revert the previous PR. |
…rea-io#4933) To avoid a known issue with Suricata 6.0.11. The main Ubuntu PPA ships Suircata 6.0.4. With this change, e2e tests for L7NetworkPolicy will stop failing. We do not "fix" the UBI build at the moment, but it will be taken care of before the Antrea v1.12 release. For antrea-io#4921 Signed-off-by: Antonin Bas <abas@vmware.com>
Describe the bug
This is a screenshot of the latest runs for the Kind Github workflow, on the main branch:
(source: https://github.com/antrea-io/antrea/actions/workflows/kind.yml?query=branch%3Amain)
The last 2 workflows have failed for the same reason:
These are not flakes: The same test failures have happened for me 4 times in a row with recent PRs, and I have also been able to reproduce the failure locally.
To Reproduce
In a K8s cluster deploy Antrea with the L7NetworkPolicy Feature Gate enabled. Then run the
TestL7NetworkPolicye2e tests. If you have a Kind cluster, the command is as follows:go test -v -run=TestL7NetworkPolicy ./test/e2e/... -provider=kindVersions:
Latest Antrea (main branch). Antrea v1.11.1 works fine.
Additional context
After investigating, I am pretty confident that this is because of a Suricata version update:
Why is this happening now?
The Suricata PPA (https://launchpad.net/~oisf/+archive/ubuntu/suricata-6.0) was updated 2 weeks ago, yet the failures started happening recently (last 24 hours). Based on my investigation, this is because the Docker build was using a cached version of Antrea base images. Some change in the build chain (I am not sure what) caused the cached images to become stale, causing the latest Suricata version to be installed.
The text was updated successfully, but these errors were encountered: