e2e: Run on other architectures #799

gquillar · 2022-02-28T16:23:17Z

To support other architectures, we should be able to generate the e2e test content images on other architectures and run the e2e tests using these images.
To do that, this patch gathers all the e2e test content images to CO repo quay.io/compliance-operator/test-broken-content and reference these images by tags instead of by digest.
New rules are added in the Makefile to build theses images and the existing rules are fixed to be able to run e2e tests using images from a personal repository.

With this patch, e2e test content images can be built and used to run e2e tests on ppc64le or s390x arch with the following sequence:

export IMAGE_REPO=quay.io/<personal_repo>

git clone https://github.com/ComplianceAsCode/content
cd content
podman build -f Dockerfiles/ocp4_content -t ${IMAGE_REPO}/ocp4 .
podman push ${IMAGE_REPO}/ocp4
cd ..

git clone https://github.com/openshift/compliance-operator
cd compliance-operator
make push-openscap-image
make push
make push-index
export E2E_BROKEN_CONTENT_IMAGE_PATH=$IMAGE_REPO/test-broken-content
export E2E_CONTENT_IMAGE_PATH=$IMAGE_REPO/ocp4
export E2E_USE_DEFAULT_IMAGES=true
make push-e2e-content
make e2e

openshift-ci · 2022-02-28T16:24:00Z

Hi @gquillar. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jhrozek · 2022-03-01T13:24:57Z

/ok-to-test

gquillar · 2022-03-01T14:14:04Z

/retest

gquillar · 2022-03-02T17:33:30Z

/test e2e-aws

jhrozek · 2022-03-03T13:56:57Z

Did you run the e2e tests in your environment? Did they work OK?
Can you rebase the patches atop origin/master and push again to get rid of the merge commit? For some reason I'm having trouble checking out the PR locally..

…ce-operator into e2e_multi-arch

gquillar · 2022-03-03T14:50:19Z

@jhrozek, e2e tests worked fine in my environment. I've rebased the patches atop origin/master and pushed again.

gquillar · 2022-03-03T14:53:01Z

I didn't understand why a lot of CI tests failed with scan timeouts although they worked fine in my environment.
I don't know if the rebase may have changed that. Let's see...

jhrozek · 2022-03-03T15:12:05Z

On Thu, Mar 03, 2022 at 06:53:17AM -0800, Gilles Quillard wrote: I didn't understand why a lot of CI tests failed with scan timeouts although they worked fine in my environment. I don't know if the rebase may have changed that. Let's see...

Yeah, I wanted to run the tests locally in my cluster to see what's going on, but had issues applying the PR. Now I can at least check it out, thanks.

jhrozek · 2022-03-03T16:07:10Z

Makefile

@@ -37,8 +38,10 @@ RELATED_IMAGE_OPENSCAP_TAG?=1.3.5
 # Image path to use. Set this if you want to use a specific path for building
 # or your e2e tests. This is overwritten if we bulid the image and push it to
 # the cluster or if we're on CI.
+DEFAULT_IMAGE_OPERATOR_PATH=quay.io/compliance-operator/$(APP_NAME):latest
+DEFAULT_IMAGE_OPENSCAP_PATH=quay.io/compliance-operator/$(RELATED_IMAGE_OPENSCAP_NAME):$(RELATED_IMAGE_OPENSCAP_TAG)


Do I see correctly that this variable definition is twice in the Makefile?

This is not the same variable: one for operator and one for openscap

Oups I see, DEFAULT_IMAGE_OPENSCAP_PATH declaration at line 2 was an extra paste. Need to be removed

jhrozek · 2022-03-03T16:09:56Z

Makefile

@@ -300,24 +308,24 @@ test-benchmark: ## Run the benchmark tests -- Note that this can only be ran for
 e2e: namespace tear-down operator-sdk image-to-cluster openshift-user deploy-crds ## Run the end-to-end tests
 	@echo "WARNING: This will temporarily modify deploy/operator.yaml"
 	@echo "Replacing workload references in deploy/operator.yaml"
-	@$(SED) 's%$(IMAGE_REPO)/$(RELATED_IMAGE_OPENSCAP_NAME):$(RELATED_IMAGE_OPENSCAP_TAG)%$(RELATED_IMAGE_OPENSCAP_PATH)%' deploy/operator.yaml
-	@$(SED) 's%$(IMAGE_REPO)/$(APP_NAME):latest%$(RELATED_IMAGE_OPERATOR_PATH)%' deploy/operator.yaml
+	@$(SED) 's%$(DEFAULT_IMAGE_OPENSCAP_PATH)%$(RELATED_IMAGE_OPENSCAP_PATH):$(RELATED_IMAGE_OPENSCAP_TAG)%' deploy/operator.yaml


Hmm, I wonder if this is the reason for failure I'm seeing..while running the e2e tests locally I see the tag twice in the scanner pods:

Warning InspectFailed 5s (x4 over 22s) kubelet Failed to apply default image tag "image-registry.openshift-image-registry.svc:5000/openshift/openscap-ocp:1.3.5:1.3.5": couldn't parse image reference "image-registry.openshift-image-registry.svc:5000/openshift/openscap-ocp:1.3.5:1.3.5": invalid reference format

I guess this might also be the reason for the CI failing. I'm sorry I didn't have enough time to dive deeper into why this is failing..

gquillar · 2022-03-04T19:14:26Z

One test failed:
--- FAIL: TestE2E/Parallel_tests/TestSuiteWithContentThatDoesNotMatch (1185.01s)
helpers.go:263: expecting NOT-APPLICABLE got NON-COMPLIANT

gquillar · 2022-03-04T20:49:07Z

/test e2e-aws

Vincent056 · 2022-03-05T04:57:48Z

@gquillar the e2e fail might be unrelated, I added this PR to see if it fixes the issue #803

gquillar · 2022-03-05T07:03:21Z

@Vincent056 Thanks to point out this timeout issue. The log confirms that the 20mn timeout is reached due too many parallel tests within this delay.

gquillar · 2022-03-08T15:38:14Z

/test e2e-aws

jhrozek · 2022-03-09T20:35:04Z

Thank you for your patience. There is only one test failing now, which is TestE2E/Parallel_tests/TestSuiteWithContentThatDoesNotMatch. We fixed that test using the old hardcoded images this week (this is why I had the delay in review as well). Can I ask you to add the following content image for the broken_os_detection image in the test? Then I think everything should be good to go.

I uploaded the file to:
https://jhrozek.fedorapeople.org/ssg-rhcos4-ds.xml

gquillar · 2022-03-10T09:42:11Z

/test e2e-aws

gquillar · 2022-03-10T12:39:32Z

/test e2e-aws

jhrozek · 2022-03-10T21:51:40Z

Just going to kick the e2e test, the one that failed is occasionally flaky (and it's my fault, I added it recently)
/retest

openshift-ci · 2022-03-10T23:39:00Z

@gquillar: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

gquillar · 2022-03-11T09:22:22Z

Thanks @jhrozek, all tests are now passed. Waiting for approved, lgtm labels.

jhrozek

/lgtm

openshift-ci · 2022-03-14T14:45:41Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gquillar, jhrozek

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jhrozek]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

e2e: Run on other architectures

8aa8aa2

openshift-ci bot requested review from rhmdnd and xiaojiey February 28, 2022 16:23

openshift-ci bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Feb 28, 2022

openshift-ci bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 1, 2022

gquillar added 2 commits March 1, 2022 06:10

fix test failure

8b069ca

Merge branch 'openshift:master' into e2e_multi-arch

dfc27a5

gquillar added 3 commits March 3, 2022 06:44

e2e: Run on other architectures

4d96f27

fix test failure

3b9e897

Merge branch 'e2e_multi-arch' of https://github.com/gquillar/complian…

892c886

…ce-operator into e2e_multi-arch

jhrozek reviewed Mar 3, 2022

View reviewed changes

fix tag usage for openscap-ocp

48cd821

add push of latest tag for test-broken-content image

2fb49f6

change profile bundle definition for broken_os_detection image

214fa7a

jhrozek approved these changes Mar 14, 2022

View reviewed changes

openshift-ci bot assigned jhrozek Mar 14, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Mar 14, 2022

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 14, 2022

openshift-merge-robot merged commit b536b36 into openshift:master Mar 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

e2e: Run on other architectures #799

e2e: Run on other architectures #799

gquillar commented Feb 28, 2022

openshift-ci bot commented Feb 28, 2022

jhrozek commented Mar 1, 2022

gquillar commented Mar 1, 2022

gquillar commented Mar 2, 2022

jhrozek commented Mar 3, 2022

gquillar commented Mar 3, 2022

gquillar commented Mar 3, 2022

jhrozek commented Mar 3, 2022 via email

jhrozek Mar 3, 2022

gquillar Mar 3, 2022

gquillar Mar 3, 2022

jhrozek Mar 3, 2022

gquillar commented Mar 4, 2022

gquillar commented Mar 4, 2022

Vincent056 commented Mar 5, 2022 •

edited

Loading

gquillar commented Mar 5, 2022

gquillar commented Mar 8, 2022

jhrozek commented Mar 9, 2022

gquillar commented Mar 10, 2022

gquillar commented Mar 10, 2022

jhrozek commented Mar 10, 2022

openshift-ci bot commented Mar 10, 2022

gquillar commented Mar 11, 2022

jhrozek left a comment

openshift-ci bot commented Mar 14, 2022

e2e: Run on other architectures #799

e2e: Run on other architectures #799

Conversation

gquillar commented Feb 28, 2022

openshift-ci bot commented Feb 28, 2022

jhrozek commented Mar 1, 2022

gquillar commented Mar 1, 2022

gquillar commented Mar 2, 2022

jhrozek commented Mar 3, 2022

gquillar commented Mar 3, 2022

gquillar commented Mar 3, 2022

jhrozek commented Mar 3, 2022 via email

jhrozek Mar 3, 2022

Choose a reason for hiding this comment

gquillar Mar 3, 2022

Choose a reason for hiding this comment

gquillar Mar 3, 2022

Choose a reason for hiding this comment

jhrozek Mar 3, 2022

Choose a reason for hiding this comment

gquillar commented Mar 4, 2022

gquillar commented Mar 4, 2022

Vincent056 commented Mar 5, 2022 • edited Loading

gquillar commented Mar 5, 2022

gquillar commented Mar 8, 2022

jhrozek commented Mar 9, 2022

gquillar commented Mar 10, 2022

gquillar commented Mar 10, 2022

jhrozek commented Mar 10, 2022

openshift-ci bot commented Mar 10, 2022

gquillar commented Mar 11, 2022

jhrozek left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Mar 14, 2022

Vincent056 commented Mar 5, 2022 •

edited

Loading