Skip to content

Improve e2e testcase for RHCOS #1045

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jan 11, 2023
Merged

Improve e2e testcase for RHCOS #1045

merged 3 commits into from
Jan 11, 2023

Conversation

vikin91
Copy link
Contributor

@vikin91 vikin91 commented Jan 10, 2023

This is a follow-up to #1004.
In the test case, we need a package that has high probability of being free from vulnerabilities because we assert on it to be recognized by scanner and have exactly 0 vulnerabilities.

In this PR, I replace the previously used grep by tzdata guesstimating that the chance of finding a vulnerability in the latter is lower than in the former.

Tested

On CI and locally with:

  • make image deploy-local
  • Opening a port-forward to scanner:8443
  • Running go test -tags e2e -timeout=10s -count=1 -v -run ^TestGRPCGetRHCOSNodeVulnerabilities$ github.com/stackrox/scanner/e2etests

We claim that the tzdata package has lower chance of finding
vulnerabilities in it than grep.
@ghost
Copy link

ghost commented Jan 10, 2023

Images are ready for the commit at 9176389.

To use the images, use the tag 2.27.x-22-g9176389cd2.

@vikin91
Copy link
Contributor Author

vikin91 commented Jan 10, 2023

The modified test case passed in CI. All failures in the recent CI run are unrelated.

@msugakov
Copy link
Contributor

/retest

@RTann
Copy link
Collaborator

RTann commented Jan 10, 2023

I think another option is to add some flag to the test to indicate if you want to bother looking at the vuln data or not (or even just check for specific vulns). We have this precedent set in other parts of the e2e tests (see testcase_test.go)

@openshift-ci
Copy link

openshift-ci bot commented Jan 11, 2023

@vikin91: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/slim-e2e-tests 9176389 link false /test slim-e2e-tests
ci/prow/e2e-tests 9176389 link false /test e2e-tests

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@vikin91
Copy link
Contributor Author

vikin91 commented Jan 11, 2023

Thanks Ross! I see that I would need to introduce such flag like onlyCheckSpecifiedVulns bool on the vulnerability level - so each expected feature would need to have an indicator whether we assert on vulnerabilities using isSubset or equals.

I will give it a try a separate PR to keep this change small.

@vikin91 vikin91 merged commit 50e7c18 into master Jan 11, 2023
@vikin91 vikin91 deleted the piotr/improve-e2e-testcase branch January 11, 2023 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants