Integration tests #151
Merged
Integration tests #151
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Honny1
force-pushed
the
integration-tests
branch
from
ca42126 to
077961c
Compare
Honny1
force-pushed
the
integration-tests
branch
2 times, most recently
from
366e9a7 to
17514e6
Compare
evgenyz
reviewed
Apr 11, 2023
plans/integration.fmf
Outdated
| - name: Generate ARF files | ||
| how: shell | ||
| script: | ||
| - ./generate_arf.sh ssg no fedora ${TMT_PLAN_DATA}/arf.xml |
Contributor
There was a problem hiding this comment.
Can we parametrize the product?
Contributor
There was a problem hiding this comment.
Or it should be marked as only executable in Fedora.
Member
Author
There was a problem hiding this comment.
Yes, it can be parameterized, but how do we determine which product to use?
Contributor
There was a problem hiding this comment.
FMF must somehow provide the name of the platform is it trying to execute the test on. We might need to map FMF name to our product name, though.
Member
Author
There was a problem hiding this comment.
I have an idea of how to map the distro value to product value.
Honny1
force-pushed
the
integration-tests
branch
from
17514e6 to
5294de1
Compare
Honny1
force-pushed
the
integration-tests
branch
from
5294de1 to
73d4371
Compare
Honny1
force-pushed
the
integration-tests
branch
from
73d4371 to
279de0f
Compare
Member
Author
|
Changes:
|
evgenyz
approved these changes
Apr 13, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR creates a set of integration tests using the
tmttool, a script that generates ARF reports with different configurations, modifies the smoke test to use the given ARF file, and creates a GitHub action to run these integration tests weekly.The tests use ARF results generated with the latest content available as an SSG package and content from the ComplianceAsCode/content repository.
Some integration tests could have failed because the test used the latest released openscap-report package. The goal is to check the released package of openscap-repot is behave right with changes in content in ComplianceAsCode/content and with shipped content in the SSG package.
The integration test could be executed in Tesitng Farm with PackIt. It would need some adjustments to the content product. In Testing Farm could be provided RPM package from changes in PR. Execution of this test in Testing Farm takes about 30min.
That's why I decided to run the integration test scheduled with GitHub Actions. I think using these tests for changes in PR would be pointless.