[wip] CI/CD: Add provider AuditRecords validation test #3548
+355
−204
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This [wip] change is to add a new test mode to the integration tests to allow validating a provider's AuditRecord rules. This exposes a new test mode flag in the integration test suite: -validateAuditRecs false (default): The tests execute according to established behavior. Providers' AuditRecords rules are used to skip broken tests true: The test case selection is inverted and normally working tests are skipped. Tests that are normally excluded by AuditRules are executed and success is treated as failure to highlight the change in API capabilities enable provider auditrecords validation workflow Update provider_auditrecords.yml register this workflow Update provider_auditrecords.yml Add INWX variables to workflow debug test just the special "validate" mode test the special integrationtest "ExpectTestFailure" mode test the "ExpectTestFailure" mode in regular integration test job refactor test workflows for reuse debug profile loading secrets have to be explicitly defined between calling and called wf add other provider secret references
Reduces the special test mode to a bool flag Removes 'ignore' option Clean up comments and rename variables for clarity
|
I'm leaving this up in draft for now, as I think there might be better ways to do accomplish the periodic check of the filtering rules. I'm still interested in feedback if this seems like the right approach |
|
I'm closing this with a plan to break the proposed GHA refactoring out into a different change, apart from reworking this with suggestions from Tom regarding the changes to the integration tests |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This [wip] change is to add a new test mode to the integration tests to allow validating a provider's AuditRecord rules.
This exposes a new test mode flag in the integration test suite
-auditrecords(bool)false (default): The tests execute according to established behavior.
Providers' AuditRecords rules are used to skip broken tests
true: The test case selection is inverted and normally working tests are
skipped. Tests that are normally excluded by AuditRules are executed and
success is treated as failure to highlight the change in API capabilities
The AuditRecords rules that each provider contains are checked during integration tests. If a rule is present that establishes the provider doesn't support some specific variation of record attributes, the integration test suite will skip tests that are expected to fail.
Over time, providers may add support for these records. This new test mode is meant to identify rules that can be removed from the DNSControl provider's AuditRecords function and allow users to use the newly supported records.