Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove use of google_kms_crypto_key_iam_binding resource in tests, to make tests stable in overnight testing #9621

Merged
merged 6 commits into from
Dec 12, 2023

Conversation

SarahFrench
Copy link
Contributor

@SarahFrench SarahFrench commented Dec 11, 2023

This PR will help address test failures like:

In this PR I remove google_kms_crypto_key_iam_binding from tests that use shared crypto keys. I have also removed that resource from tests that provision their own crypto keys to improve signal-to-noise when we need to solve this same problem again in future.

This PR stops these tests affecting shared crypto keys:

  • TestAccCloudFunctionsFunction_cmek (beta)
  • TestAccComputeInstance_confidentialHyperDiskBootDisk (beta)

Release Note Template for Downstream PRs (will be copied)


…red crypto key affected)

By removing this usage of `google_kms_crypto_key_iam_binding` I intend to make it easier to identify when acc tests affect shared resources that aren't provisioned by the test
@modular-magician

This comment was marked as outdated.

@modular-magician

This comment was marked as outdated.

@modular-magician

This comment was marked as outdated.

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 4 insertions(+), 8 deletions(-))
Terraform Beta: Diff ( 5 files changed, 31 insertions(+), 26 deletions(-))
TF OiCS: Diff ( 1 file changed, 2 insertions(+), 4 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 3279
Passed tests 2942
Skipped tests: 336
Affected tests: 1

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccCloudFunctionsFunction_cmek

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

$\textcolor{red}{\textsf{Tests failed during RECORDING mode:}}$
TestAccCloudFunctionsFunction_cmek[Error message] [Debug log]

$\textcolor{red}{\textsf{Please fix these to complete your PR.}}$
View the build log or the debug log for each test

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 4 insertions(+), 8 deletions(-))
Terraform Beta: Diff ( 5 files changed, 31 insertions(+), 26 deletions(-))
TF OiCS: Diff ( 1 file changed, 2 insertions(+), 4 deletions(-))

@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 3279
Passed tests 2942
Skipped tests: 336
Affected tests: 1

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccCloudFunctionsFunction_cmek

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccCloudFunctionsFunction_cmek[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{red}{\textsf{Tests failed when rerunning REPLAYING mode:}}$
TestAccCloudFunctionsFunction_cmek[Error message] [Debug log]

Tests failed due to non-determinism or randomness when the VCR replayed the response after the HTTP request was made.

Please fix these to complete your PR. If you believe these test failures to be incorrect or unrelated to your change, or if you have any questions, please raise the concern with your reviewer.


$\textcolor{red}{\textsf{Errors occurred during RECORDING mode. Please fix them to complete your PR.}}$
View the build log or the debug log for each test

@SarahFrench
Copy link
Contributor Author

The TestAccCloudFunctionsFunction_cmek tests is experiencing the Requested interaction not found error; skipping test in VCR

@modular-magician
Copy link
Collaborator

Hi there, I'm the Modular magician. I've detected the following information about your changes:

Diff report

Your PR generated some diffs in downstreams - here they are.

Terraform GA: Diff ( 2 files changed, 4 insertions(+), 8 deletions(-))
Terraform Beta: Diff ( 5 files changed, 33 insertions(+), 26 deletions(-))
TF OiCS: Diff ( 1 file changed, 2 insertions(+), 4 deletions(-))

@SarahFrench SarahFrench requested a review from c2thorn December 11, 2023 18:33
@modular-magician
Copy link
Collaborator

Tests analytics

Total tests: 3279
Passed tests 2940
Skipped tests: 338
Affected tests: 1

Action taken

Found 1 affected test(s) by replaying old test recordings. Starting RECORDING based on the most recent commit. Click here to see the affected tests
TestAccSpannerInstanceIamPolicy

Get to know how VCR tests work

@modular-magician
Copy link
Collaborator

$\textcolor{green}{\textsf{Tests passed during RECORDING mode:}}$
TestAccSpannerInstanceIamPolicy[Debug log]

Rerun these tests in REPLAYING mode to catch issues

$\textcolor{green}{\textsf{No issues found for passed tests after REPLAYING rerun.}}$


$\textcolor{green}{\textsf{All tests passed!}}$
View the build log or the debug log for each test

@SarahFrench
Copy link
Contributor Author

Thanks!

@SarahFrench SarahFrench merged commit b4195a6 into main Dec 12, 2023
12 checks passed
kapreus pushed a commit to kapreus/magic-modules that referenced this pull request Jan 2, 2024
…to make tests stable in overnight testing (GoogleCloudPlatform#9621)

* Remove `google_kms_crypto_key_iam_binding` resources that affect shared crypto keys

* Remove unnecessary use of `google_kms_crypto_key_iam_binding` (no shared crypto key affected)

By removing this usage of `google_kms_crypto_key_iam_binding` I intend to make it easier to identify when acc tests affect shared resources that aren't provisioned by the test

* Remove unnecessary use of `google_kms_crypto_key_iam_binding` (no shared crypto key affected)

* Fix call to config function in acc test

* Update mmv1/third_party/terraform/services/cloudfunctions/resource_cloudfunctions_function_test.go.erb

* Skip `TestAccCloudFunctionsFunction_cmek` in VCR
@SarahFrench SarahFrench deleted the remove-iam-bindings-crypto-keys branch February 6, 2024 11:01
balanaguharsha pushed a commit to balanaguharsha/magic-modules that referenced this pull request May 2, 2024
…to make tests stable in overnight testing (GoogleCloudPlatform#9621)

* Remove `google_kms_crypto_key_iam_binding` resources that affect shared crypto keys

* Remove unnecessary use of `google_kms_crypto_key_iam_binding` (no shared crypto key affected)

By removing this usage of `google_kms_crypto_key_iam_binding` I intend to make it easier to identify when acc tests affect shared resources that aren't provisioned by the test

* Remove unnecessary use of `google_kms_crypto_key_iam_binding` (no shared crypto key affected)

* Fix call to config function in acc test

* Update mmv1/third_party/terraform/services/cloudfunctions/resource_cloudfunctions_function_test.go.erb

* Skip `TestAccCloudFunctionsFunction_cmek` in VCR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants