Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test(s): TestAccAccessContextManager/access_levels #20000

Open
SarahFrench opened this issue Oct 24, 2024 · 3 comments
Open

Failing test(s): TestAccAccessContextManager/access_levels #20000

SarahFrench opened this issue Oct 24, 2024 · 3 comments
Labels
forward/linked service/accesscontextmanager test-failure test-failure-10
Milestone
Goals

Comments

@SarahFrench
Copy link
Member

SarahFrench commented Oct 24, 2024 •
edited by modular-magician
Loading

Impacted tests

  • TestAccAccessContextManager

Affected Resource(s)

  • google_access_context_manager_access_levels

Failure rates

  • 27% (5 failed out of last 18 runs) as of 2024-10-24

Message(s)

------- Stdout: -------
=== RUN   TestAccAccessContextManager
=== RUN   TestAccAccessContextManager/service_perimeter_dry_run_ingress_policy
=== RUN   TestAccAccessContextManager/service_perimeters
=== RUN   TestAccAccessContextManager/gcp_user_access_binding
=== PAUSE TestAccAccessContextManager/gcp_user_access_binding
=== RUN   TestAccAccessContextManager/authorized_orgs_desc
=== RUN   TestAccAccessContextManager/service_perimeter
=== RUN   TestAccAccessContextManager/service_perimeter_dry_run_resource
=== RUN   TestAccAccessContextManager/access_level
=== RUN   TestAccAccessContextManager/service_perimeter_dry_run_egress_policy
=== RUN   TestAccAccessContextManager/service_perimeter_update
=== RUN   TestAccAccessContextManager/access_level_custom
=== RUN   TestAccAccessContextManager/service_perimeter_egress_policy
=== RUN   TestAccAccessContextManager/service_perimeter_resource
=== RUN   TestAccAccessContextManager/access_level_full
=== RUN   TestAccAccessContextManager/service_perimeter_ingress_policy
=== RUN   TestAccAccessContextManager/access_policy
=== RUN   TestAccAccessContextManager/access_policy_scoped
=== RUN   TestAccAccessContextManager/access_levels
    resource_access_context_manager_access_levels_test.go:24: Step 3/6 error: Error running apply: exit status 1
        Error: Error creating AccessLevels: googleapi: Error 403: Permission 'compute.networks.get' denied on resource '//compute.googleapis.com/projects/PROJECT_ID/global/networks/test-vpc-nmhc0pq8ve' (or it may not exist)
          with google_access_context_manager_access_levels.test-access,
          on terraform_plugin_test.tf line 11, in resource "google_access_context_manager_access_levels" "test-access":
          11: resource "google_access_context_manager_access_levels" "test-access" {
=== RUN   TestAccAccessContextManager/access_level_condition
=== CONT  TestAccAccessContextManager/gcp_user_access_binding
--- FAIL: TestAccAccessContextManager (780.85s)
    --- PASS: TestAccAccessContextManager/service_perimeter_dry_run_ingress_policy (21.16s)
    --- PASS: TestAccAccessContextManager/service_perimeters (33.98s)
    --- PASS: TestAccAccessContextManager/authorized_orgs_desc (129.57s)
    --- PASS: TestAccAccessContextManager/service_perimeter (10.54s)
    --- PASS: TestAccAccessContextManager/service_perimeter_dry_run_resource (22.56s)
    --- PASS: TestAccAccessContextManager/access_level (132.35s)
    --- PASS: TestAccAccessContextManager/service_perimeter_dry_run_egress_policy (18.70s)
    --- PASS: TestAccAccessContextManager/service_perimeter_update (49.83s)
    --- PASS: TestAccAccessContextManager/access_level_custom (10.55s)
    --- PASS: TestAccAccessContextManager/service_perimeter_egress_policy (19.02s)
    --- PASS: TestAccAccessContextManager/service_perimeter_resource (22.78s)
    --- PASS: TestAccAccessContextManager/access_level_full (10.10s)
    --- PASS: TestAccAccessContextManager/service_perimeter_ingress_policy (18.11s)
    --- PASS: TestAccAccessContextManager/access_policy (15.41s)
    --- PASS: TestAccAccessContextManager/access_policy_scoped (11.06s)
    --- FAIL: TestAccAccessContextManager/access_levels (119.31s)
    --- PASS: TestAccAccessContextManager/access_level_condition (135.82s)
    --- PASS: TestAccAccessContextManager/gcp_user_access_binding (23.81s)
FAIL



Copy to clipboard
------- Stdout: -------
=== RUN   TestAccAccessContextManager
=== RUN   TestAccAccessContextManager/service_perimeter_dry_run_ingress_policy
=== RUN   TestAccAccessContextManager/service_perimeters
=== RUN   TestAccAccessContextManager/access_policy_scoped
=== RUN   TestAccAccessContextManager/service_perimeter_dry_run_resource
=== RUN   TestAccAccessContextManager/service_perimeter_egress_policy
=== RUN   TestAccAccessContextManager/service_perimeter_ingress_policy
=== RUN   TestAccAccessContextManager/gcp_user_access_binding
=== PAUSE TestAccAccessContextManager/gcp_user_access_binding
=== RUN   TestAccAccessContextManager/authorized_orgs_desc
    resource_access_context_manager_authorized_orgs_desc_test.go:24: Step 1/2 error: Error running apply: exit status 1
        Error: Provider produced inconsistent result after apply
        When applying changes to
        google_access_context_manager_authorized_orgs_desc.authorized-orgs-desc,
        provider "provider[\"registry.terraform.io/hashicorp/google\"]" produced an
        unexpected new value: Root object was present, but now absent.
        This is a bug in the provider, which should be reported in the provider's own
        issue tracker.
=== RUN   TestAccAccessContextManager/access_level_custom
=== RUN   TestAccAccessContextManager/access_levels
=== RUN   TestAccAccessContextManager/access_level_condition
=== RUN   TestAccAccessContextManager/access_policy
=== RUN   TestAccAccessContextManager/service_perimeter_update
=== RUN   TestAccAccessContextManager/service_perimeter_resource
=== RUN   TestAccAccessContextManager/access_level_full
=== RUN   TestAccAccessContextManager/service_perimeter
=== RUN   TestAccAccessContextManager/access_level
=== RUN   TestAccAccessContextManager/service_perimeter_dry_run_egress_policy
=== CONT  TestAccAccessContextManager/gcp_user_access_binding
--- FAIL: TestAccAccessContextManager (808.88s)
    --- PASS: TestAccAccessContextManager/service_perimeter_dry_run_ingress_policy (21.28s)
    --- PASS: TestAccAccessContextManager/service_perimeters (33.09s)
    --- PASS: TestAccAccessContextManager/access_policy_scoped (11.85s)
    --- PASS: TestAccAccessContextManager/service_perimeter_dry_run_resource (21.62s)
    --- PASS: TestAccAccessContextManager/service_perimeter_egress_policy (51.17s)
    --- PASS: TestAccAccessContextManager/service_perimeter_ingress_policy (19.12s)
    --- FAIL: TestAccAccessContextManager/authorized_orgs_desc (125.05s)
    --- PASS: TestAccAccessContextManager/access_level_custom (11.02s)
    --- PASS: TestAccAccessContextManager/access_levels (133.13s)
    --- PASS: TestAccAccessContextManager/access_level_condition (124.73s)
    --- PASS: TestAccAccessContextManager/access_policy (16.25s)
    --- PASS: TestAccAccessContextManager/service_perimeter_update (43.88s)
    --- PASS: TestAccAccessContextManager/service_perimeter_resource (22.08s)
    --- PASS: TestAccAccessContextManager/access_level_full (11.24s)
    --- PASS: TestAccAccessContextManager/service_perimeter (11.11s)
    --- PASS: TestAccAccessContextManager/access_level (131.99s)
    --- PASS: TestAccAccessContextManager/service_perimeter_dry_run_egress_policy (20.29s)
    --- PASS: TestAccAccessContextManager/gcp_user_access_binding (23.74s)
FAIL

------- Stdout: -------
=== RUN   TestAccAccessContextManager
=== RUN   TestAccAccessContextManager/service_perimeter_dry_run_egress_policy
=== RUN   TestAccAccessContextManager/access_level
    resource_access_context_manager_access_level_test.go:25: Step 3/4 error: After applying this test step and performing a `terraform refresh`, the plan was not empty.
        stdout
        Terraform used the selected providers to generate the following execution
        plan. Resource actions are indicated with the following symbols:
          + create
        Terraform will perform the following actions:
          # google_access_context_manager_access_level.test-access will be created
          + resource "google_access_context_manager_access_level" "test-access" {
              + description = "hello"
              + id          = (known after apply)
              + name        = (known after apply)
              + parent      = (known after apply)
              + title       = "level"
              + basic {
                  + combining_function = "OR"
                  + conditions {
                      + ip_subnetworks = [
                          + "192.0.2.0/24",
                        ]
                    }
                }
            }
          # google_access_context_manager_access_policy.test-access will be created
          + resource "google_access_context_manager_access_policy" "test-access" {
              + create_time = (known after apply)
              + id          = (known after apply)
              + name        = (known after apply)
              + parent      = "organizations/1234567890"
              + title       = "my new policy"
              + update_time = (known after apply)
            }
        Plan: 2 to add, 0 to change, 0 to destroy.
=== RUN   TestAccAccessContextManager/access_level_full
=== RUN   TestAccAccessContextManager/service_perimeter
=== RUN   TestAccAccessContextManager/access_policy
=== RUN   TestAccAccessContextManager/access_policy_scoped
=== RUN   TestAccAccessContextManager/service_perimeter_ingress_policy
=== RUN   TestAccAccessContextManager/service_perimeter_dry_run_ingress_policy
=== RUN   TestAccAccessContextManager/gcp_user_access_binding
=== PAUSE TestAccAccessContextManager/gcp_user_access_binding
=== RUN   TestAccAccessContextManager/authorized_orgs_desc
=== RUN   TestAccAccessContextManager/access_levels
=== RUN   TestAccAccessContextManager/access_level_condition
=== RUN   TestAccAccessContextManager/service_perimeter_dry_run_resource
=== RUN   TestAccAccessContextManager/access_level_custom
=== RUN   TestAccAccessContextManager/service_perimeter_egress_policy
=== RUN   TestAccAccessContextManager/service_perimeters
=== RUN   TestAccAccessContextManager/service_perimeter_update
=== RUN   TestAccAccessContextManager/service_perimeter_resource
=== CONT  TestAccAccessContextManager/gcp_user_access_binding
--- FAIL: TestAccAccessContextManager (807.57s)
    --- PASS: TestAccAccessContextManager/service_perimeter_dry_run_egress_policy (20.23s)
    --- FAIL: TestAccAccessContextManager/access_level (130.44s)
    --- PASS: TestAccAccessContextManager/access_level_full (10.57s)
    --- PASS: TestAccAccessContextManager/service_perimeter (12.57s)
    --- PASS: TestAccAccessContextManager/access_policy (15.55s)
    --- PASS: TestAccAccessContextManager/access_policy_scoped (11.39s)
    --- PASS: TestAccAccessContextManager/service_perimeter_ingress_policy (20.37s)
    --- PASS: TestAccAccessContextManager/service_perimeter_dry_run_ingress_policy (18.30s)
    --- PASS: TestAccAccessContextManager/authorized_orgs_desc (131.15s)
    --- PASS: TestAccAccessContextManager/access_levels (142.89s)
    --- PASS: TestAccAccessContextManager/access_level_condition (144.32s)
    --- PASS: TestAccAccessContextManager/service_perimeter_dry_run_resource (21.26s)
    --- PASS: TestAccAccessContextManager/access_level_custom (11.27s)
    --- PASS: TestAccAccessContextManager/service_perimeter_egress_policy (17.92s)
    --- PASS: TestAccAccessContextManager/service_perimeters (29.64s)
    --- PASS: TestAccAccessContextManager/service_perimeter_update (45.69s)
    --- PASS: TestAccAccessContextManager/service_perimeter_resource (24.03s)
    --- PASS: TestAccAccessContextManager/gcp_user_access_binding (23.22s)
FAIL

Nightly build test history

https://hashicorp.teamcity.com/test/-944881702515174777?currentProjectId=TerraformProviders_GoogleCloud_GOOGLE_NIGHTLYTESTS&expandTestHistoryChartSection=true

b/375371873
@github-actions github-actions bot added forward/review In review; remove label to forward service/accesscontextmanager labels Oct 24, 2024
@SarahFrench
Copy link
Member Author

Different sub tests are failing on different days, and the errors suggest different tests are interfering with each other.

@SarahFrench SarahFrench removed the forward/review In review; remove label to forward label Oct 24, 2024
@SarahFrench SarahFrench added this to the Goals milestone Oct 24, 2024
@Charlesleonius
Copy link

These tests are designed to run synchronously per test suite execution. Do multiple instances of the test suite sometimes get run at once? From example for two PRs being created at the same time?

@SarahFrench
Copy link
Member Author

There shouldn't be multiple instances in nightly tests. I was imagining that there could be some issues due to the tests rapidly editing the same things, e.g some degree of eventual consistency. Only looking at the debug logs would tell for sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
forward/linked service/accesscontextmanager test-failure test-failure-10
Projects
None yet
Milestone
Goals
Development

No branches or pull requests
3 participants
@Charlesleonius @SarahFrench @modular-magician