Errors with `ImportStateVerify` test steps after `terraform-plugin-testing@v1.6.0` update

[ewbankkit]

Ever since the terraform-plugin-testing@v1.6.0 update a number of Terraform AWS Provider ImportStateVerify state verify test steps that previously passed have been failing.

For example:

=== RUN   TestAccAppMesh_serial/Route/grpcRouteEmptyMatch
    route_test.go:568: Step 2/2 error running import: ImportStateVerify attributes not equivalent. Difference is shown below. The - symbol indicates attributes missing after import.
          map[string]string{
        +   "spec.0.grpc_route.0.action.0.weighted_target.0.port": "0",
          }

The pattern I have observed is that these tests cases are all to do with values of attributes inside nested TypeSets that are themselves nested in other configuration blocks. The affected resource all use Terraform Plugin SDK v2.

For the example above (internal/service/appmesh/route.go):

				"grpc_route": {
					Type:          schema.TypeList,
					Optional:      true,
					MinItems:      0,
					MaxItems:      1,
					ConflictsWith: []string{"spec.0.http2_route", "spec.0.http_route", "spec.0.tcp_route"},
					Elem: &schema.Resource{
						Schema: map[string]*schema.Schema{
							"action": {
								Type:     schema.TypeList,
								Required: true,
								MinItems: 1,
								MaxItems: 1,
								Elem: &schema.Resource{
									Schema: map[string]*schema.Schema{
										"weighted_target": {
											Type:     schema.TypeSet,
											Required: true,
											MinItems: 1,
											MaxItems: 10,
											Elem: &schema.Resource{
												Schema: map[string]*schema.Schema{
													"port": {
														Type:         schema.TypeInt,
														Optional:     true,
														ValidateFunc: validation.IsPortNumber,
													},

Acceptance test case (testAccRoute_grpcRouteEmptyMatch) configuration (internal/service/appmesh/route_test.go):

resource "aws_appmesh_route" "test" {
  name                = "test"
  mesh_name           = aws_appmesh_mesh.test.id
  virtual_router_name = aws_appmesh_virtual_router.test.name

  spec {
    grpc_route {
      match {}

      action {
        weighted_target {
          virtual_node = aws_appmesh_virtual_node.test1.name
          weight       = 100
        }
      }
    }
  }
}

terraform-plugin-testing version

% go list -m github.com/hashicorp/terraform-plugin-testing/...
github.com/hashicorp/terraform-plugin-testing v1.6.0

References

Relates hashicorp/terraform-provider-aws#35054.
Relates hashicorp/terraform-provider-aws#35108.

[bendbennett]

Hey @ewbankkit 👋,

Following some discussion within the team, we're wondering whether you see the same issue in a "real world" terraform run. For instance, if you run a terraform apply, remove the state, then run a terraform import, do you see a plan difference as highlighted by the test that you illustrated? If not, then this may suggest the presence of a bug in SDK v2.

[ewbankkit]

@bendbennett I'll try the scenario you suggested -- I hazard a guess that this is not happening in real-life scenarios given that we have had no associated bug reports and the affected resources are heavily used.

[ewbankkit]

With this configuration

provider "aws" {}

resource "aws_appmesh_mesh" "test" {
  name = "ewbankkit-test"
}

resource "aws_appmesh_virtual_router" "test" {
  name      = "ewbankkit-test"
  mesh_name = aws_appmesh_mesh.test.id

  spec {
    listener {
      port_mapping {
        port     = 8080
        protocol = "grpc"
      }
    }
  }
}

resource "aws_appmesh_virtual_node" "test" {
  name      = "ewbankkit-test"
  mesh_name = aws_appmesh_mesh.test.id

  spec {
    listener {
      port_mapping {
        port     = 8080
        protocol = "grpc"
      }
    }

    service_discovery {
      dns {
        hostname = "test1.simpleapp.local"
      }
    }
  }
}

resource "aws_appmesh_route" "test" {
  name                = "ewbankkit-test"
  mesh_name           = aws_appmesh_mesh.test.id
  virtual_router_name = aws_appmesh_virtual_router.test.name

  spec {
    grpc_route {
      match {}

      action {
        weighted_target {
          virtual_node = aws_appmesh_virtual_node.test.name
          weight       = 100
        }
      }
    }
  }
}

Run terraform apply, then

% terraform plan
aws_appmesh_mesh.test: Refreshing state... [id=ewbankkit-test]
aws_appmesh_virtual_router.test: Refreshing state... [id=cc20c272-7781-48f4-9afa-5071cd10a16c]
aws_appmesh_virtual_node.test: Refreshing state... [id=899a37e7-1a16-473d-a32e-76cad32add62]
aws_appmesh_route.test: Refreshing state... [id=f9118122-592e-49fa-af24-9d38b4c7e92a]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_appmesh_mesh.test has been changed
  ~ resource "aws_appmesh_mesh" "test" {
        id                = "ewbankkit-test"
        name              = "ewbankkit-test"
      + tags              = {}
        # (6 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }
  # aws_appmesh_route.test has been changed
  ~ resource "aws_appmesh_route" "test" {
        id                  = "f9118122-592e-49fa-af24-9d38b4c7e92a"
        name                = "ewbankkit-test"
      + tags                = {}
        # (8 unchanged attributes hidden)

      ~ spec {
            # (1 unchanged attribute hidden)

          ~ grpc_route {
              ~ action {
                  + weighted_target {
                      + port         = 0
                      + virtual_node = "ewbankkit-test"
                      + weight       = 100
                    }
                  - weighted_target {
                      - virtual_node = "ewbankkit-test" -> null
                      - weight       = 100 -> null
                    }
                }

                # (1 unchanged block hidden)
            }
        }
    }
  # aws_appmesh_virtual_node.test has been changed
  ~ resource "aws_appmesh_virtual_node" "test" {
        id                = "899a37e7-1a16-473d-a32e-76cad32add62"
        name              = "ewbankkit-test"
      + tags              = {}
        # (7 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }
  # aws_appmesh_virtual_router.test has been changed
  ~ resource "aws_appmesh_virtual_router" "test" {
        id                = "cc20c272-7781-48f4-9afa-5071cd10a16c"
        name              = "ewbankkit-test"
      + tags              = {}
        # (7 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like to update the Terraform state to match, create and apply a refresh-only
plan:
  terraform apply -refresh-only

Remove aws_appmesh_route.test from state, then

% terraform plan
aws_appmesh_mesh.test: Refreshing state... [id=ewbankkit-test]
aws_appmesh_virtual_router.test: Refreshing state... [id=cc20c272-7781-48f4-9afa-5071cd10a16c]
aws_appmesh_virtual_node.test: Refreshing state... [id=899a37e7-1a16-473d-a32e-76cad32add62]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_appmesh_virtual_node.test has been changed
  ~ resource "aws_appmesh_virtual_node" "test" {
        id                = "899a37e7-1a16-473d-a32e-76cad32add62"
        name              = "ewbankkit-test"
      + tags              = {}
        # (7 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }
  # aws_appmesh_virtual_router.test has been changed
  ~ resource "aws_appmesh_virtual_router" "test" {
        id                = "cc20c272-7781-48f4-9afa-5071cd10a16c"
        name              = "ewbankkit-test"
      + tags              = {}
        # (7 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }
  # aws_appmesh_mesh.test has been changed
  ~ resource "aws_appmesh_mesh" "test" {
        id                = "ewbankkit-test"
        name              = "ewbankkit-test"
      + tags              = {}
        # (6 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # aws_appmesh_route.test will be created
  + resource "aws_appmesh_route" "test" {
      + arn                 = (known after apply)
      + created_date        = (known after apply)
      + id                  = (known after apply)
      + last_updated_date   = (known after apply)
      + mesh_name           = "ewbankkit-test"
      + mesh_owner          = (known after apply)
      + name                = "ewbankkit-test"
      + resource_owner      = (known after apply)
      + tags_all            = (known after apply)
      + virtual_router_name = "ewbankkit-test"

      + spec {
          + grpc_route {
              + action {
                  + weighted_target {
                      + virtual_node = "ewbankkit-test"
                      + weight       = 100
                    }
                }

              + match {
                }
            }
        }
    }

Plan: 1 to add, 0 to change, 0 to destroy.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

Import aws_appmesh_route.test

% terraform import aws_appmesh_route.test ewbankkit-test/ewbankkit-test/ewbankkit-test
aws_appmesh_route.test: Importing from ID "ewbankkit-test/ewbankkit-test/ewbankkit-test"...
aws_appmesh_route.test: Import prepared!
  Prepared aws_appmesh_route for import
aws_appmesh_route.test: Refreshing state... [id=f9118122-592e-49fa-af24-9d38b4c7e92a]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

then

% terraform plan
aws_appmesh_mesh.test: Refreshing state... [id=ewbankkit-test]
aws_appmesh_virtual_router.test: Refreshing state... [id=cc20c272-7781-48f4-9afa-5071cd10a16c]
aws_appmesh_virtual_node.test: Refreshing state... [id=899a37e7-1a16-473d-a32e-76cad32add62]
aws_appmesh_route.test: Refreshing state... [id=f9118122-592e-49fa-af24-9d38b4c7e92a]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the last "terraform apply":

  # aws_appmesh_virtual_node.test has been changed
  ~ resource "aws_appmesh_virtual_node" "test" {
        id                = "899a37e7-1a16-473d-a32e-76cad32add62"
        name              = "ewbankkit-test"
      + tags              = {}
        # (7 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }
  # aws_appmesh_virtual_router.test has been changed
  ~ resource "aws_appmesh_virtual_router" "test" {
        id                = "cc20c272-7781-48f4-9afa-5071cd10a16c"
        name              = "ewbankkit-test"
      + tags              = {}
        # (7 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }
  # aws_appmesh_mesh.test has been changed
  ~ resource "aws_appmesh_mesh" "test" {
        id                = "ewbankkit-test"
        name              = "ewbankkit-test"
      + tags              = {}
        # (6 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

Unless you have made equivalent changes to your configuration, or ignored the relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

No changes. Your infrastructure matches the configuration.

Your configuration already matches the changes detected above. If you'd like to update the Terraform state to match, create and apply a refresh-only
plan:
  terraform apply -refresh-only

[ewbankkit]

So the terraform plan straight after the initial terraform apply is showing the same behavior as the acceptance test case.

[bendbennett]

Thanks for checking that @ewbankkit. So looking at the output from terraform plan, following terraform import, it appears that there is no diff, and the state matches the plan. It seems that the output from the acceptance test differs from the "real world" and as you suspected, the behaviour of the test differs from the practitioner experience. As you have now confirmed that the real-life behaviour does not mirror that in the test, one option would be to use the ImportStateVerifyIgnore field on TestStep to work around this. It seems likely that diff that is highlighted in the acceptance test (i.e., spec.0.grpc_route.0.action.0.weighted_target.0.port": "0") could be an SDKv2 bug which would require further investigation on our part.

[ewbankkit]

Given all the known problems with nested TypeSets in Terraform Plugin SDK v2 (e.g. hashicorp/terraform-plugin-sdk#1248, hashicorp/terraform-plugin-sdk#783, hashicorp/terraform-plugin-sdk#652, hashicorp/terraform-plugin-sdk#588, ...) this is surely an upstream problem and not a problem with the testing library.
For now (until we can migrate the resources to Terraform Plugin Framework) I think ImportStateVerifyIgnore is the only answer.
Thanks.

[bendbennett]

Thanks for the feedback Kit.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.