[Bug]: Inconsistent Behavior of AWS Secret ID's

[wschult23]

Terraform Core Version

1.4.5

AWS Provider Version

5.4.0

Affected Resource(s)

aws_secretsmanager_secret
data.aws_secretsmanager_secret

Expected Behavior

The usage of the 'id' attribute should be consistent.

Actual Behavior

The resource changes the meaning of "id" depending of the lifecycle of the aws_secretsmanager_secret

Relevant Error/Panic Output Snippet

╷
│ Error: "arn" (test-secret) is an invalid ARN: arn: invalid prefix
│ 
│   with data.aws_secretsmanager_secret.test,
│   on test.tf line 6, in data "aws_secretsmanager_secret" "test":
│    6:   arn=aws_secretsmanager_secret.test.id
│ 
╵

Terraform Configuration Files

resource "aws_secretsmanager_secret" "test" {
  name="test-secret"
}

data "aws_secretsmanager_secret" "test" {
  arn=aws_secretsmanager_secret.test.id
}

output "test" {
  value = data.aws_secretsmanager_secret.test.name
}

Steps to Reproduce

$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_secretsmanager_secret.test will be read during apply
  # (config refers to values not yet known)
 <= data "aws_secretsmanager_secret" "test" {
      + arn         = (known after apply)
      + description = (known after apply)
      + id          = (known after apply)
      + kms_key_id  = (known after apply)
      + name        = (known after apply)
      + policy      = (known after apply)
      + tags        = (known after apply)
    }

  # aws_secretsmanager_secret.test will be created
  + resource "aws_secretsmanager_secret" "test" {
      + arn                            = (known after apply)
      + force_overwrite_replica_secret = false
      + id                             = (known after apply)
      + name                           = "test-secret"
      + name_prefix                    = (known after apply)
      + policy                         = (known after apply)
      + recovery_window_in_days        = 30
      + tags_all                       = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Changes to Outputs:
  + test = (known after apply)

Do you want to perform these actions in workspace "prod"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

aws_secretsmanager_secret.test: Creating...
aws_secretsmanager_secret.test: Creation complete after 0s [id=arn:aws:secretsmanager:eu-central-1:299093934558:secret:test-secret-GOXNJE]
data.aws_secretsmanager_secret.test: Reading...
data.aws_secretsmanager_secret.test: Read complete after 0s [id=arn:aws:secretsmanager:eu-central-1:299093934558:secret:test-secret-GOXNJE]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Outputs:

test = "test-secret"

Now delete the secret:

terraform destroy

Resurrect it and try to import it manually:

$ aws secretsmanager restore-secret --secret-id test-secret 
{
    "ARN": "arn:aws:secretsmanager:eu-central-1:XXXXXXXX:secret:test-secret-GOXNJE",
    "Name": "test-secret"
}

When you now try to reimport the secret, the following error occurs on the data source:

$ terraform import aws_secretsmanager_secret.test test-secret
aws_secretsmanager_secret.test: Importing from ID "test-secret"...
aws_secretsmanager_secret.test: Import prepared!
  Prepared aws_secretsmanager_secret for import
aws_secretsmanager_secret.test: Refreshing state... [id=test-secret]
╷
│ Error: "arn" (test-secret) is an invalid ARN: arn: invalid prefix
│
│   with data.aws_secretsmanager_secret.test,
│   on /Users/wolle/Projects/terraform/4flow-terraform-aws/confluence/test.tf line 6, in data "aws_secretsmanager_secret" "test":
│    6:   arn=aws_secretsmanager_secret.test.id
│
╵

You can workaround this by fixing line number 6 to:

  arn=aws_secretsmanager_secret.test.arn

Now you can apply again, but you cannot go back to the code that was originally working.

Debug Output

No response

Panic Output

No response

Important Factoids

It seems that the root cause is that the Amazon API accepts secret names instead of ARN's as long as they are unique. When you delete and restore a secret, it's probably no longer the case.

References

https://docs.aws.amazon.com/cli/latest/reference/secretsmanager/index.html

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @wschult23 👋 Thank you for taking the time to raise this! This is an interesting corner case that seems to be caused by the underlying read function (secretsmanager.DescribeSecret) accepting either the ARN or the secret name.

In normal operation, the id attribute of the aws_secretsmanager_secret resource is set to the ARN of the secret. Because of this, the import documentation indicates that the ARN should be used during import.

Normally, providing something other than what's expected during import (and thus setting id to the incorrect value) results in errors in Terraform, as attempting to read with the incorrect value causes the upstream API to return an error. In this case, since the read function accepts the secret name or the ARN, when you imported the resource using the secret name, Terraform was still able to read the resource and import it into the state (albeit with the incorrect value set for the id).

In the short term, the fix for this is to import the aws_secretsmanager_secret resource using the appropriate value -- the ARN of the existing resource. That said, I'd like to leave this issue open so that we can take a look at the idea of adding validation to ensure that the id value gets set to the ARN, rather than the secret name.

[wschult23]

Hey @justinretzolk,
I know that you can import it with the ARN of the existing resource. But as mentioned in the description this will break the code either. The problem is that data.aws_secretsmanager_secret should not accept strings other than ARN's to prevent this issue.

Best,

Wolfgang.

[justinretzolk]

Hey @wschult23 👋 In this case, data.aws_secretsmanager_secret is not accepting a value other than the ARN, that's why this error is thrown:

│ Error: "arn" (test-secret) is an invalid ARN: arn: invalid prefix
│
│   with data.aws_secretsmanager_secret.test,

The issue in this case is incorrectly importing the aws_secretsmanager_secret resource into the state with the ID set to the name of the secret rather than the ARN of the secret, thus causing aws_secretsmanager_secret.test.id to be the name of the secret rather than the ARN of the secret. Due to this incorrect import, when you interpolate aws_secretsmanager_secret.test.id in the arn argument of data.aws_secretsmanager_secret.test, the name of the secret is used rather than the ARN (and thus data.aws_secretsmanager_secret.test throws the validation error that you initially reported).

[wschult23]

Hey @justinretzolk,
Totally agree that it is incorrect to use the name instead of the id - but it is possible under certain conditions. And this is my point here. We've found this issue in our code that was running for years and then broke randomly. And it cost us nerves to find the cause. You should prevent the users from misusing the id. I know that the root cause is the Amazon API here but the provider could have an additional check that the id is a ARN and not a name.