Set the latest google_secret_manager_secret_version to a specific value

[philip-harvey]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to the modular-magician user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If an issue is assigned to a user, that user is claiming responsibility for the issue. If an issue is assigned to hashibot, a community member has claimed the issue already.

Terraform Version

1.4.4
provider registry.terraform.io/hashicorp/google v4.63.1

Affected Resource(s)

google_secret_manager_secret_version

Terraform Configuration Files

resource "google_secret_manager_secret_version" "version" {
secret = google_secret_manager_secret.secret.id
secret_data = "secret data here"
}

Expected Behavior

After running an apply the value of the latest version of the secret should contain the contents of secret_data

Actual Behavior

Latest version does not contain the contents of secret_data

Steps to Reproduce

1. terraform apply
2. add a new version to the secret outside of terraform. e.g. via the console. Do not destroy the old version.
3. terraform apply

The apply on line 3 will show no changes, but it should have created a new version with the value of secret_data

Important Factoids

google_secret_manager_secret_version doesn't have any input value for version or a flag to ensure that the configured version is the latest.

[edwardmedia]

@philip-harvey I think this is by design as secret_data is marked as sensitive

[philip-harvey]

@philip-harvey I think this is by design as secret_data is marked as sensitive

Hi @edwardmedia. I'm not following, Terraform know what's in the secret_data, but doesn't actually put this into the GCP secret version.

[edwardmedia]

@philip-harvey I see what you meant. After you finished step 2, the terraform has no knowledge of it. It stays with the old version. Apply doesn't work as it keeps pulling the existing version. You need to import by providing the specific version. As you may see version is a readonly attribute which helps terraform to main the state.

terraform import google_secret_manager_secret_version.default projects/{{project}}/secrets/{{secret_id}}/versions/{{version}}

[philip-harvey]

Since there is no way to provide a version number google_secret_manager_secret_version should set the value in the latest version, but it doesn't, which is the cause of the bug I'm reporting. This is supposed to be desired state, so terraform should keep the state per the config, but currently it doesn't for secrets

[edwardmedia]

@philip-harvey do you mean set the value in the latest version should be done in terraform apply? Usually apply is to bring your real settings and state to match your config, not the other way around. To bring your state to match your real settings, import is the way to go.

@c2thorn what do you think?

[c2thorn]

@philip-harvey I believe the behavior you want is in the data source https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/secret_manager_secret_version, not the resource.

In the data source, you can omit the version field to always retrieve the latest.

With the resource, you are managing a single version that gets assigned based on the payload from your first apply in step 1. Versions added outside of TF shouldn't affect the version that this resource is managing.

[philip-harvey]

Datasource doesn't help here, Terraform should be setting the real world state to match what is in the config. i.e. I expect that the serect data will be what is in secret_data, but this doesn't work currently. If the secret_data is changed outside of Terrafrom then Terraform will never fix it. I don't want to use a data source to get the latest version of the secret, I want Terraform to set the secret data to be the value in secret_data, this is currently broken.

[brokenjacobs]

If admin-fatfingers comes in and adds a new version to the secret, 'version 2' as a new version on the secret, the latest version of the secret won't be the version that terraform created. And terraform provides no way to correct that from the plan-apply process. It can't even detect that the secret has a new 'latest' version.

It makes sense that ignore_changes doesn't work here because the versions in secret manager are immutable.

An additional resource that lets you determine what secret should be 'latest', would resolve this. Or a way to specify that you want a version created by terraform to be the latest. The logic would be a bit odd though.

Create new version (terraform) -> v1.
External user creates version -> v2
Apply runs and sees change, makes new version -> v3

Right now terraform will only create a new version if the old version gets deleted or 'changes' (Which isn't possible). Or you remove the state for the version from the statefile. You can change the metadata of a secret version (for instance disabling a secret) and terraform will re-enable. But the contents of the secret cannot change.

[c2thorn]

Datasource doesn't help here, Terraform should be setting the real world state to match what is in the config. i.e. I expect that the serect data will be what is in secret_data, but this doesn't work currently. If the secret_data is changed outside of Terrafrom then Terraform will never fix it. I don't want to use a data source to get the latest version of the secret, I want Terraform to set the secret data to be the value in secret_data, this is currently broken.

The API defines each version as a separate resource. By design, google_secret_manager_secret_version manages an individual version resource, not the latest version of a secret. There is no way to update the secret_data in a version, so the only way would be to create another new version as latest.

To enforce the latest version on the secret, a new Terraform-specific resource would likely have to be added as @brokenjacobs has suggested.
(Perhaps there is a way to get this functionality immediately by utilizing replace_triggered_by to cause Terraform to recreate the version when the latest version changes, but it's not coming to me right now).

IMO, it would be better if the parent secret resource had a version field that it could set so that Terraform could enforce it to the google_secret_manager_secret_version defined in config without creating a new one.

[c2thorn]

As I see it, the functionality desired would be a new separate feature (either a new field on google_secret_manager_secret_version or a new resource entirely). I'll flag this as an enhancement and our triage team will review it next week.

[philip-harvey]

Hence why I'm suggesting that google_secret_manager_secret_version takes an argument like "enforce_latest_version" that means it should ensure that the version it is managing is the latest version. I guess it would work to make a separate resource for this but it seems like overkill.
I don't think replace_triggered_by will help here since nothing in the resource changes when it's no longer the latest version.

[philip-harvey]

Thanks @c2thorn. I will say that these two new issues, this and #14490, are both less critical than #11652
Terraform shouldn't always disable all the previous versions, in the GUI the default is to do nothing with the old versions and often disable is a better option then destroy.

[dawidmalina]

I was also really surprised today when one of the secrets was manually modified that terraform did not detect any drift and didn't automatically fixed that situation. I agree that in most cases uses will expect that latest version is exactly what they have in the terraform state (this is what I saw in other cloud provider). In the current behaviour I have no trust that terraform is keeping my desired state.

[melinath]

Marking exempt because I believe we need to do some investigation here - @rileykarson feel free to remove if that's not correct.

[psvaiter]

I came across this issue and decided to do some experiments to understand the behaviors of the resource in different situations. I have two suggestions, if they are welcome:

• Create an attribute of type bool called detect_drift or force_latest (default false) that will lead to fetching the value of the latest enabled version and deciding if a new version should be added. The deletion policy would apply to both managed version and the latest version that is being "overwritten".
• Create an attribute of type bool called disable_all_past_versions (default false) that will have the same behavior of marking the checkbox in UI when adding a new version.

With this approach, the current behavior would not be changed, and those who want to keep Terraform away from messing up with future versions would still be supported.