Switching to managed master password and managed rotation not possible

[ghinch]

Description

I have a RDS database I created with this module, which has been using a password from the random_password resource, which I have then stored in a Secret Manager Secret separately. I want to switch this database to use the self-managed master password, and I want to manage the timing of the rotation of this. However, when making the changes and trying to apply the code, I get the following issue in the plan step:

│ Error: Invalid index
│ 
│   on .terraform/modules/db/modules/db_instance/main.tf line 217, in resource "aws_secretsmanager_secret_rotation" "this":
│  217:   secret_id          = aws_db_instance.this[0].master_user_secret[0].secret_arn
│     ├────────────────
│     │ aws_db_instance.this[0].master_user_secret is empty list of object
│ 
│ The given key does not identify an element in this collection value: the collection has no elements.

• ✋ I have searched the open/closed issues and my issue is not listed.

Versions

• Module version [Required]: 6.10.0

• Terraform version: 1.8.3

• Provider version(s):

• provider registry.terraform.io/hashicorp/aws v5.89.0
• provider registry.terraform.io/hashicorp/random v3.7.1

Reproduction Code [Required]

Initially created the database with this configuration:

module "db" {
  source  = "terraform-aws-modules/rds/aws"
  version = "6.10.0"
  
  identifier = "test-database"

  engine               = "postgres"
  engine_version       = "14.12"
  family               = "postgres14"
  major_engine_version = "14"
  instance_class       = "db.t4g.micro"

  allocated_storage     = 20
  max_allocated_storage = 100
  storage_type          = "gp3"
  storage_encrypted = true

  manage_master_user_password  = false
  password                     = random_password.master_password.result

  iam_database_authentication_enabled               = true
  db_name                                           = "amplifi"
  username                                          = "amplifi"
  port                                              = 5432

  multi_az               = false
  db_subnet_group_name   = "default"
  vpc_security_group_ids = ["sg-xxxxxxxxxxxxxxx"]

  maintenance_window              = "Sun:04:00-Sun:07:00"
  backup_window                   = "02:00-04:00"

  backup_retention_period = 7
  skip_final_snapshot     = true
  deletion_protection     = false

  apply_immediately = true

  create_db_parameter_group = true
}

Then modified the configuration as follows:

module "db" {
  source  = "terraform-aws-modules/rds/aws"
  version = "6.10.0"
  
  identifier = "test-database"

  engine               = "postgres"
  engine_version       = "14.12"
  family               = "postgres14"
  major_engine_version = "14"
  instance_class       = "db.t4g.micro"

  allocated_storage     = 20
  max_allocated_storage = 100
  storage_type          = "gp3"
  storage_encrypted = true

  manage_master_user_password                       = true
  manage_master_user_password_rotation              = true
  master_user_password_rotation_schedule_expression = "cron(0 22 ? * SAT *)"
  master_user_password_rotation_duration            = "1h"
  master_user_password_rotate_immediately           = false

  iam_database_authentication_enabled               = true
  db_name                                           = "amplifi"
  username                                          = "amplifi"
  port                                              = 5432

  multi_az               = false
  db_subnet_group_name   = "default"
  vpc_security_group_ids = ["sg-xxxxxxxxxxxxxxx"]

  maintenance_window              = "Sun:04:00-Sun:07:00"
  backup_window                   = "02:00-04:00"

  backup_retention_period = 7
  skip_final_snapshot     = true
  deletion_protection     = false

  apply_immediately = true

  create_db_parameter_group = true
}

Running terraform plan after changing the configuration as above yields the error above. If I set manage_master_user_password_rotation to false, the error is not raised. But I want to manage the password rotation timing.

Expected behavior

I should be able to make this configuration change and apply it in one go.

Actual behavior

The above error is raised at the plan stage.

Terminal Output Screenshot(s)

Additional context

[tiagovmvieira]

I am facing the same issue...

[tiagovmvieira]

@magreenbaum could you provide any help in here?

[tiagovmvieira]

I got it working, by using the following in the secret_id of the resource "aws_secretsmanger_secret_rotation"

secret_id = join("", aws_db_instance.postgres_rds[0].master_user_secret.*.secret_arn) as per suggested in here

But it looks it's a bug on the provider side, and not in the implementation of this rds module.

[magreenbaum]

I get the This is a bug in the provider... message even with this mentioned workaround.

For switching from password to managed password, looks like a two-step approach might also work. For me, I first set manage_master_user_password = true but keep manage_master_user_password_rotation = false. This will change the password though. Then once that changes to the managed password, then enable manage_master_user_password_rotation with the other related configs set as needed to change the frequency/timing. I did a brief test though, if you wanted to go this route, I would highly suggest testing it to ensure it works for your use case before running it in production.

[tiagovmvieira]

Thanks for your help in here @magreenbaum

I am going to follow your suggestion on two-step approach in this use case where I already have a password set up in my RDS instance. Going to perform my tests, before promoting it to production, so later I'll come back here with some feedback.

Again, thanks for your help!

[tiagovmvieira]

@magreenbaum -> I confirm, I also got the following error when trying the workaround secret_id = join("", aws_db_instance.postgres_rds[0].master_user_secret.*.secret_arn), so it really looks that for this usecase it's an error in the provider side

╷
│ Error: Provider produced inconsistent final plan
│ 
│ When expanding the plan for <resource_name> to include new values learned so far during apply, provider
│ "registry.terraform.io/hashicorp/aws" produced an invalid new value for .secret_id: was cty.StringVal(""), but now
│ cty.StringVal("<secret_arn>").
│ 
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.
╵

Can also confirmed that the two step approach suggested in here worked just fine, so I am going to stick with it.

Many thanks!