Unable to create an aurora cluster with a cross region replica in a single plan/apply operation

This issue was originally opened by @silviabotros as hashicorp/terraform#19524. It was migrated here as a result of the provider split. The original body of the issue is below.

---

Terraform Version

Terraform v0.11.10

Terraform Configuration Files

module "rds-cluster-vpc-1" {
  source = "../../modules/rds_cluster"
  name = "${var.db_name}-rds-cluster-${var.user}"
  user = "${var.user}"
  availability_zones = ["${data.aws_availability_zones.vpc-1-azs.names[0]}",
                        "${data.aws_availability_zones.vpc-1-azs.names[1]}",
                        "${data.aws_availability_zones.vpc-1-azs.names[2]}"
                       ]
  rds_final_snapshot_id   = "${var.db_name}-final-snapshot-${var.user}"
  skip_final_rds_snapshot = true
  vpc_id                  = "${module.vpc-1.vpc_id}"
  aws_subnet_ids          = ["${module.vpc-1.database_subnets}"]
  rds_access_sg              = ["${module.vpc-1-jump.security_group_id}"]
  providers = {
    "aws" = "aws.us-east-1"
  }
  db_name = "${var.db_name}"
  rds_admin_user = "${var.rds_admin_user}"
  rds_admin_password = "${var.rds_admin_password}"
  port = "${var.port}"
  tags = "${local.tags}"
}

module "rds-cluster-vpc-2" {

  source = "../../modules/rds_cluster"
  name = "${var.db_name}-rds-cluster-${var.user}"
  user = "${var.user}"
  availability_zones = ["${data.aws_availability_zones.vpc-2-azs.names[0]}",
                        "${data.aws_availability_zones.vpc-2-azs.names[1]}",
                        "${data.aws_availability_zones.vpc-2-azs.names[2]}"
                       ]
  replication_source_identifier = "${module.rds-cluster-vpc-1.rds_cluster_arn}"
  rds_final_snapshot_id   = "${var.db_name}-final-snapshot-${var.user}"
  skip_final_rds_snapshot = true
  vpc_id                  = "${module.vpc-2.vpc_id}"
  aws_subnet_ids          = ["${module.vpc-2.database_subnets}"]
  rds_access_sg              = ["${module.vpc-2-jump.security_group_id}"]
  providers = {
    "aws" = "aws.us-west-2"
  }
  db_name = "${var.db_name}"
  rds_admin_user = "${var.rds_admin_user}"
  rds_admin_password = "${var.rds_admin_password}"
  port = "${var.port}"
  tags = "${local.tags}"
}

Expected Behavior

Terraform creates primary cluster then the second cluster, using the first one ARN as the replication_source_identifier

Actual Behavior

[2018-11-30T17:29:14Z] * aws_rds_cluster.rds_cluster: error creating RDS cluster: InvalidDBClusterStateFault: Source cluster arn:aws:rds:us-east-1:224182330776:cluster:rdstest-cluster-dev doesn't have a writer instance
[2018-11-30T17:29:14Z] 	status code: 400, request id: c8e34d9a-d287-496a-9f02-3ca6037a6c80
[2018-11-30T17:29:14Z] 

Steps to Reproduce

Create a module that creates an aurora cluster then tries to use its ARN as the replication_source_identifier in a second one

Additional Context

I am creating this to be a reuseable module for my teams to out of the box get an Aurora cluster with a cross region replica cluster. Ideally, I'd like it all to work within terraform but as it behaves now I have to wrap it in a shell script that does phased, separate apply phases as it seems that AWS marks the replication_source_identifier ARN 'ready' before it has an assigned writer OR is inappropriately sending back that state fault when the writer is actually there and available.

[SathyaBhat]

Running into the same problem, I worked around this by adding a depends_on the RDS Cluster instance. This ensures that cross-region replica is created only after the all the primary region instances have been created

resource "aws_rds_cluster" "aurora_cluster_uw2" {
  cluster_identifier = "${var.service_name}-uw2"
  replication_source_identifier = "${aws_rds_cluster.aurora_cluster_ue1.arn}"
  db_cluster_parameter_group_name = "${aws_rds_cluster_parameter_group.aurora_param_group_uw2.name}"
  vpc_security_group_ids = 
  db_subnet_group_name = 
  engine = "aurora-mysql"
  master_username = "${var.master_username}"
  master_password = "${var.master_password}"
  backup_retention_period = 7
  enabled_cloudwatch_logs_exports = ["audit", "error", "general", "slowquery"]
  final_snapshot_identifier = "${var.service_name}-uw2-final-snapshot"
  depends_on = ["aws_rds_cluster_instance.aurora_instance_ue1"]
  provider = "aws.west"

[bflad]

Hi folks 👋

The above configuration usage of depends_on mentioned by @SathyaBhat is the correct answer here. To match RDS' expected order of operations when creating cross-region replicas (which requires a writer instance to be deployed before setting up the replica cluster), the Terraform configuration needs additional hints to perform its operations in the expected order.

If there are suggestions for how to improve our documentation in this regard, please submit a new GitHub issue with any specific recommendations or configurations. 👍

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!