aws_efs_file_system_policy inconsistent final plan when upgrading resources created with 3.58.0 using 3.59.0

[gwvandesteeg]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.0.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.59.0

Affected Resource(s)

• aws_efs_file_system_policy

Terraform Configuration Files

Extracted from the module being used (check validity).

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key: https://keybase.io/hashicorp

# find out the account id
data "aws_caller_identity" "current" {}

# find out the region
data "aws_region" "current" {}

#######################################################################
#
# KMS key policies
#

# iam policy for the root account
data "aws_iam_policy_document" "kms-root" {
  statement {
    # Allow direct access to key metadata to the account
    sid    = "AllowAccountOwnerKMSAccess"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]
    }
    actions = [
      "kms:Describe*",
      "kms:Get*",
      "kms:List*",
      "kms:RevokeGrant",
    ]
    resources = ["*"]

  }
}

# iam policy for the assumed role account
data "aws_iam_policy_document" "kms-caller" {
  statement {
    sid    = "Allow calling account"
    effect = "Allow"
    principals {
      type = "AWS"
      identifiers = [
        data.aws_caller_identity.current.arn
      ]
    }
    actions   = ["kms:*"]
    resources = ["*"]
  }
}

# iam policy for the use of the KMS key by EFS in the account
data "aws_iam_policy_document" "kms-usage" {
  statement {
    # Allow access to EFS for all principals in the account that are authorized to use EFS
    sid    = "AllowAccessToKMSKeys"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
    actions = [
      "kms:Encrypt",
      "kms:Decrypt",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:CreateGrant",
      "kms:DescribeKey",
    ]
    resources = ["*"]
    condition {
      test     = "StringEquals"
      variable = "kms:ViaService"
      values   = ["elasticfilesystem.${data.aws_region.current.name}.amazonaws.com"]
    }
    condition {
      test     = "StringEquals"
      variable = "kms:CallerAccount"
      values   = [data.aws_caller_identity.current.account_id]
    }
  }
}

data "aws_iam_policy_document" "kms-combined" {
  source_policy_documents = concat([
    data.aws_iam_policy_document.kms-root.json,
    data.aws_iam_policy_document.kms-caller.json,
    data.aws_iam_policy_document.kms-usage.json,
  ], var.additional_kms_policy_documents)
}

resource "aws_kms_key" "efs" {
  description             = "efs encryption key"
  deletion_window_in_days = 14
  enable_key_rotation     = true

  policy = data.aws_iam_policy_document.kms-combined.json
}

# the EFS resource
resource "aws_efs_file_system" "this" {
  creation_token                  ="testname"
  encrypted                       = true
  kms_key_id                      = aws_kms_key.efs.id
  performance_mode                = "generalPurpose"
  throughput_mode                 = "bursting"
  lifecycle_policy {
    transition_to_ia = "AFTER_90_DAYS"
  }
}

# iam policy for requiring mount and write actions of TLS
data "aws_iam_policy_document" "efs-in-flight" {
  statement {
    sid    = "ForceEncryptionForInFlightTraffic"
    effect = "Allow"
    principals {
      type        = "AWS"
      identifiers = ["*"]
    }
    actions = sort([
      "elasticfilesystem:ClientMount",
      "elasticfilesystem:ClientWrite",
      "elasticfilesystem:ClientRootAccess",
    ])
    resources = [aws_efs_file_system.this.arn]
  }
}

data "aws_iam_policy_document" "efs-combined" {
  source_policy_documents = concat([
    data.aws_iam_policy_document.efs-in-flight.json,
  ], var.additional_efs_policy_documents)
}

# the file system access policy, forces connections to be in TLS
resource "aws_efs_file_system_policy" "policy" {
  file_system_id = aws_efs_file_system.this.id

  policy = data.aws_iam_policy_document.efs-combined.json
}

Debug Output

None needed

Panic Output

No panic

Expected Behavior

Terraform apply applies with no changes

Actual Behavior

The plan generates the following error for each EFS filesystem created

Error: Provider produced inconsistent final plan

When expanding the plan for
module.voipmonitor.module.this.aws_efs_file_system_policy.policy to include
new values learned so far during apply, provider
"registry.terraform.io/hashicorp/aws" produced an invalid new value for
.bypass_policy_lockout_safety_check: was cty.False, but now null.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Steps to Reproduce

1. create the above resources using 3.58.0
2. without making any changes, apply using 3.59.0 (basically run an update); issue appears

Important Factoids

The issue occurs when applying the plan using the updated 3.59.0 provider. We have two stages in our CI/CD pipeline to deal with the IaC code. One creates all resources from scratch (for Disaster Recovery) and then tears them down again. Since all resources in this stage are created using 3.59.0 there is no issue and everything works. The second stage of the test is to Upgrade existing resources created with a previous version to allow for incremental changes as they happen. It is the upgrade path that is broken.

This error also indicates that upgrading existing resources do not seem to be part of the CI/CD process for this provider itself.

References

none

[ewbankkit]

Relates: #20838.

[DrStrangepork]

I saw this issue in 3.57.0, 3.58.0, 3.59.0, 3.63.0, and I am still seeing this issue in 3.73.0. As far as I can tell, whenever the KMS key policy changes, terraform plan will generate + bypass_policy_lockout_safety_check = false, but since False is the default setting for this argument, terraform apply does not include it in the state file, so the error is generated.

We are not setting this argument in our templates. There is no "fix", as the state file is saved correctly with no value for bypass_policy_lockout_safety_check because it is the default value of False. Subsequent runs without changing the KMS key policy produce no errors. This is a false error reported for what appears to be terraform plan adding a value to the plan it does not need to add.

[DrStrangepork]

I have now confirmed that even adding the setting bypass_policy_lockout_safety_check = false to an aws_kms_key resource does not fix this problem. The state file is always saved without the bypass_policy_lockout_safety_check argument set (null), and any runs of terraform plan that requires a change will add this argument with value false only to have terraform apply not save it to the state file if that value is false.

[campbellmcneill]

Seen the same issue trying to update a service that was created on v.3.55.0 and now modifying with v.4.59.0. Seeing the very same. Going to try a later version of the aws provider and see if can create an explicity policy on the EFS volume.