[Bug]: Unable to update MSK cluster policy after enabling multi-VPC connectivity

[Czeffik]

Terraform Core Version

1.7.0

AWS Provider Version

5.34.0

Affected Resource(s)

aws_msk_cluster_policy

Expected Behavior

After enabling multi-VPC connectivity for MSK cluster it should be possible to update aws_msk_cluster_policy.

Actual Behavior

When I create a MSK cluster policy using aws_msk_cluster_policy I am receiving:

Error: setting MSK Cluster Policy (arn:aws:kafka:us-west-2:${account_id}:cluster/${cluster_name}/${cluster_id}): operation error Kafka: PutClusterPolicy, https response error StatusCode: 400, RequestID: ${request_id}, BadRequestException: Current cluster policy version needed for Update

during terraform plan it looks like terraform is not aware of existing aws_msk_cluster_policy - it does not modify resource but adds new one.

Relevant Error/Panic Output Snippet

Basically, terraform is not aware that `aws_msk_cluster_policy` exist and is not pulling the current policy version prior to being updated.

Terraform Configuration Files

resource "aws_msk_cluster_policy" "example" {
  cluster_arn = var.msk_cluster_arn

  policy = jsonencode({
    Version = "2012-10-17",
    Statement = [{
      Sid    = "ExampleMskClusterPolicy"
      Effect = "Allow"
      Principal = {
        "AWS" = "arn"aws:iam::[account_id]:root"
      }
      Action = [
        "kafka:Describe*",
        "kafka:Get*",
        "kafka:CreateVpcConnection",
        "kafka:GetBootstrapBrokers",
      ]
      Resource = var.msk_cluster_arn
    }]
  })
}

Steps to Reproduce

• Create MSK cluster with disabled multi-VPC connectivity
• Modify MSK cluster- enable multi-VPC connectivity
• Create MSK cluster policy

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

This issue looks similar to this one: #33107

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Potentially related #34419 (see this comment specifically)

[vishwa-trulioo]

Is there any update on this? I raised the same issue sometime back and they rolled out a fix. but, looks like it's back again. How can this happen?
#33107

[vishwa-trulioo]

@ewbankkit Sometime back you fixed this same issue with #33107. But, It looks to have come back. I even used same provider version which originally worked (5.15).
Gives me this error.

aws_msk_cluster_policy.default: Creating...
╷
│ Error: setting MSK Cluster Policy (arn:aws:kafka:us-west-2:532269373286:cluster/basic/da77abde-8672-4b94-42f8-2e0cb730fc49-10): operation error Kafka: PutClusterPolicy, https response error StatusCode: 400, RequestID: ab940298-8635-4e78-8ef1-da15b2b55052, BadRequestException: Current cluster policy version needed for Update
│ 
│   with aws_msk_cluster_policy.default,
│   on data.tf line 12, in resource "aws_msk_cluster_policy" "default":
│   12: resource "aws_msk_cluster_policy" "default" {

Here is the output from the TF Debug log enabled,

aws_msk_cluster_policy.default: Creating...
2024-06-14T18:33:47.874-0700 [INFO]  Starting apply for aws_msk_cluster_policy.default
2024-06-14T18:33:47.875-0700 [DEBUG] aws_msk_cluster_policy.default: applying the planned Create change
2024-06-14T18:33:53.612-0700 [ERROR] provider.terraform-provider-aws_v5.20.0_x5: Response contains error diagnostic: @caller=github.com/hashicorp/terraform-plugin-go@v0.19.0/tfprotov5/internal/diag/diagnostics.go:58 tf_req_id=a33a88be-c1f6-af9a-85df-92061840931e tf_proto_version=5.4 tf_provider_addr=registry.terraform.io/hashicorp/aws @module=sdk.proto diagnostic_detail="" diagnostic_severity=ERROR diagnostic_summary="setting MSK Cluster Policy (arn:aws:kafka:us-west-2:532269373286:cluster/basic/da77abde-8672-4b94-42f8-2e0cb730fc49-10): operation error Kafka: PutClusterPolicy, https response error StatusCode: 400, RequestID: 3979bd4b-bda1-4dd1-ac16-e0cca14ae784, BadRequestException: Current cluster policy version needed for Update" tf_resource_type=aws_msk_cluster_policy tf_rpc=ApplyResourceChange timestamp=2024-06-14T18:33:53.612-0700
2024-06-14T18:33:53.636-0700 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2024-06-14T18:33:53.636-0700 [ERROR] vertex "aws_msk_cluster_policy.default" error: setting MSK Cluster Policy (arn:aws:kafka:us-west-2:532269373286:cluster/basic/da77abde-8672-4b94-42f8-2e0cb730fc49-10): operation error Kafka: PutClusterPolicy, https response error StatusCode: 400, RequestID: 3979bd4b-bda1-4dd1-ac16-e0cca14ae784, BadRequestException: Current cluster policy version needed for Update
╷
│ Error: setting MSK Cluster Policy (arn:aws:kafka:us-west-2:532269373286:cluster/basic/da77abde-8672-4b94-42f8-2e0cb730fc49-10): operation error Kafka: PutClusterPolicy, https response error StatusCode: 400, RequestID: 3979bd4b-bda1-4dd1-ac16-e0cca14ae784, BadRequestException: Current cluster policy version needed for Update
│ 
│   with aws_msk_cluster_policy.default,
│   on data.tf line 12, in resource "aws_msk_cluster_policy" "default":
│   12: resource "aws_msk_cluster_policy" "default" {
│ 
╵
2024-06-14T18:33:53.661-0700 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-06-14T18:33:53.667-0700 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/aws/5.20.0/darwin_amd64/terraform-provider-aws_v5.20.0_x5 pid=6146
2024-06-14T18:33:53.667-0700 [DEBUG] provider: plugin exited

Are you able to see if you can help with this? Thank you in advance.

[vishwa-trulioo]

Here is another thing I discovered. This error only shows up when Updating the MSK cluster policy. i.e. If you are adding Cluster policy for the first time to an MSK cluster it won't give error.