[EPTBP-33] - Use the same kafka cert for retry deployments.

[agkountis]

currently receiving this error when our pod starts when using separate kafka certs in our retry deployments.

msg="unable to run app" error="failed when fetching from  │
│ kafka. errors: [{energy-billing.internal.billing_adapter_retry_1 -1 TOPIC_AUTHORIZATION_FAILED: Not authorized to access topics: [Topic authorization failed.]} {energy-billing.internal │
│ .billing_adapter_retry_1 -1 TOPIC_AUTHORIZATION_FAILED: Not authorized to access topics: [Topic authorization failed.]}]"

Attempting to see if a single kafka cert would fix kafka topic access issues.

[linear]

EPTBP-33 Send energy billing data to Billing team

[sbuliarca]

You should add your team to CODEOWNERS, so that you don't depend on pubsub approvals https://github.com/utilitywarehouse/kafka-cluster-config/blob/main/CODEOWNERS

[sbuliarca]

Linting is failing. See https://github.com/utilitywarehouse/tflint-ruleset-kafka-config/blob/main/rules/msk_unique_app_names.md

[sbuliarca]

To run linters locally, please see: https://github.com/utilitywarehouse/kafka-cluster-config/blob/main/CONTRIBUTING.md#linting

[uw-infra]

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-energy-billing
Path: dev-aws/kafka-shared-msk/energy-billing
Commit ID: ff38a1a758beb787b6e9f394240200d8993ba63b

✅ Run Status: Ok, Run Summary: Plan: 10 to add, 0 to change, 0 to destroy.

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # kafka_topic.internal_energy_billing_billing_adapter_deadletter will be created
  + resource "kafka_topic" "internal_energy_billing_billing_adapter_deadletter" {
      + config             = {
          + "cleanup.policy"        = "delete"
          + "compression.type"      = "zstd"
          + "local.retention.ms"    = "86400000"
          + "max.message.bytes"     = "1048576"
          + "remote.storage.enable" = "true"
          + "retention.ms"          = "2419200000"
        }
      + id                 = (known after apply)
      + name               = "energy-billing.internal.billing-adapter-deadletter"
      + partitions         = 1
      + replication_factor = 3
    }

  # kafka_topic.internal_energy_billing_billing_adapter_retry_1 will be created
  + resource "kafka_topic" "internal_energy_billing_billing_adapter_retry_1" {
      + config             = {
          + "cleanup.policy"        = "delete"
          + "compression.type"      = "zstd"
          + "local.retention.ms"    = "86400000"
          + "max.message.bytes"     = "1048576"
          + "remote.storage.enable" = "true"
          + "retention.bytes"       = "8053063680"
          + "retention.ms"          = "259200000"
        }
      + id                 = (known after apply)
      + name               = "energy-billing.internal.billing-adapter-retry-1"
      + partitions         = 5
      + replication_factor = 3
    }

  # kafka_topic.internal_energy_billing_billing_adapter_retry_2 will be created
  + resource "kafka_topic" "internal_energy_billing_billing_adapter_retry_2" {
      + config             = {
          + "cleanup.policy"        = "delete"
          + "compression.type"      = "zstd"
          + "local.retention.ms"    = "86400000"
          + "max.message.bytes"     = "1048576"
          + "remote.storage.enable" = "true"
          + "retention.bytes"       = "8053063680"
          + "retention.ms"          = "259200000"
        }
      + id                 = (known after apply)
      + name               = "energy-billing.internal.billing-adapter-retry-2"
      + partitions         = 5
      + replication_factor = 3
    }

  # module.billing_adapter.kafka_acl.producer_acl["energy-billing.internal.billing-adapter-retry-1"] will be created
  + resource "kafka_acl" "producer_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Write"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=energy-billing/billing-adapter"
      + id                           = (known after apply)
      + resource_name                = "energy-billing.internal.billing-adapter-retry-1"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

  # module.billing_adapter_retry_1.kafka_acl.group_acl["energy-billing.billing-adapter-retry-1"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=energy-billing/billing-adapter"
      + id                           = (known after apply)
      + resource_name                = "energy-billing.billing-adapter-retry-1"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

  # module.billing_adapter_retry_1.kafka_acl.producer_acl["energy-billing.internal.billing-adapter-retry-2"] will be created
  + resource "kafka_acl" "producer_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Write"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=energy-billing/billing-adapter"
      + id                           = (known after apply)
      + resource_name                = "energy-billing.internal.billing-adapter-retry-2"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

  # module.billing_adapter_retry_1.kafka_acl.topic_acl["energy-billing.internal.billing-adapter-retry-1"] will be created
  + resource "kafka_acl" "topic_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=energy-billing/billing-adapter"
      + id                           = (known after apply)
      + resource_name                = "energy-billing.internal.billing-adapter-retry-1"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

  # module.billing_adapter_retry_2.kafka_acl.group_acl["energy-billing.billing-adapter-retry-2"] will be created
  + resource "kafka_acl" "group_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=energy-billing/billing-adapter"
      + id                           = (known after apply)
      + resource_name                = "energy-billing.billing-adapter-retry-2"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Group"
    }

  # module.billing_adapter_retry_2.kafka_acl.producer_acl["energy-billing.internal.billing-adapter-deadletter"] will be created
  + resource "kafka_acl" "producer_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Write"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=energy-billing/billing-adapter"
      + id                           = (known after apply)
      + resource_name                = "energy-billing.internal.billing-adapter-deadletter"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

  # module.billing_adapter_retry_2.kafka_acl.topic_acl["energy-billing.internal.billing-adapter-retry-2"] will be created
  + resource "kafka_acl" "topic_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Read"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=energy-billing/billing-adapter"
      + id                           = (known after apply)
      + resource_name                = "energy-billing.internal.billing-adapter-retry-2"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

Plan: 10 to add, 0 to change, 0 to destroy.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/energy-billing as comment.