Skip to content

Use a single kafka topic for all FWF processing events #879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Use a single kafka topic for all FWF processing events #879

wants to merge 2 commits into from
+6 −82

Conversation

josephsilcock
Copy link
Collaborator

No description provided.

@linear Linear
Copy link

linear bot commented Jun 5, 2025

BA-138 Create indexers for invoice-engine events

@uw-infra
Copy link

uw-infra commented Jun 5, 2025

Auto plan is disabled for this PR.
Please post @terraform-applier plan <module_name> as comment if you want to request terraform plan for a particular module.

@josephsilcock josephsilcock force-pushed the BA-138-fwf-processing-events branch from 0d38320 to 42d123d Compare June 5, 2025 09:03
@josephsilcock josephsilcock marked this pull request as ready for review June 10, 2025 13:47
@josephsilcock josephsilcock requested a review from a team as a code owner June 10, 2025 13:47
@uw-infra
Copy link

uw-infra commented Jun 10, 2025 •
edited
Loading

Terraform run output for

Cluster: dev-aws
Module: pubsub/kafka-shared-msk-billing
Path: dev-aws/kafka-shared-msk/billing
Commit ID: 42d123df4786dd181e01d5ea39f08d4f82c8caac
✅ Run Status: Ok, Run Summary: Plan: 2 to add, 0 to change, 6 to destroy. 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # kafka_topic.core_bill_rectifications will be destroyed
  # (because kafka_topic.core_bill_rectifications is not in configuration)
  - resource "kafka_topic" "core_bill_rectifications" {
      - config             = {
          - "cleanup.policy"        = "delete"
          - "compression.type"      = "zstd"
          - "local.retention.ms"    = "172800000"
          - "remote.storage.enable" = "true"
          - "retention.ms"          = "2629800000"
        } -> null
      - id                 = "billing.core-bill-rectifications" -> null
      - name               = "billing.core-bill-rectifications" -> null
      - partitions         = 2 -> null
      - replication_factor = 3 -> null
    }

  # kafka_topic.fixed_width_file_deadletter will be destroyed
  # (because kafka_topic.fixed_width_file_deadletter is not in configuration)
  - resource "kafka_topic" "fixed_width_file_deadletter" {
      - config             = {
          - "cleanup.policy"        = "delete"
          - "compression.type"      = "zstd"
          - "local.retention.ms"    = "172800000"
          - "remote.storage.enable" = "true"
          - "retention.ms"          = "2629800000"
        } -> null
      - id                 = "billing.fixed-width-file-deadletter" -> null
      - name               = "billing.fixed-width-file-deadletter" -> null
      - partitions         = 2 -> null
      - replication_factor = 3 -> null
    }

  # kafka_topic.fixed_width_file_load_errors will be destroyed
  # (because kafka_topic.fixed_width_file_load_errors is not in configuration)
  - resource "kafka_topic" "fixed_width_file_load_errors" {
      - config             = {
          - "cleanup.policy"        = "delete"
          - "compression.type"      = "zstd"
          - "local.retention.ms"    = "172800000"
          - "remote.storage.enable" = "true"
          - "retention.ms"          = "2629800000"
        } -> null
      - id                 = "billing.fixed-width-file-load-errors" -> null
      - name               = "billing.fixed-width-file-load-errors" -> null
      - partitions         = 2 -> null
      - replication_factor = 3 -> null
    }

  # kafka_topic.fixed_width_file_processing_events will be created
  + resource "kafka_topic" "fixed_width_file_processing_events" {
      + config             = {
          + "cleanup.policy"        = "delete"
          + "compression.type"      = "zstd"
          + "local.retention.ms"    = "172800000"
          + "remote.storage.enable" = "true"
          + "retention.ms"          = "2629800000"
        }
      + id                 = (known after apply)
      + name               = "billing.fixed-width-file-processing-events"
      + partitions         = 2
      + replication_factor = 3
    }

  # module.bill_composition_engine.kafka_acl.producer_acl["billing.core-bill-rectifications"] will be destroyed
  # (because key ["billing.core-bill-rectifications"] is not in for_each map)
  - resource "kafka_acl" "producer_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Write" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=billing/bill-composition-engine" -> null
      - id                           = "User:CN=billing/bill-composition-engine|*|Write|Allow|Topic|billing.core-bill-rectifications|Literal" -> null
      - resource_name                = "billing.core-bill-rectifications" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.bill_composition_engine.kafka_acl.producer_acl["billing.fixed-width-file-deadletter"] will be destroyed
  # (because key ["billing.fixed-width-file-deadletter"] is not in for_each map)
  - resource "kafka_acl" "producer_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Write" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=billing/bill-composition-engine" -> null
      - id                           = "User:CN=billing/bill-composition-engine|*|Write|Allow|Topic|billing.fixed-width-file-deadletter|Literal" -> null
      - resource_name                = "billing.fixed-width-file-deadletter" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.bill_composition_engine.kafka_acl.producer_acl["billing.fixed-width-file-load-errors"] will be destroyed
  # (because key ["billing.fixed-width-file-load-errors"] is not in for_each map)
  - resource "kafka_acl" "producer_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Write" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=billing/bill-composition-engine" -> null
      - id                           = "User:CN=billing/bill-composition-engine|*|Write|Allow|Topic|billing.fixed-width-file-load-errors|Literal" -> null
      - resource_name                = "billing.fixed-width-file-load-errors" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.bill_composition_engine.kafka_acl.producer_acl["billing.fixed-width-file-processing-events"] will be created
  + resource "kafka_acl" "producer_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Write"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=billing/bill-composition-engine"
      + id                           = (known after apply)
      + resource_name                = "billing.fixed-width-file-processing-events"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

Plan: 2 to add, 0 to change, 6 to destroy.

To manually trigger plan again please post @terraform-applier plan dev-aws/kafka-shared-msk/billing as comment.

@uw-infra
Copy link

uw-infra commented Jun 10, 2025 •
edited
Loading

Terraform run output for

Cluster: prod-aws
Module: pubsub/kafka-shared-msk-billing
Path: prod-aws/kafka-shared-msk/billing
Commit ID: 42d123df4786dd181e01d5ea39f08d4f82c8caac
✅ Run Status: Ok, Run Summary: Plan: 2 to add, 0 to change, 6 to destroy. 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # kafka_topic.core_bill_rectifications will be destroyed
  # (because kafka_topic.core_bill_rectifications is not in configuration)
  - resource "kafka_topic" "core_bill_rectifications" {
      - config             = {
          - "cleanup.policy"        = "delete"
          - "compression.type"      = "zstd"
          - "local.retention.ms"    = "604800000"
          - "remote.storage.enable" = "true"
          - "retention.ms"          = "2629800000"
        } -> null
      - id                 = "billing.core-bill-rectifications" -> null
      - name               = "billing.core-bill-rectifications" -> null
      - partitions         = 2 -> null
      - replication_factor = 3 -> null
    }

  # kafka_topic.fixed_width_file_deadletter will be destroyed
  # (because kafka_topic.fixed_width_file_deadletter is not in configuration)
  - resource "kafka_topic" "fixed_width_file_deadletter" {
      - config             = {
          - "cleanup.policy"        = "delete"
          - "compression.type"      = "zstd"
          - "local.retention.ms"    = "604800000"
          - "remote.storage.enable" = "true"
          - "retention.ms"          = "2629800000"
        } -> null
      - id                 = "billing.fixed-width-file-deadletter" -> null
      - name               = "billing.fixed-width-file-deadletter" -> null
      - partitions         = 2 -> null
      - replication_factor = 3 -> null
    }

  # kafka_topic.fixed_width_file_load_errors will be destroyed
  # (because kafka_topic.fixed_width_file_load_errors is not in configuration)
  - resource "kafka_topic" "fixed_width_file_load_errors" {
      - config             = {
          - "cleanup.policy"        = "delete"
          - "compression.type"      = "zstd"
          - "local.retention.ms"    = "604800000"
          - "remote.storage.enable" = "true"
          - "retention.ms"          = "2629800000"
        } -> null
      - id                 = "billing.fixed-width-file-load-errors" -> null
      - name               = "billing.fixed-width-file-load-errors" -> null
      - partitions         = 2 -> null
      - replication_factor = 3 -> null
    }

  # kafka_topic.fixed_width_file_processing_events will be created
  + resource "kafka_topic" "fixed_width_file_processing_events" {
      + config             = {
          + "cleanup.policy"        = "delete"
          + "compression.type"      = "zstd"
          + "local.retention.ms"    = "604800000"
          + "remote.storage.enable" = "true"
          + "retention.ms"          = "2629800000"
        }
      + id                 = (known after apply)
      + name               = "billing.fixed-width-file-processing-events"
      + partitions         = 2
      + replication_factor = 3
    }

  # module.bill_composition_engine.kafka_acl.producer_acl["billing.core-bill-rectifications"] will be destroyed
  # (because key ["billing.core-bill-rectifications"] is not in for_each map)
  - resource "kafka_acl" "producer_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Write" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=billing/bill-composition-engine" -> null
      - id                           = "User:CN=billing/bill-composition-engine|*|Write|Allow|Topic|billing.core-bill-rectifications|Literal" -> null
      - resource_name                = "billing.core-bill-rectifications" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.bill_composition_engine.kafka_acl.producer_acl["billing.fixed-width-file-deadletter"] will be destroyed
  # (because key ["billing.fixed-width-file-deadletter"] is not in for_each map)
  - resource "kafka_acl" "producer_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Write" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=billing/bill-composition-engine" -> null
      - id                           = "User:CN=billing/bill-composition-engine|*|Write|Allow|Topic|billing.fixed-width-file-deadletter|Literal" -> null
      - resource_name                = "billing.fixed-width-file-deadletter" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.bill_composition_engine.kafka_acl.producer_acl["billing.fixed-width-file-load-errors"] will be destroyed
  # (because key ["billing.fixed-width-file-load-errors"] is not in for_each map)
  - resource "kafka_acl" "producer_acl" {
      - acl_host                     = "*" -> null
      - acl_operation                = "Write" -> null
      - acl_permission_type          = "Allow" -> null
      - acl_principal                = "User:CN=billing/bill-composition-engine" -> null
      - id                           = "User:CN=billing/bill-composition-engine|*|Write|Allow|Topic|billing.fixed-width-file-load-errors|Literal" -> null
      - resource_name                = "billing.fixed-width-file-load-errors" -> null
      - resource_pattern_type_filter = "Literal" -> null
      - resource_type                = "Topic" -> null
    }

  # module.bill_composition_engine.kafka_acl.producer_acl["billing.fixed-width-file-processing-events"] will be created
  + resource "kafka_acl" "producer_acl" {
      + acl_host                     = "*"
      + acl_operation                = "Write"
      + acl_permission_type          = "Allow"
      + acl_principal                = "User:CN=billing/bill-composition-engine"
      + id                           = (known after apply)
      + resource_name                = "billing.fixed-width-file-processing-events"
      + resource_pattern_type_filter = "Literal"
      + resource_type                = "Topic"
    }

Plan: 2 to add, 0 to change, 6 to destroy.

To manually trigger plan again please post @terraform-applier plan prod-aws/kafka-shared-msk/billing as comment.

tginszt
tginszt approved these changes Jun 10, 2025
View reviewed changes
Copy link
Contributor

@tginszt tginszt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a nice improvement!
Not sure if the invoice engine was updated yet, it definetely needs to reflect these changes too

@josephsilcock
Copy link
Collaborator Author

That's a nice improvement! Not sure if the invoice engine was updated yet, it definetely needs to reflect these changes too

Thanks!
The bill-composition-engine MR isn't merged yet

https://github.com/utilitywarehouse/go-mono/pull/1826, so I'll merge this after that's done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tginszt tginszt tginszt approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@josephsilcock @uw-infra @tginszt