feat: Integration Tests for Denormalization

[vaibhav-datazip]

Problem

Currently, the codebase supports both normalized and denormalized data storage modes through the normalization flag in stream metadata, but lacks comprehensive integration tests that validate denormalization behavior across different drivers and scenarios. This gap creates several problems:

1. Insufficient End-to-End Validation: While the codebase has logic to handle denormalized data storage (storing raw JSON in StringifiedData column when normalization=false), there are no integration tests that verify this works correctly from source to destination. Without these tests, we cannot be certain that denormalized data is correctly:

   • Stored as JSON strings in the destination
   • Preserved with all nested structures intact
   • Readable and parseable after storage
   • Handled correctly during CDC operations (insert, update, delete)

2. Driver-Specific Behavior Unknown: Different drivers (MySQL, Postgres, MongoDB) have different data structures:

   • Relational drivers (MySQL, Postgres) default to normalization=true and store flattened columns
   • Non-relational drivers (MongoDB) default to normalization=false and store raw JSON
   • Without integration tests, we cannot verify that denormalization works correctly when explicitly set to false for relational drivers or when set to true for non-relational drivers

etc.
Without proper integration tests, users cannot be confident that denormalized mode works as expected for their use cases.

Solution

Architecture

The integration tests should follow the existing test structure in utils/testutils/test_utils.go and extend the IntegrationTest type to support denormalization testing scenarios.

Expected Outcomes

• Comprehensive integration test coverage for denormalization functionality across MySQL, Postgres, and MongoDB
• Validation that denormalized data is correctly stored as JSON strings
• Verification that complex nested data structures are preserved in denormalized mode

[sujaLs-codes]

Picking up #671! Adding integration tests for denormalization across drivers. PR coming in 2 days

[nayanj98]

Hi @sujaLs-codes,
You had previously asked for and picked up feat: BigQuery as a source (#446 ), which is a major contribution. You can continue working on it, do the requested changes, and if you need help at any point, feel free to ask we’ll be happy to guide you.
However, if you find it difficult at the moment, it's completely fine to start with a good first issue to get more familiar with the codebase and project structure. In that case, please let me know, I’ll unassign you from the BigQuery issue and you can close your existing PR, and then pick this one up instead.
Thanks!

[codewarnab]

Hey, I went ahead and worked on this. Would be great if you could assign it to me, happy to raise a PR!

Basically made the test framework flexible enough to test normalization = false:

• Added a Normalization flag to IntegrationTest (nil defaults to true so existing tests dont break)
• updateStreamsCommand() now takes this flag instead of hardcoding true
• VerifyIcebergSync() can now check the raw JSON in the data column when denormalized
• New test file postgres_denorm_test.go for testing this path

[nayanj98]

Sure @codewarnab I am assigning this issue to you. You can raise your PR as well.

[codewarnab]

raised #674