-
Notifications
You must be signed in to change notification settings - Fork 178
Open
Labels
Description
Problem Statement
The current VerifyIcebergSync test function contains tightly coupled query parsing, schema checks, and partition validation logic. This makes the function large, difficult to maintain. Refactoring is required to isolate parsing and validation into reusable utility functions.
Solution Summary
Extract the query execution, schema comparison, and partition parsing logic into dedicated utility functions. Keep VerifyIcebergSync focused only on orchestration and assertions. This reduces duplication, improves readability, and enables reuse across other Iceberg sync tests.
olake/utils/testutils/test_utils.go
Lines 519 to 524 in 23130d1
| // TODO: Refactor parsing logic into a reusable utility functions | |
| // verifyIcebergSync verifies that data was correctly synchronized to Iceberg | |
| func VerifyIcebergSync(t *testing.T, tableName, icebergDB string, datatypeSchema map[string]string, schema map[string]interface{}, opSymbol, partitionRegex, driver string) { | |
| t.Helper() | |
| ctx := context.Background() | |
| spark, err := sql.NewSessionBuilder().Remote(sparkConnectAddress).Build(ctx) |