fix: Prevent materialization job from failing when some shards require more work than others

[tkaemming]

Problem

If column is materialized in only a subset of shards within a partition, this task was failing.

Changes

Updates implementation to accommodate the situation where some shards require materialization mutations for a partition and others do not.

Does this work well for both Cloud and self-hosted?

Yes

How did you test this code?

Updated existing tests - it's not really feasible to write an integration test for this because our CI ClickHouse only has a single shard, but the updated unit tests should cover the non-overlapping keys (in our case, partition IDs) case that we have been seeing. Existing integration tests and type checks should ensure it runs correctly overall

[tkaemming]

This not a very good name for this function (it is actually a pretty bad one), but I'm struggling to come up with a good name that describes what this actually does. If you would like to help put me out of my misery, the type signature and tests are probably the best examples of what this is doing. Maybe this is something similar to transposition?

[greptile-apps]

PR Summary

This PR fixes materialization jobs that were failing when only some shards required mutations for a partition by implementing a more robust mapping approach.

• Added new join_mappings function to handle non-overlapping keys between shards, replacing the previous zip_values function
• Changed get_mutations_to_run return type from sequence to mapping with partition IDs as keys for better organization
• Modified run_materialize_mutations to process mutations by partition rather than assuming uniform shard requirements
• Added detailed logging to track materialization progress by partition
• Implemented comprehensive tests for the new join_mappings function with various mapping scenarios

_{2 file(s) reviewed, no comment(s)
Edit PR Review Bot Settings | Greptile}