[Feature Request]: [IcebergIO] Allow users to specify a partition spec when creating tables

[ahmedabu98]

What would you like to happen?

Add a new configuration option to pass a Map<String, String> representing columns and their partition transforms

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[Suvrat1629]

Hey @ahmedabu98, I was taking a look at this issue. If we added a setPartitionSpec in the IcebergIO.java and then tweaked the WriteToDestination.java file it is also causng a problem in the AutoValue_IcebergIO_WriteRows.java. What does this file do?

[ahmedabu98]

Notice that IcebergIO's subclass "WriteRows" is annotated with "@autovalue". It's a nice tool that helps generate boilerplate code, we use it quite a bit in this code base. Nice article here that describes how it works: https://www.baeldung.com/introduction-to-autovalue

[Suvrat1629]

@ahmedabu98 ok so this will generate the AutoValue_IcebergIO_WriteRows.java automatically so I dont have make any changes in this file; I just need to tweak the "WriteRows" and the "@autovalue" will auto-geherate this file? Is this right? Can I give this issue a shot?

[Suvrat1629]

@ahmedabu98 Hey I did made the changes to the IcerbegIO.java, and added createTableWithPartitionSpec() and createPartitionSpec() to WriteToDestinations.java, WriteUngroupedRowsToFiles.java and WriteGroupedRowsToFiles.java files to handle the partition spec creationl; but I am in a bit confusion if it was right to make changes to the WriteToDestinations.java, WriteUngroupedRowsToFiles.java and WriteGroupedRowsToFiles.java files. Does this issue concern any changes to the RecordWriterManager.java? Could you guide me a bit?
Thank you.