Add support for importing from iceberg table definition #452
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Import side of #449 .
I'm not going to have time for writing up the export side of this for a few days (at least) so just going to start with the import side and see what folks think. I think the export side is also a bit more involved since it is necessary to parse
definitions
into nested struct types of the iceberg schema.I've mostly copied the model of the Spark import for this since the Iceberg table format is quite similar to the Spark table definitions. Eventually it might be nice to add support for ingesting the table definition directly from the catalog (via the pyiceberg sdk), but I did not attempt to tackle that in this first iteration. I'm guessing I missed things so please just let me know what needs to be changed.
This assumes the table definition is in JSON format as defined by https://iceberg.apache.org/spec/#appendix-c-json-serialization.