Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for importing from iceberg table definition #452

Merged
merged 1 commit into from
Oct 10, 2024

Conversation

paulcichonski
Copy link
Contributor

@paulcichonski paulcichonski commented Oct 6, 2024

Import side of #449 .

I'm not going to have time for writing up the export side of this for a few days (at least) so just going to start with the import side and see what folks think. I think the export side is also a bit more involved since it is necessary to parse definitions into nested struct types of the iceberg schema.

I've mostly copied the model of the Spark import for this since the Iceberg table format is quite similar to the Spark table definitions. Eventually it might be nice to add support for ingesting the table definition directly from the catalog (via the pyiceberg sdk), but I did not attempt to tackle that in this first iteration. I'm guessing I missed things so please just let me know what needs to be changed.

This assumes the table definition is in JSON format as defined by https://iceberg.apache.org/spec/#appendix-c-json-serialization.

  • Tests pass
  • ruff format
  • README.md updated (if relevant)
  •  CHANGELOG.md entry added

Assumes the table definition is in JSON format as defined by https://iceberg.apache.org/spec/#appendix-c-json-serialization.

Eventually it might be nice to add support for ingesting the table definition directly from the catalog (via the pyiceberg sdk), but I did not attempt to tackle that in this first iteration.
@paulcichonski
Copy link
Contributor Author

re: tests, I get a lot of local test failures when running from the main branch so I haven't been able to confirm all tests are passing, but I don't think my changes should impact any tests other than the ones I added directly.

@jochenchrist
Copy link
Contributor

Thank you for your contribution. Looks good to me.

@jochenchrist jochenchrist merged commit bd7d8b4 into datacontract:main Oct 10, 2024
5 checks passed
@paulcichonski paulcichonski deleted the add-iceberg branch October 10, 2024 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants