Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for nested types, geoparquet groups, and postgres jsonb in data table #860

Merged
merged 9 commits into from
Jun 3, 2024

Conversation

bchapuis
Copy link
Member

@bchapuis bchapuis commented Jun 1, 2024

No description provided.

@bchapuis bchapuis marked this pull request as ready for review June 1, 2024 11:35
Copy link
Contributor

@Drabble Drabble left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks good, I just added two comments.

for (int i = 0; i < fields.size(); i++) {
Field field = fields.get(i);
nested.put(field.name(), switch (field.type()) {
case BINARY -> group.getBinaryValue(i).getBytes();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When an optional field or a group is null in the geoparquet file, this will raise an exception. The same thing happens in the asRowValues method. I tested it with real Overture data.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good point. I'm not a fan of returning Optional everywhere, but this is something we could consider. Another option could be to return null and let the user check for it.

@Drabble @sebr72 Which variant do you prefer?

Copy link
Contributor

@sebr72 sebr72 Jun 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bchapuis My instinct would say that when a field is optional we use Optional. This way we can make a difference (in the way our code explains it) with fields which are not optional with a null value.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sebr72 The problem is that we also need to return Optional for each values in a potentially very big geoparquet file. As for the List, I'm a bit worried about the overhead. It also means that we need to return Optional for required values.

Here are some interesting arguments for and against the use of Optional:
https://blogs.oracle.com/javamagazine/post/optional-class-null-pointer-drawbacks

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bchapuis based on the article and your concern about the overhead, I am happy if we don't use Optional.

Copy link

sonarqubecloud bot commented Jun 3, 2024

Quality Gate Failed Quality Gate failed

Failed conditions
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarCloud

Catch issues before they fail your Quality Gate with our IDE extension SonarLint

@bchapuis bchapuis merged commit cd2018d into main Jun 3, 2024
8 of 9 checks passed
@bchapuis bchapuis deleted the 849-nested-type-and-jsonb branch June 3, 2024 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants