Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rename table property for enabling Parquet dictionary encoding #7663

Closed
aokolnychyi opened this issue May 19, 2023 · 4 comments · Fixed by #7665
Closed

Rename table property for enabling Parquet dictionary encoding #7663

aokolnychyi opened this issue May 19, 2023 · 4 comments · Fixed by #7665
Milestone

Comments

@aokolnychyi
Copy link
Contributor

Feature Request / Improvement

See here.

Query engine

None

@aokolnychyi aokolnychyi added this to the Iceberg 1.3.0 milestone May 19, 2023
@amogh-jahagirdar
Copy link
Contributor

Commented on #7301 (comment) , we may want to see if we even want a table property in the first place, curious to know others thoughts @singhpk234 @Fokko @rdblue

@aokolnychyi
Copy link
Contributor Author

I agree with @amogh-jahagirdar.

@Fokko
Copy link
Contributor

Fokko commented May 20, 2023

Dictionary encoding is useful for low cardinality columns, so the space difference between the two is negligible, with the tradeoff being deterministic lookups vs False positives from the bloom filter.

If it is low cardinality, the likelihood of having false positives is also low (assuming a fixed size bit for the bloom filter). I'm not sure if the dictionary is used for skipping data for example, but I don't think that should influence or impact this decision. Because if that's not the case, then we should fix that :)

I'm not super strong on this, but in the end, I think less configuration is better.

@rdblue
Copy link
Contributor

rdblue commented May 21, 2023

I'm +1 for removing this property entirely. It is only used for testing and we've never had a request for it in production.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants