Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support new Arrow types decimal32 and decimal64 #6661

Open
CurtHagenlocher opened this issue Oct 31, 2024 · 1 comment
Open

Support new Arrow types decimal32 and decimal64 #6661

CurtHagenlocher opened this issue Oct 31, 2024 · 1 comment
Labels
enhancement Any new improvement worthy of a entry in the changelog help wanted

Comments

@CurtHagenlocher
Copy link

The Arrow specification was recently amended to support 32-bit and 64-bit decimal values. It would be good for the Rust implementation to support these.

I have made preliminary changes at https://github.com/CurtHagenlocher/arrow-rs/tree/decimal3264. The gaps which remain are

  1. Support for conversions between decimal32/decimal64 and other types.
  2. Support for reading lower-precision decimal values from Parquet as decimal32 or decimal64. Parquet already supports storing 32-bit and 64-bit decimal values, but these are currently being expanded to at least 128-bits on read when they could be returned more efficiently. This support should probably be opt-in to avoid breaking client code which isn't yet aware of the new types.

I could submit a PR as-is; otherwise, will try to find the time in the next month to implement the gaps I've identified.

@CurtHagenlocher CurtHagenlocher added the enhancement Any new improvement worthy of a entry in the changelog label Oct 31, 2024
@CurtHagenlocher
Copy link
Author

(I should note that my changes take the most straightforward approach of defining additional types. There may be significant advantages in the approach outlined by #2001, but it's also hard to imagine that this approach could be made backwards-compatible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog help wanted
Projects
None yet
Development

No branches or pull requests

2 participants