Skip to content

[SPARK-50464] Support unsigned integer for Arrow #49022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

chenkovsky
Copy link

What changes were proposed in this pull request?

Support unsigned integers for arrow format.

  1. Unsigned byte is converted into short.
  2. Unsigned short is converted into integer.
  3. Unsigned integer is converted into long.
  4. Unsinged long is converted into Decimal(20, 0)

Why are the changes needed?

currently spark will throw UNSUPPORTED_ARROWTYPE exception when reading arrow format file which contains unsigned integer.

Does this PR introduce any user-facing change?

No

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

No

@chenkovsky chenkovsky marked this pull request as draft December 2, 2024 05:20
@chenkovsky chenkovsky force-pushed the feature/arrow-unsigned branch 2 times, most recently from 1708b9c to 474d95d Compare December 3, 2024 14:34
@chenkovsky chenkovsky force-pushed the feature/arrow-unsigned branch from 474d95d to 58db218 Compare December 3, 2024 15:01
@chenkovsky chenkovsky changed the title [WIP][SPARK-50464] Support unsigned integer for Arrow [SPARK-50464] Support unsigned integer for Arrow Dec 4, 2024
@chenkovsky chenkovsky marked this pull request as ready for review December 4, 2024 00:34
@miscenko
Copy link

Hi,
I am curious, what is the status of this PR? Was it abandoned ?

@chenkovsky
Copy link
Author

Hi, I am curious, what is the status of this PR? Was it abandoned ?

it seems that nobody is interested in reviewing this pr. ಥ_ಥ

@miscenko
Copy link

Hi, I am curious, what is the status of this PR? Was it abandoned ?

it seems that nobody is interested in reviewing this pr. ಥ_ಥ

It's a pity, I think this is a very useful fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants