Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support specify endianness for serialization #6917

Open
tisonkun opened this issue Dec 28, 2024 · 3 comments
Open

Support specify endianness for serialization #6917

tisonkun opened this issue Dec 28, 2024 · 3 comments
Labels
enhancement Any new improvement worthy of a entry in the changelog

Comments

@tisonkun
Copy link
Member

tisonkun commented Dec 28, 2024 •
edited
Loading

It seems arrow-rs's serialization uses machine native endianness.

Is it possible to add a feature flag to stick to an endianness, said little endian? This would helps the software outputs a stable serialize format on all the machine.
@tisonkun tisonkun added the enhancement Any new improvement worthy of a entry in the changelog label Dec 28, 2024
@tisonkun
Copy link
Member Author

But arrow is used for memory transfer/ipc also .. I'm not sure if it's defined in the spec.

@tisonkun
Copy link
Member Author

Found apache/arrow#15587

Unsure the implementation details of arrow-rs.

@tustvold
Copy link
Contributor

tustvold commented Dec 28, 2024 •
edited
Loading

Arrow-rs does not support running on big endian architectures at this time, it is not tested and as such I expect lots will be broken.

Regarding IPC, I believe it will error if the data file is big endian encoded. All data is written little endian.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tustvold @tisonkun