-
Notifications
You must be signed in to change notification settings - Fork 203
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: use arrow-ipc to communicate between remote server and client #552
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
645ff59
to
cd7408b
Compare
|
a7c4794
to
863be12
Compare
863be12
to
c497766
Compare
@Rachelint Thanks for remind. I have add a |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
…pache#552) * feat: use arrow-ipc to communicate bewteen remote server and client * add benchmarks * add zstd for ipc * add RemoteEngineVersion * fix message
Which issue does this PR close?
Closes #
Rationale for this change
In current implementation, remote service communication bewteen ceresdb using
avro to serialize RecordBatch. For arrow ecosystem, there is a designated serialization method -- Arrow IPC
Some benchmark:
Encoded bytes using Arrow IPC will take more space 1348496/904634 = 1.4, but it's more much less time.
After zstd compression, IPC is better than avro in both size and speed.
What changes are included in this PR?
Replace avro with arrow ipc for remote service communication
Are there any user-facing changes?
No
How does this change test
New UT
test_ipc_encode_decode