Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C#] Flight DoExchange server is incompatible with C++/PyArrow client #44360

Closed
adamreeve opened this issue Oct 9, 2024 · 3 comments
Closed
Assignees
Milestone

Comments

@adamreeve
Copy link
Contributor

adamreeve commented Oct 9, 2024

Describe the bug, including details regarding any error messages, version, and platform.

A .NET Flight Server with a DoExchange method is incompatible with a PyArrow Flight client. The C# implementation expects the first message received from the data stream to contain both the descriptor and the schema, but the client first sends the descriptor with an empty data_header, followed by another message with the schema.

This results in an error like this in the C# server:

fail: Grpc.AspNetCore.Server.ServerCallHandler[6]
      Error when executing service method 'DoExchange'.
      System.ArgumentOutOfRangeException: Specified argument was out of the range of valid values. (Parameter 'length')
         at Google.FlatBuffers.ByteBuffer.GetUint(Int32 offset)
         at Google.FlatBuffers.ByteBuffer.GetInt(Int32 offset)
         at Apache.Arrow.Flatbuf.Message.GetRootAsMessage(ByteBuffer _bb, Message obj)
         at Apache.Arrow.Flatbuf.Message.GetRootAsMessage(ByteBuffer _bb)
         at Apache.Arrow.Flight.Internal.RecordBatchReaderImplementation.ReadSchemaAsync(CancellationToken cancellationToken)
         at Apache.Arrow.Flight.Internal.RecordBatchReaderImplementation.ReadFlightDescriptor()
         at ExchangeServer.Server.DoExchange(FlightServerRecordBatchStreamReader requestStream, FlightServerRecordBatchStreamWriter responseStream, ServerCallContext context) in /home/adam/dev/flight-exchange/DotnetExample/ExchangeServer/Server.cs:line 15
         at Grpc.Shared.Server.DuplexStreamingServerMethodInvoker`3.Invoke(HttpContext httpContext, ServerCallContext serverCallContext, IAsyncStreamReader`1 requestStream, IServerStreamWriter`1 responseStream)
         at Grpc.Shared.Server.DuplexStreamingServerMethodInvoker`3.Invoke(HttpContext httpContext, ServerCallContext serverCallContext, IAsyncStreamReader`1 requestStream, IServerStreamWriter`1 responseStream)
         at Grpc.AspNetCore.Server.Internal.CallHandlers.DuplexStreamingServerCallHandler`3.HandleCallAsyncCore(HttpContext httpContext, HttpContextServerCallContext serverCallContext)
         at Grpc.AspNetCore.Server.Internal.CallHandlers.ServerCallHandlerBase`3.<HandleCallAsync>g__AwaitHandleCall|8_0(HttpContextServerCallContext serverCallContext, Method`2 method, Task handleCall)

DoExchange with a .NET client and .NET server does work, and using a PyArrow server and .NET client also works.

Component(s)

C#

@adamreeve
Copy link
Contributor Author

take

@adamreeve
Copy link
Contributor Author

I have a fix for this (adamreeve@8910d69) but don't see a way to easily add a test. I think it's worth setting up C# Flight integration tests first (#44361).

CurtHagenlocher pushed a commit that referenced this issue Oct 16, 2024
…ntation (#44424)

### Rationale for this change

See #44360

### What changes are included in this PR?

* Adds a new integration test to allow testing `do_exchange` between C++/Python and .NET.
* Updates the Flight stream reader to handle when a descriptor is sent in the first message without any schema.

### Are these changes tested?

* Yes, using the new integration test.

### Are there any user-facing changes?

No
* GitHub Issue: #44360

Authored-by: Adam Reeve <adreeve@gmail.com>
Signed-off-by: Curt Hagenlocher <curt@hagenlocher.org>
@CurtHagenlocher CurtHagenlocher added this to the 19.0.0 milestone Oct 16, 2024
@CurtHagenlocher
Copy link
Contributor

Issue resolved by pull request 44424
#44424

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants