GH-44361: [C#][Integration] Include .NET in Flight integration tests

[adamreeve]

Rationale for this change

See #44361. This allows testing compatibility of the .NET Flight implementation with other Flight implementations.

What changes are included in this PR?

• Adds a new Apache.Arrow.Flight.IntegrationTest project that can run in server or client mode for Flight integration tests.
• Includes the integration tests that send then retrieve data defined in JSON files, but doesn't add any of the named scenarios
• Configures archery to include C# in the Flight integration tests, but skip all the named scenarios
• Also skips tests that use dictionary data due to [C#] Add support for sending dictionary arrays via flight #38045, and the empty data test due to [C#] Flight client does not allow putting zero batches #44363

Are these changes tested?

These changes are tests.

Are there any user-facing changes?

No

• GitHub Issue: [C#][Integration] Set up integration testing of C# Flight #44361

[adamreeve]

This isn't required for any of the included tests, but before disabling the primitive_no_batches test, it would crash here with a NullReferenceException due to the writer being disposed before the stream was created.

With this fix, writing doesn't crash, but the data isn't found when trying to retrieve it.

[CurtHagenlocher]

This is awesome; I didn't even know this test gap existed.

By the way, I have a proposed fix for #38045 at https://github.com/CurtHagenlocher/arrow/tree/dev/curth/FlightDictionaries which I didn't feel great about committing due to lack of test coverage. It'll be interesting to see whether it works.

[adamreeve]

There was a failure running the new tests in CI. Surprisingly it's a C++ and C# test, which was the only combination I tested locally... I'll try to figure out what's gone wrong here:

   FAILED TEST: binary_view C++ producing,  C# consuming
  <class 'RuntimeError'>: Command failed: /arrow/csharp/artifacts/Apache.Arrow.Flight.IntegrationTest/Debug/net8.0/Apache.Arrow.Flight.IntegrationTest client --port 37771 --path /tmp/arrow-integration-be07w72w/generated_binary_view.json
  With output:
  --------------
  Unhandled exception: Grpc.Core.RpcException: Status(StatusCode="NotFound", Detail="Could not find flight./tmp/arrow-integration-be07w72w/generated_binary_view.json")
     at Apache.Arrow.Flight.Client.FlightClient.<>c.<<GetInfo>b__10_0>d.MoveNext() in /arrow/csharp/src/Apache.Arrow.Flight/Client/FlightClient.cs:line 90
  --- End of stack trace from previous location ---
     at Apache.Arrow.Flight.IntegrationTest.JsonTestScenario.RunClient() in /arrow/csharp/test/Apache.Arrow.Flight.IntegrationTest/JsonTestScenario.cs:line 73
     at Apache.Arrow.Flight.IntegrationTest.FlightClientCommand.Execute() in /arrow/csharp/test/Apache.Arrow.Flight.IntegrationTest/FlightClientCommand.cs:line 49
     at Apache.Arrow.Flight.IntegrationTest.Program.<>c.<<Main>b__0_0>d.MoveNext() in /arrow/csharp/test/Apache.Arrow.Flight.IntegrationTest/Program.cs:line 50
  --- End of stack trace from previous location ---
     at System.CommandLine.Invocation.AnonymousCommandHandler.InvokeAsync(InvocationContext context)
     at System.CommandLine.Invocation.InvocationPipeline.<>c__DisplayClass4_0.<<BuildInvocationChain>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass17_0.<<UseParseErrorReporting>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass12_0.<<UseHelp>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass22_0.<<UseVersionOption>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass19_0.<<UseTypoCorrections>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<UseSuggestDirective>b__18_0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass16_0.<<UseParseDirective>b__0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c.<<RegisterWithDotnetSuggest>b__5_0>d.MoveNext()
  --- End of stack trace from previous location ---
     at System.CommandLine.Builder.CommandLineBuilderExtensions.<>c__DisplayClass8_0.<<UseExceptionHandler>b__0>d.MoveNext()

[adamreeve]

By the way, I have a proposed fix for #38045 at https://github.com/CurtHagenlocher/arrow/tree/dev/curth/FlightDictionaries which I didn't feel great about committing due to lack of test coverage. It'll be interesting to see whether it works.

Cool, it would be nice to fix that!

[adamreeve]

There was a race condition where the C# client could try to request the data before the server stored it here:

arrow/cpp/src/arrow/flight/integration_tests/test_integration_server.cc

Line 153 in dcc1ee5

uploaded_chunks[key] = dataset;

I could reliably reproduce it by adding a small sleep in the C++ server before that line.

Reading to the end of the response stream in the client seems to fix it.

[CurtHagenlocher]

Thanks, this is great!

I'll give some people more familiar with the infrastructure some time to comment before committing.

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 1dcd145.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details.