Arrow FlightSQL client fails on s390x platform

[piotr-rachwalik]

Describe the bug, including details regarding any error messages, version, and platform.

We are retrieving data from Dremio server using Arrow Flight SQL client:

    for (final FlightEndpoint endpoint : flightInfo.getEndpoints()) {
        try (FlightStream stream = flightSqlClient.getStream(endpoint.getTicket(), options)) {
            while (stream.next()) {
               ...
            }
        }

On x86 architecture it works fine, but on s390x platform it fails with:

Caused by: java.lang.IndexOutOfBoundsException: index: -3690987520, length: 4 (expected: range(0, 808))
	at org.apache.arrow.memory.ArrowBuf.checkIndexD(ArrowBuf.java:320)
	at org.apache.arrow.memory.ArrowBuf.chk(ArrowBuf.java:307)
	at org.apache.arrow.memory.ArrowBuf.getInt(ArrowBuf.java:426)
	at org.apache.arrow.vector.BaseVariableWidthVector.getStartOffset(BaseVariableWidthVector.java:1314)
	at org.apache.arrow.vector.BaseVariableWidthVector.setReaderAndWriterIndex(BaseVariableWidthVector.java:398)
	at org.apache.arrow.vector.BaseVariableWidthVector.setValueCount(BaseVariableWidthVector.java:986)
	at org.apache.arrow.vector.complex.ListVector.setValueCount(ListVector.java:940)
	at org.apache.arrow.vector.complex.DenseUnionVector.setChildVectorValueCounts(DenseUnionVector.java:1322)
	at org.apache.arrow.vector.complex.DenseUnionVector.setValueCount(DenseUnionVector.java:1310)
	at org.apache.arrow.vector.VectorSchemaRoot.setRowCount(VectorSchemaRoot.java:240)
	at org.apache.arrow.vector.VectorLoader.load(VectorLoader.java:86)
	at org.apache.arrow.flight.FlightStream.next(FlightStream.java:270)

We are using Arrow in version: 16.0.0.

Component(s)

Java

[ravigumm]

@kiszk , Would you please provide your thoughts/inputs on this issue ? Thanks

[kiszk]

Thank you for raising an issue and getting me to the thread. It looks like an endianness issue.
It would be appreciated if you could provide a standalone reproduction program.

[piotr-rachwalik]

@kiszk it will be difficult to provide you a standalone reproduction program, because we are connecting to a Dremio server which serves as a Flight server.
I haven't mentioned before, but the scenario looks like this:

• Dremio server which serves as a Flight server runs on x86 machine
• The client side which uses Flight SQL client runs on s390x machine.

Perhaps the difference in endianness between the origin machine and the target machine is the root cause?

[ravigumm]

@piotr-rachwalik , Which version of arrow is in use on x86 side -- dremio server side ? May be newer version of arrow on x86 side could resolve this issue --- assuming that newer versions of arrow already has many fixes for big-endian s390x side issues ?

[piotr-rachwalik]

@ravigumm this is the version used on x86 side: https://docs.dremio.com/current/release-notes/dependencies
14.0.2-20240131150813-452ae43b2e-dremio
Unfortunately, I don't have a control over what Arrow version is used on the Dremio server side :/