arrow-integration test failing on both C++ and Rust repos w/ Go

[alamb]

Describe the bug, including details regarding any error messages, version, and platform.

I am not sure this is actually a problem with arrow-go. Related to

• "Archery test With other arrows" Integration test failing on main: arrow-rs#8813

There are failures in the integration test in both the C++ repo as well as the Rust repo when interoperating with Go

Given Go is implicated in both, maybe there is something related to Go.

Here is an example failure in arrow: https://github.com/apache/arrow/actions/runs/19208648378/job/54907555826

 ################# FAILURES #################
  FAILED TEST: flight_sql:extension C++ producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /root/go/bin/arrow-flight-integration-client -host localhost -port=40325 -scenario flight_sql:extension
  With output:
  --------------
  panic: expected: schema:
  	  fields: 1
  	    - pkey: type=int32, nullable
  	      metadata: ["ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test", "ARROW:FLIGHT:SQL:REMARKS": "test column", "ARROW:FLIGHT:SQL:PRECISION": "100"], got: schema:
  	  fields: 1
  	    - pkey: type=int32, nullable
  	      metadata: ["ARROW:FLIGHT:SQL:TABLE_NAME": "test", "ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT": "1", "ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE": "0", "ARROW:FLIGHT:SQL:TYPE_NAME": "type_test", "ARROW:FLIGHT:SQL:SCHEMA_NAME": "schema_test", "ARROW:FLIGHT:SQL:IS_SEARCHABLE": "1", "ARROW:FLIGHT:SQL:CATALOG_NAME": "catalog_test", "ARROW:FLIGHT:SQL:PRECISION": "100", "ARROW:FLIGHT:SQL:REMARKS": "test column"]
  
  goroutine 1 [running]:
  main.main()
  	/arrow/go/arrow/internal/flight_integration/cmd/arrow-flight-integration-client/main.go:52 +0x277

Here is an example from arrow-rs: https://github.com/apache/arrow-rs/actions/runs/19208833571/job/54907987381

################# FAILURES #################
  FAILED TEST: custom_metadata Rust producing,  Go consuming
  <class 'RuntimeError'>: Command failed: /github/home/go/bin/arrow-json-integration-test -arrow /tmp/tmp2kbucd3t/9af639a7_generated_custom_metadata.json_as_file -json /tmp/arrow-integration-a1e3ey2k/generated_custom_metadata.json -mode VALIDATE
  With output:
  --------------
  arrow-json: JSON schema:
  schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["a": "{}", "b": "{}", "c": "{}", "d": "{}", "..": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
  Arrow schema:
  schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["..": "{}", "a": "{}", "b": "{}", "c": "{}", "d": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:metadata": "", "ARROW:extension:name": "!nonexistent", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
  arrow-json: schemas did not match
  
  --------------
  
  
  FAILED TEST: custom_metadata Rust producing,  Go consuming
  <class 'RuntimeError'>: Go C Data Integration call failed: Schemas are different:
  - Json Schema: schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["a": "{}", "b": "{}", "c": "{}", "d": "{}", "..": "{}", "w": "{}", "x": "{}", "y": "{}", "z": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_0": "{}", "schema_custom_1": "{}"]
  - Imported Schema: schema:
    fields: 4
      - sort_of_pandas: type=int8, nullable
                  metadata: ["pandas": "{}"]
      - lots_of_meta: type=int8, nullable
                metadata: ["x": "{}", "y": "{}", "w": "{}", "d": "{}", "a": "{}", "..": "{}", "c": "{}", "z": "{}", "b": "{}"]
      - unregistered_extension: type=int8, nullable
                          metadata: ["ARROW:extension:name": "!nonexistent", "ARROW:extension:metadata": "", "ARROW:integration:allow_unregistered_extension": "true"]
      - list_with_odd_values: type=list<item: int32, nullable>, nullable
    metadata: ["schema_custom_1": "{}", "schema_custom_0": "{}"]
  
  2 failures, 454 skips

real	2m16.517s
user	8m2.953s
sys	0m41.614s
ERROR conda.cli.main_run:execute(125): `conda run ci/scripts/integration_arrow.sh /__w/arrow-rs/arrow-rs /build` failed. (See above for error)

Component(s)

Integration

[zeroshade]

Figured out the issue, a previous change to improve custom metadata handling had a bug in it when comparing Metadata whose keys are in different orders under specific circumstances. Putting up a PR now with the fix!

[alamb]

Thanks for the quick resolution @zeroshade