pyarrow `RuntimeError: AppendRowGroups requires equal schemas`

[ianna]

Version of Awkward Array

2.6.4 (master branch)

pyarrow.version '11.0.0'

pandas.version '2.0.3'

Description and code to reproduce

I do not see the error in CI, so I assume there must be a conflict in my environment, because when I run:

% python -m pytest tests/test_2898_to_parquet_dataset.py

I get a RuntimeError: AppendRowGroups requires equal schemas.

Indeed, my installed version of pyarrow is 11.0.0, while the CI installs 16.1.0. Should we set a minimum required version?

[ianna]

It might be related to apache/arrow#31678?

[jpivarski]

Indeed, my installed version of pyarrow is 11.0.0, while the CI installs 16.1.0. Should we set a minimum required version?

There's a test that runs the minimal pyarrow version, which is 7.0.0:

awkward/.github/workflows/test.yml

Lines 56 to 59 in 3317b0c

	- python-version: '3.8'
	python-architecture: x64
	runs-on: ubuntu-latest
	dependencies-kind: minimal

awkward/requirements-test-minimal.txt

Line 3 in 3317b0c

pyarrow==7.0.0

So we are testing the minimal version. If there's an error that occurs for pyarrow 11 but not 7 or 16... that would be interesting. In that case, yes, we'd probably want to increase the minimum up to 16 or wherever we get a contiguous set of versions that pass.

[jpivarski]

No error in tests/test_2898_to_parquet_dataset.py with pyarrow 10.0.1 on MacOS...