Support new versions of pyarrow in apache-beam #28410
Description
What happened?
Facing pyarrow issue on PPC4LE while executing below commands:
tox -c tox.ini run -e py311
` -- Found Python3: /beam/sdks/python/target/.tox/py311/bin/python (found version "3.11.2") found components: Interpreter Development.Module NumPy
-- Found Python3Alt: /beam/sdks/python/target/.tox/py311/bin/python
CMake Error at CMakeLists.txt:268 (find_package):
By not providing "FindArrow.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "Arrow", but
CMake did not find one.
Could not find a package configuration file provided by "Arrow" with any of
the following names:
ArrowConfig.cmake
arrow-config.cmake
Add the installation prefix of "Arrow" to CMAKE_PREFIX_PATH or set
"Arrow_DIR" to a directory containing one of the above files. If "Arrow"
provides a separate development package or SDK, be sure it has been
installed.
-- Configuring incomplete, errors occurred!
See also "/beam/sdks/python/target/.tox/py311/tmp/pip-install-9464v7kv/pyarrow_ef7786cb8dcf4c3a84231f093d6c09c1/build/temp.linux-ppc64le-cpython-311/CMakeFiles/CMakeOutput.log".
error: command '/usr/bin/cmake' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pyarrow
ERROR: Could not build wheels for cryptography, pyarrow, which is required to install pyproject.toml-based projects
py311: exit 1 (34.57 seconds) /beam/sdks/python> target/.tox/py311/bin/python /beam/sdks/python/target/.tox/py311/bin/pip install --retries 10 --pre 'cloudpickle~=2.2.1' 'crcmod<2.0,>=1.7' 'cryptography>=41.0.2' 'dill<0.3.2,>=0.3.1.1' 'fastavro<2,>=0.23.6' 'fasteners<1.0,>=0.3' 'freezegun>=0.3.12' 'grpcio!=1.48.0,<2,>=1.33.1' 'hdfs<3.0.0,>=2.1.0' 'httplib2<0.23.0,>=0.8' 'hypothesis<=7.0.0,>5.0.0' 'joblib>=1.0.1' 'mock<6.0.0,>=1.0.1' 'numpy<1.25.0,>=1.14.3' 'objsize<0.7.0,>=0.6.1' 'orjson<3.9.3' 'packaging>=22.0' 'pandas!=1.5.0,!=1.5.1,<1.6,>=1.4.3; python_version >= "3.8"' 'pandas<2.0.0' 'parameterized<0.10.0,>=0.7.1' 'proto-plus<2,>=1.7.1' 'protobuf<4.24.0,>=3.20.3' 'psycopg2-binary<3.0.0,>=2.8.5' 'pyarrow<12.0.0,>=3.0.0' 'pydot<2,>=1.2.0' 'pyhamcrest!=1.10.0,<3.0.0,>=1.9' 'pymongo<5.0.0,>=3.8.0' 'pytest-timeout<3,>=2.1.0' 'pytest-xdist<4,>=2.5.0' 'pytest<8.0,>=7.1.2' 'python-dateutil<3,>=2.8.0' 'pytz>=2018.3' 'pyyaml<7.0.0,>=3.12' 'regex>=2020.6.8' 'requests-mock<2.0,>=1.7' 'requests<3.0.0,>=2.24.0' 'scikit-learn>=0.20.0' 'sqlalchemy<2.0,>=1.3' 'tenacity<9,>=8.0.0' 'testcontainers[mysql]<4.0.0,>=3.0.3' 'typing-extensions>=3.7.0' 'zstandard<1,>=0.18.0' pid=60536
.pkg: _exit> python /usr/local/lib/python3.11/site-packages/pyproject_api/_backend.py True setuptools.build_meta legacy
py311: FAIL code 1 (37.57 seconds)
evaluation failed :( (37.77 seconds)
`
Issue Priority
Priority: 3 (minor)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Spark Runner
- Component: Flink Runner
- Component: Samza Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner