[Python] Add version suffix to libarrow_python* libraries

[eltoder]

Describe the enhancement requested

pyarrow wheels ship a number of shared libraries. Most of these libraries have a version suffix, but the "python" libraries do not:

$ readelf -d .venv/lib/python3.10/site-packages/pyarrow/*.so* | grep "NEEDED.*arrow" | sort -u
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_acero.so.1800]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_dataset.so.1800]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_flight.so.1800]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_python_flight.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_python_parquet_encryption.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_python.so]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow.so.1800]
 0x0000000000000001 (NEEDED)             Shared library: [libarrow_substrait.so.1800]

Version suffixes are nice, because they prevent us from loading an incompatible version of the library if it somehow happens to be present in LD_LIBRARY_PATH. (Note that LD_LIBRARY_PATH takes precedence over RUNPATH entries added during linking.)

It would be good to add version suffixes for libarrow_python* libraries as well -- for the same reason. We currently have a situation where (for hard to fix reasons) we end up with LD_LIBRARY_PATH that contains an old install of pyarrow. This breaks installing newer pyarrow into venvs -- even though the correct version of python scripts and lib.cpython-310-x86_64-linux-gnu.so is used, the latter picks up the old libarrow_python.so and fails to resolve symbols.

Component(s)

Python

[pitrou]

cc @raulcd

[assignUser]

Issue resolved by pull request 44702
#44702