[SPARK-45953][INFRA] Add Python 3.10 to Infra docker image

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to add Python 3.10 to Infra docker images.

Why are the changes needed?

This is a preparation to add a daily Python 3.10 GitHub Action job later for Apache Spark 4.0.0.

Note that Python 3.10 is installed at the last step to avoid the following issues which happens when we install Python 3.9 and 3.10 at the same stage by package manager.

#21 13.03 ERROR: Cannot uninstall 'blinker'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
#21 ERROR: process "/bin/sh -c python3.9 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn==1.1.*'" did not complete successfully: exit code: 1

Does this PR introduce any user-facing change?

No.

How was this patch tested?

1. I verified that the Python CI is not affected and still use Python 3.9.5 only.

========================================================================
Running PySpark tests
========================================================================
Running PySpark tests. Output is in /__w/spark/spark/python/unit-tests.log
Will test against the following Python executables: ['python3.9']
Will test the following Python modules: ['pyspark-errors']
python3.9 python_implementation is CPython
python3.9 version is: Python 3.9.5
Starting test(python3.9): pyspark.errors.tests.test_errors (temp output: /__w/spark/spark/python/target/fd967f24-3607-4aa6-8190-3f8d7de522e1/python3.9__pyspark.errors.tests.test_errors___zauwgy1.log)
Finished test(python3.9): pyspark.errors.tests.test_errors (0s)
Tests passed in 0 seconds

2. Pass Base Image Build step for new Python 3.10.

3. Since new Python 3.10 is not used in CI, we need to validate like the following.

$ docker run -it --rm ghcr.io/dongjoon-hyun/apache-spark-ci-image:master-6895105871 python3.10 --version
Python 3.10.13

$ docker run -it --rm ghcr.io/dongjoon-hyun/apache-spark-ci-image:master-6895105871 python3.10 -m pip freeze
alembic==1.12.1
annotated-types==0.6.0
blinker==1.7.0
certifi==2019.11.28
chardet==3.0.4
charset-normalizer==3.3.2
click==8.1.7
cloudpickle==2.2.1
contourpy==1.2.0
coverage==7.3.2
cycler==0.12.1
databricks-cli==0.18.0
dbus-python==1.2.16
deepspeed==0.12.3
distro-info==0.23+ubuntu1.1
docker==6.1.3
entrypoints==0.4
et-xmlfile==1.1.0
filelock==3.9.0
Flask==3.0.0
fonttools==4.44.3
gitdb==4.0.11
GitPython==3.1.40
googleapis-common-protos==1.56.4
greenlet==3.0.1
grpcio==1.56.2
grpcio-status==1.48.2
gunicorn==21.2.0
hjson==3.1.0
idna==2.8
importlib-metadata==6.8.0
itsdangerous==2.1.2
Jinja2==3.1.2
joblib==1.3.2
kiwisolver==1.4.5
lxml==4.9.3
Mako==1.3.0
Markdown==3.5.1
MarkupSafe==2.1.3
matplotlib==3.8.1
memory-profiler==0.60.0
mlflow==2.8.1
mpmath==1.3.0
networkx==3.0
ninja==1.11.1.1
numpy==1.26.2
oauthlib==3.2.2
openpyxl==3.1.2
packaging==23.2
pandas==2.1.3
Pillow==10.1.0
plotly==5.18.0
protobuf==3.20.3
psutil==5.9.6
py-cpuinfo==9.0.0
pyarrow==14.0.1
pydantic==2.5.1
pydantic_core==2.14.3
PyGObject==3.36.0
PyJWT==2.8.0
pynvml==11.5.0
pyparsing==3.1.1
python-apt==2.0.1+ubuntu0.20.4.1
python-dateutil==2.8.2
pytz==2023.3.post1
PyYAML==6.0.1
querystring-parser==1.2.4
requests==2.31.0
requests-unixsocket==0.2.0
scikit-learn==1.1.3
scipy==1.11.3
six==1.14.0
smmap==5.0.1
SQLAlchemy==2.0.23
sqlparse==0.4.4
sympy==1.12
tabulate==0.9.0
tenacity==8.2.3
threadpoolctl==3.2.0
torch==2.0.1+cpu
torcheval==0.0.7
torchvision==0.15.2+cpu
tqdm==4.66.1
typing_extensions==4.8.0
tzdata==2023.3
unattended-upgrades==0.1
unittest-xml-reporting==3.2.0
urllib3==2.1.0
websocket-client==1.6.4
Werkzeug==3.0.1
zipp==3.17.0

Was this patch authored or co-authored using generative AI tooling?

No.

[yaooqinn]

LGTM

draft

[dongjoon-hyun]

Thank you. Yes this is a draft. When I install python3.10 via package manager, it seems to break the existing python3.9 like the following.

#21 13.03 ERROR: Cannot uninstall 'blinker'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.
#21 ERROR: process "/bin/sh -c python3.9 -m pip install numpy 'pyarrow>=14.0.0' 'pandas<=2.1.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn==1.1.*'" did not complete successfully: exit code: 1

[dongjoon-hyun]

All Python tests passed.

[dongjoon-hyun]

Could you review this when you have some time, @viirya .
This is for better PySpark support in Apache Spark 4.0.0.

[dongjoon-hyun]

Thank you! Merged to master.