[SPARK-52224][CONNECT][PYTHON] Introduce pyyaml as a dependency for the Python client

[sryza]

What changes were proposed in this pull request?

Introduces pyyaml as a dependency for the Python client. When pip install-ing the pyspark client, it will be installed with it.

Why are the changes needed?

The pipeline spec file described in the Declarative Pipelines SPIP expects data in a YAML format. YAML is superior to alternatives, for a few reasons: 

• Unlike the flat files that are used for spark-submit confs, it supports the hierarchical data required by the pipeline spec.
• It's much more user-friendly to author than JSON.
• It's consistent with the config files used for similar tools, like dbt.

The Declarative Pipelines CLI will be a Spark Connect Python client, and thus require a Python library for loading YAML. The pyyaml library is an extremely stable dependency. The safe_load function that we'll use to load YAML files was introduced more than a decade ago.

Does this PR introduce any user-facing change?

Yes – users who pip install the PySpark client library will see the pyyaml library installed.

How was this patch tested?

• Made a clean virtualenv
• Ran pip install python/packaging/client
• Confirmed that I could import yaml in a Python shell

Was this patch authored or co-authored using generative AI tooling?

No

[grundprinzip]

You need 2 more places to edit the setup.py

1. python/packaging/connect/setup.py
2. python/packaging/classic/setup.py -> here only for the connect additional requirements

[sryza]

Thanks @grundprinzip – I added those too