Upgrade sphinx and related dependencies

[shahar1]

closes: #31963
related: #39449

This PR update the following dependencies:

• sphinx to v7+ (there's already v8, but it is supported only for Python 3.10)
• docutils to v0.21+ (haven't encountered any <section> tags as mentioned in the deleted inline comment)
• sphinx-autoapi to v3+ and astroid to v3+ (possible now that sphinx is v7+)
• sphinxcontrib-serializinghtml to v1.1.5+ (pinning it causes dependency conflicts)

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[shahar1]

Drafting, weirdly on v2-10-test it worked fine but here there are errors

[potiuk]

one more thing - last time when we tried to upgrade sphinx and docutils, the documentation looked ....weird .... lots of extra whitespace

[shahar1]

Apparently it is a quite a tough task, here are some insights:

• A requirement for upgrading Sphinx to v7 is to merge [BREAKING CHANGE] Update link tag for compatibility with Sphinx v7 airflow-site#1103 and create a new version for sphinx_airflow_theme (0.1.0). I struggled with mounting the changes locally, so for now, I made a workaround by injecting the contents of the file that was modified via Dockerfile.ci (created an issue for anyone who wants to tackle it in the future).

• When running only the build docs workflow, locally and on GitHub Actions, using breeze build-docs apache-airflow docker-stack --docs-only --clean-build, the build succeeds without any errors (see GitHub Action workflow). When running the docs server locally, the extra spaces issue indeed persists - we need to find out how to modify the related CSS (no success with it so far).
  

• When running only the spellchecks workflow, locally and on GitHub Actions, using breeze build-docs apache-airflow docker-stack --spellchecks-only --clean-build (see raw logs - BIG FILE warning) - we get a lot of weird errors. If you take a look at the logs, it often fails when parsing from /opt/airflow/providers/src/airflow/providers/amazon/aws/hooks/ssm.py::SsmHook:: get_parameter_value. Edit: it happened due to missing words in the spellcheck wordlist (see Update spelling wordlist #45579) - now it should be fine.

• When running the workflow for building docs and spelling altogether (breeze build-docs apache-airflow docker-stack --clean-build), the build fails due to a large number of implicit errors (which might be related to the failure of the spelling checks?).

• The MyPy workflow fails, but it doesn't seem too bad - in the worst case we just ignore it.

I'll be happy for some help in solving some of the riddles above, it will gets us closer to upgrading to Sphinx 7 :)

[potiuk]

#protm

[potiuk]

It needs conflict resolution though :)

[jscheffl]

For me "good enough" to go.

[kaxil]

Merge conflicts on hatch_build.py @shahar1 and then hopefully we are good 🤞

[shahar1]

Merge conflicts on hatch_build.py @shahar1 and then hopefully we are good 🤞

Resolved conclicts, I'd be happy if @potiuk could review before merging :)

[jscheffl]

Good to merge in my view.

[potiuk]

LGTM ! 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

[github-actions]

Backport failed to create: v2-10-test. View the failure log Run details

Status	Branch	Result
❌	v2-10-test

You can attempt to backport this manually by running:

cherry_picker a51ebe4 v2-10-test

This should apply the commit to the v2-10-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue