Use the latest Sphinx version

[AA-Turner]

Documentation

Python's documentation uses the Sphinx toolchain. Originally written for the Python documentation, it is now developed and distributed as an independent project. The two remain closely linked, e.g. when new syntax is added to the Python language. To document this syntax in the best way, Sphinx must support it and Python must require a version of Sphinx that supports the syntax.

Using the new syntax in PEP-695 as an example:

1. The implementation was committed on 16 May (gh-103763: Implement PEP 695 #103764)
2. The first draft of the documentation was written on 19 May (gh-103921: Document PEP 695 #104642)
3. A feature request was created in Sphinx on 23 May (Support PEP 695 syntax for class definitions sphinx-doc/sphinx#11438)
4. Sphinx added support on 23 July (Support PEP 695 and PEP 696 syntax in py:function and py:class directives. sphinx-doc/sphinx#11444)
5. Sphinx published a release with support on 24 July (v7.1.0)
6. The Python project can use those features, 19 October [+ 1 year] (GH-125277: Increase minimum supported Sphinx to 7.2.6 #125368)

This is a ~two month window from implementation to support in a release of Sphinx. It took a further 15 months and two feature releases (3.12, 3.13) until these features can be used in Python. Due to this, our documentation is meaningfully worse for readers and programmers. Using older versions of Sphinx mean that we cannot succinctly cover the features and syntax that exist within released versions of Python.

Core developers responsible for these features have expressed interest, but have been hampered by our self-imposed restriction of the minimum version of Sphinx that we support.

We adopt these restrictions for the benefit of downstream Linux redistributors, as can be seen in the more recent issues in the summary table below. This has not always been the case. From the adoption of Sphinx in 2007 (8ec7f65 / 116aa62) until 2014 (f7b2f36), the latest source-tree checkout of Sphinx at https://svn.python.org/ was used. From then until ~2018 the minimum required Sphinx version (controlled by needs_sphinx in conf.py) tracked the latest release promptly. This has since ossified.

In a recent informal discussion, several committers supported the idea of removing or relaxing this restriction, allowing the Python documentation to use the latest version(s) of Sphinx. This is the context for this note, which for now is a proposal. The status quo will prevail if there is not sufficient support.

As a concrete suggestion, I propose that when evaulating increasing the minimum required version of Sphinx, we no longer consult downstream redistributors' Sphinx versions. The procedure will follow standard Python development processes in all other ways. I expect that the minimum version would be updated if and when a new Sphinx feature is of sufficient benefit to Python. This, though, may be a greater version than a downstream redistributor provides.

We would like to solicit views from representatives of downstream redistributors as to how (in)feasible this proposal would be. My understanding is that Fedora and SUSE have processes whereby a newer version of Sphinx can be used soley for the Python documentation. I do not know how this will impact Debian or Gentoo.

Thank you in advance for your consideration,
Adam

Table of past changes to needs_sphinx

needs_sphinx	Issue	PR / Commit	Latest Sphinx 1
7.2.6	#125277	#125368	8.1.3
6.2.1	#117928	#121986	7.4.6
4.2	#109209	#109210	7.2.6
3.2	#86986	#93337	5.0.1
1.8	#80188	#11887	1.8.4
1.7	doc-sig	#9423	1.8.1
1.2	#65630	90d76ca	1.2.3
1.1	#64860	f7b2f36	1.2.1

Previous discussion relating to the minumum version:

• Docs build error with Sphinx 3.0 due to invalid C declaration #84385
• What min_sphinx for Python 3.10 #87009
• Require Sphinx 6.2 to build Python 3.13 documentation #104818

cc:

• @doko42 @mitya57 / Debian
• @hroncok / Fedora / RHEL
• @mgorny / Gentoo
• @danigm @mcepl / openSUSE
• @AA-Turner @hugovk / CPython

Linked PRs

• GH-125722: Increase minimum supported Sphinx to 8.1.3 #128922
• [3.13] GH-125722: Increase minimum supported Sphinx to 8.1.3 (GH-128922) #129037
• [3.12] GH-125722: Increase minimum supported Sphinx to 8.1.3 (GH-128922) #129038
• GH-125722: Use long options for Sphinx #129039
• [3.13] GH-125722: Use long options for Sphinx (GH-129039) #129041
• [3.12] GH-125722: Use long options for Sphinx (GH-129039) #129042
• GH-125722: Remove Sphinx patches #129277
• [3.13] GH-125722: Remove Sphinx patches (GH-129277) #129278
• [3.12] GH-125722: Remove Sphinx patches (GH-129277) #129279
• GH-121970: Remove pyspecific PyDecoratorMixin patches #129306
• GH-125722: Increase minimum supported Sphinx to 8.2.0 #130444
• [3.13] GH-125722: Increase minimum supported Sphinx to 8.2.0 (GH-130444) #130858
• [3.12] GH-125722: Increase minimum supported Sphinx to 8.2.0 (GH-130444) #130859

Footnotes

1. At commit date. ↩

[willingc]

@AA-Turner It would be helpful to have the latest version of Sphinx build the latest release of CPython. I suspect most users will view documentation on python.org.

[AA-Turner]

Yep, and we currently use the latest version of Sphinx on docs.python.org (see the footer for "Created using ..."). This proposal would in effect mean we can narrow the range between the minimum supported and what we use in Doc/requirements.txt.

A

[hroncok]

From the Fedora point of view, we have no trouble allowing the latest/development Python 3.X version documentation to use the latest version of Sphinx. My understanding it that this is usually the only Python version that needs to use the new features (such as new Python syntax). Once the features of a Python branch are stable, the required Sphinx version IMHO should also remain stable.

In the past, you proposed bumping the minimum Sphinx version for older Python versions, such as 3.12. That might sometimes be problematic for us. That said, your communication wrt. such bumps was always appreciated but never taken for granted. If you decide to always use the latest Sphinx for Python 3.12, 3.12, etc. at this point, it might be a bit challenging for us, but it won't be a blocker.

[AA-Turner]

Thank you @hroncok, this is good to know.

In the past, you proposed bumping the minimum Sphinx version for older Python versions, such as 3.12. That might sometimes be problematic for us. [...] If you decide to always use the latest Sphinx for Python 3.12, 3.12, etc. at this point, it might be a bit challenging for us, but it won't be a blocker.

Our development workflow means that we backport almost all documentation PRs to stable versions of Python (currently 3.12 & 3.13). This is mainly so that the website at https://docs.python.org/3/ shows the latest documentation, as /3/ is a symlink to the most recent stable (currently /3.13).

This would mean that we would increase the minimum version for both HEAD and also all bugfix branches (or at least the most recent bugfix branch). The proposal wouldn't have much value without this, as it would effectively prohibit us from being able to confidently backport PRs.

Hopefully this is a little clearer. I think that it falls into the 'challenging but manageable' category of your response, if I'm understanding correctly.

A

[AA-Turner]

@doko42 @mitya57 @mgorny would this proposal cause significant issues for Debian or Gentoo?

A

[mitya57]

It can cause some issues.

When a new major Sphinx version is released, it usually breaks a lot of reverse dependencies, in particular third-party extensions and themes. So I cannot upload it to Debian immediately, instead I need to wait a few months to give upstreams time to fix these issues. In many cases I have to submit my own PRs to various projects to support new Sphinx versions, to make them build fine in Debian.

For example, here is a list of packages which were broken by Sphinx 7 release. The test rebuild was done after Sphinx 7.1 was released, so quite some time has already passed since 7.0:
https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=sphinx7.1;users=python-modules-team@lists.alioth.debian.org

To give another example: Sphinx 8.0 was released on Jul 29, 2024, but sphinx-rtd-theme (which is one of the most popular Sphinx themes) made a release with Sphinx 8.0 support only on Oct 7, 2024.

So if a new Python version requires a Sphinx version which was released a month or two earlier, it will be a problem. If more time has passed (like 4-6 months), then it is probably okay.

[AA-Turner]

Thanks @mitya57. If we adopted this, it would mean using the latest Sphinx version for all stable Python versions (e.g. currently 3.12 and 3.13, and git HEAD (to-become 3.14)). I don't know how up-to-date Debian keeps the system Python (e.g. stable has 3.11), as Python 3.12 will fall into security-only in April 2025 (3.13: Oct 2026), meaning it would be frozen in terms of Sphinx.

Would this sort of timeline be likely to affect Debian? We could adopt a policy of 3 <= N <= 6 months, but that doesn't really achieve the goals cited in the initial post.

A

[mgorny]

@doko42 @mitya57 @mgorny would this proposal cause significant issues for Debian or Gentoo?

Not for Gentoo, since we use prebuilt docs right now (and in general try not to stay behind with Sphinx much, at the code of losing docs in packages that don't fix their builds).

[mitya57]

Regarding updates to stable Python branches: good news is that we usually don’t upload patch releases of Python to Debian stable, instead we cherry-pick selected fixes. For example, Debian stable has Python 3.11.2 + patches, not 3.11.11.

Bad news is that we do it for Ubuntu: for example, Ubuntu 22.04 (jammy) was released with Python 3.10.4 but it was later updated to 3.10.12. And updating Sphinx to new major version is impossible in an Ubuntu stable release for the same reason I mentioned above (it breaks many other packages). So if this policy is accepted, Ubuntu will have to choose: not to update Python, or to update Python but revert changes that require new Sphinx, or to not build documentation at all.

@doko42 knows better about Python stable updates in Debian/Ubuntu, maybe he will correct me.

[doko42]

I don't have do add more for the Debian perspective. However I do backports of Python point release in Ubuntu. That would make also backports of an additional sphinx package, and backports of additional sphinx extensions necessary. If I would have a choice, I'd like to avoid with these backports, and stay with the same major sphinx version.