Circular dependencies between ``Sphinx`` and ``sphinxcontrib-*``

[bruno-digitbio]

Describe the bug

I'm not sure if this is something that the Sphinx team even is interested in explicitly supporting, but the latest release of the sphinxcontrib tooling cannot be used with rules_python (i.e. by users that use Bazel to install Sphinx).

We build our monorepo's docs at work with Sphinx, and today when running some automated tooling to bump package versions, I encountered issues like the following:

$ bazel build //...
ERROR: /home/bruno/.cache/bazel/_bazel_bruno/8f2b6fc37214e29f3e0fada1717a9254/external/pip_sphinx/BUILD.bazel:22:11: in py_library rule @pip_sphinx//:pkg: cycle in dependency graph:
    //building/python/sphinx:build_wrapper (ebf05eacb2537ac9057c50f99af724bc4fbe8ed0ce57ceeb546c8f679dec84e5)
.-> @pip_sphinx//:pkg (ebf05eacb2537ac9057c50f99af724bc4fbe8ed0ce57ceeb546c8f679dec84e5)
|   @pip_sphinxcontrib_devhelp//:pkg (ebf05eacb2537ac9057c50f99af724bc4fbe8ed0ce57ceeb546c8f679dec84e5)
`-- @pip_sphinx//:pkg (ebf05eacb2537ac9057c50f99af724bc4fbe8ed0ce57ceeb546c8f679dec84e5)

These errors can easily be solved by pinning the older versions at one patch number below the current one:

sphinxcontrib-applehelp==1.0.4
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-htmlhelp==2.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5

The error is caused by the fact that at some point since the last release of these packages, Sphinx>5 was added as an explicit dependency of these packages, creating the dependency loops demonstrated above. For example, compare the setup.py of serializainghtml at 1.1.5 to its new pyproject.toml. This particular breakage appears to originate here.

I recognize that most Python users are not also Bazel users, so I also created a minimal Git repo that reproduces the issue.

• <link to minimal repro repo>

The requirements_lock.txt in this repo was produced by:

$ conda create -n sphinx_repro python=3.10
$ conda activate sphinx_repro
$ pip install sphinx
$ pip freeze > requirements_lock.txt

Please let me know if there is interest in resolving this issue here, and if so if I can help in any way!

How to Reproduce

$ git clone git@github.com:brunobeltran/minimal-sphinx-repro.git
$ cd minimal-sphinx-repro
$ bazel build //...

Environment Information

Platform:              linux; (Linux-6.1.0-10-amd64-x86_64-with-glibc2.36)
Python version:        3.10.12 (main, Jul  5 2023, 18:54:27) [GCC 11.2.0])
Python implementation: CPython
Sphinx version:        7.1.2
Docutils version:      0.20.1
Jinja2 version:        3.1.2
Pygments version:      2.16.1

Sphinx extensions

No response

Additional context

• Bazel.

[bruno-digitbio]

After reading up on the conversation starting here, it seems like the cleanest workaround is the one described here, which would allow one to manually remove the back-dependencies of sphinxcontrib-* on Sphinx manually for each package.

This should work 100% cleanly for all Bazel users who know that they will only be using sphinxcontrib-* as a dependency of Sphinx (and not ever as a stand-alone package).

I'll leave it up to discussion here whether having the sphinxcontrib-* be independently installable is worth creating these circular dependencies at the wheel-level, but after reading some recent Python mailing list discussion, it seems like there is some consensus that this is just how Python packaging works.

[AA-Turner]

Hi Bruno,

Thank you for your detailed issue. I have no experience with Bazel or rules_python, so please suffer me in my ignorance.

In sphinxcontrib-XXX, I want to declare a minimum allowed version of Sphinx through the packaging system, to make things easier for users. I used the project.dependencies table within pyproject.toml to achieve this, as there isn't (to my knowledge) a way to declare "constraints" within a distributed package.

What would be a Bazel-compatible way of achieving this? I'm happy to explore solutions, but as you say, from my perspective "this is just how Python packaging works" -- I never considered that adding this would cause any issues.

Thanks,
Adam

[bruno-digitbio]

@AA-Turner sorry for the delay, I was in the process of vendoring several dozen more Python libraries at work, and wanted to see how pervasive this problem is before proposing any "solutions" here.

Of the ~400 dependencies we (transitively) rely on, only Sphinx, PyTorch, and Airflow have this problem of breaking Bazel's underlying expectations that all dependencies should form a DAG.

AFAICT, the only "Bazel-compliant" way to break this dependency cycle is to not allow sphinxcontrib-XXX to depend on Sphinx. There is no way to do this without breaking users' abilities to do

$ pip install sphinxcontrib-XXX

and get a completely-working package.

One workaround that would work would be to define

[project.optional-dependencies]
standalone = [
  "sphinx > 5",
]

then users that really wanted to just install a contrib package directly could manually do

$ pip install sphinxcontrib-XXX[standalone]

this would need to be paired with a runtime import check that prints a friendly error message if sphinx fails to import from sphinxcontrib (prompting the user to use [standalone] if needed).

That said, this is a large amount of faff, when I think the Bazel team has basically decided that an upcoming version of Bazel will allow users to explicitly annotate groups of pip packages with dependency cycles so that Bazel can "ignore" these errors the same way pip does---there is even a production-ready PR already open adding this feature.

Given these considerations, I will close this Issue on the assumption(s) that

• we are not interested in breaking this dependency cycle intrinsically
• other Bazel users will hopefully be able to find this issue via Google and quickly deduce the solution via the links provided
• we can always reopen/revisit if it this also becomes a pain for other people that rely on DAG-like dependencies (e.g. if Debian package maintainers try to package any of the sphinxcontrib-*).

If my first assumption is wrong and you don't mind the solution I propose above, feel free to re-open and I can shoot over a PR that implements that approach. Thanks for your time!

[jrmarino]

It's not just a "Bazel" thing. Any script that traces through dependencies trees will hit this. These commits broke the auto-generation of python packages at Ravenports. The workaround was to sed out the new Sphinx >= 5 lines out of the requirements before tracing the requirements. There are well over 400 python modules in Ravenports and the Sphinx family is the ONLY one that caused a circular dependency on itself.

But yes, if you want to leave Sphinx as a dependency of htmlhelp and friends, then removing those as dependencies of Sphinx will resolve the circular dependency.

[mtelka]

pipdeptree reports this as an issue too: #11614

[mtelka]

In sphinxcontrib-XXX, I want to declare a minimum allowed version of Sphinx through the packaging system, to make things easier for users. I used the project.dependencies table within pyproject.toml to achieve this, as there isn't (to my knowledge) a way to declare "constraints" within a distributed package.

Isn't possible to check the low Sphinx version boundary during runtime? For example directly in sphinxcontrib/applehelp/__init__.py. This won't be so convenient for users, but I believe it will solve issues like this one. Thank you.

[AA-Turner]

It seems this causes more consternation than expected. @bruno-digitbio would you still be willing to implement the [standalone] workaround in each of the 6 sphinxcontrib packages? Ideally we'd also set app.require_sphinx('5.0').

A

[phlax]

fwiw, we are also hitting this (using bazel)