readthedoc from pyproject.toml

[IaroslavR]

Hi. Thank you for this great " Hypermodern Python" series.
With setup like this:

# pyproject.toml
...
[tool.poetry.dependencies]
# project deps
...
python = "<3.8,>=3.7"
# dev deps
black = { version = "^19.10b0", optional = true }
coverage = {version = "^5.1", optional = true }
...
sphinx-rtd-theme = { version = "^0.4.3", optional = true }
toml = { version = "^0.10.0", optional = true }
typeguard = { version = "^2.7.1", optional = true }

[tool.poetry.extras]
dev = ["black", "coverage", ... "toml", "typeguard"]
docs = ["m2r", "sphinx", "sphinx-autodoc-typehints", "sphinx-rtd-theme", "toml"]

# readthedocs.yml
version: 2

sphinx:
  configuration: docs/conf.py

python:
  version: 3.7
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

we can

• read pyproject.toml from docs/conf.py for metadata
• use pyproject.toml "as is" without additional docs/requirements.txt file

[cjolowicz]

That's right, but the docs dependencies won't be pinned then. Read the Docs builds your project with pip, and pip uses Poetry to build a wheel, not for installing. The wheel does not include version pins.

The idea to read pyproject.toml from docs/conf.py for metadata is interesting. Do you have an example? If you install your package for building docs, you could also use importlib.metadata for that, though.

Many projects use extras for development dependencies, but I'm not a fan of that. I think extras are best used for optional user-facing features. Extras force you to install the package itself, even though you don't need it for many development tasks (e.g. linting, type-checking, code formatting, other static analysis, docs building if you don't need autodoc).

[IaroslavR]

Do you have an example?

https://github.com/IaroslavR/multiconsumers-queue/blob/master/docs/conf.py

[IaroslavR]

That's right, but the docs dependencies won't be pinned then.

See docs build log for the repo above https://readthedocs.org/api/v2/build/10913417.txt Looks like all deps pinned

[cjolowicz]

Reading the package metadata in docs/conf.py is a great idea! 🚀

As I said earlier, take a look at importlib.metadata. It allows you to read the package metadata directly from your installed package, the way every other packaging tool sees it, using only the standard library, without any need for parsing.

To use it, you also need to configure Read the Docs to use Python 3.8 via .readthedocs.yml.

Looks like all deps pinned

I was really hoping I would be wrong 😄But the deps are not pinned. Direct dependencies are constrained to major versions, as specified in pyproject.toml. Indirect dependencies ("subdependencies") are constrained by whatever depends on them, if at all. None of the dependencies are locked to a specific version.

Took me a while to find the right pip invocation in the log file, so I pasted it below:

pip output

/home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/bin/python -m pip install --upgrade --upgrade-strategy eager --no-cache-dir .[docs]
Processing /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/checkouts/stable
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
Collecting loguru<0.5.0,>=0.4.1
  Downloading loguru-0.4.1-py3-none-any.whl (54 kB)
Collecting attrs<20.0.0,>=19.3.0
  Downloading attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting arrow<0.16.0,>=0.15.5
  Downloading arrow-0.15.5-py2.py3-none-any.whl (46 kB)
Collecting toml<0.11.0,>=0.10.0; extra == "dev" or extra == "docs"
  Downloading toml-0.10.0-py2.py3-none-any.whl (25 kB)
Collecting sphinx-autodoc-typehints<2.0.0,>=1.10.3; extra == "docs"
  Downloading sphinx_autodoc_typehints-1.10.3-py3-none-any.whl (8.4 kB)
Collecting m2r<0.3.0,>=0.2.1; extra == "docs"
  Downloading m2r-0.2.1.tar.gz (16 kB)
Requirement already up-to-date: sphinx-rtd-theme<0.5.0,>=0.4.3; extra == "docs" in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from multiconsumers-queue==0.1.4) (0.4.3)
Collecting sphinx==2.3.1; extra == "docs"
  Downloading Sphinx-2.3.1-py3-none-any.whl (2.7 MB)
Collecting python-dateutil
  Downloading python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting mistune
  Downloading mistune-0.8.4-py2.py3-none-any.whl (16 kB)
Collecting docutils
  Downloading docutils-0.16-py2.py3-none-any.whl (548 kB)
Collecting sphinxcontrib-serializinghtml
  Downloading sphinxcontrib_serializinghtml-1.1.4-py2.py3-none-any.whl (89 kB)
Collecting sphinxcontrib-applehelp
  Downloading sphinxcontrib_applehelp-1.0.2-py2.py3-none-any.whl (121 kB)
Collecting Pygments>=2.0
  Downloading Pygments-2.6.1-py3-none-any.whl (914 kB)
Collecting sphinxcontrib-qthelp
  Downloading sphinxcontrib_qthelp-1.0.3-py2.py3-none-any.whl (90 kB)
Requirement already up-to-date: alabaster<0.8,>=0.7 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (0.7.12)
Collecting setuptools
  Downloading setuptools-46.1.3-py3-none-any.whl (582 kB)
Collecting sphinxcontrib-devhelp
  Downloading sphinxcontrib_devhelp-1.0.2-py2.py3-none-any.whl (84 kB)
Collecting sphinxcontrib-jsmath
  Downloading sphinxcontrib_jsmath-1.0.1-py2.py3-none-any.whl (5.1 kB)
Requirement already up-to-date: Jinja2>=2.3 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (2.11.2)
Requirement already up-to-date: requests>=2.5.0 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (2.23.0)
Collecting sphinxcontrib-htmlhelp
  Downloading sphinxcontrib_htmlhelp-1.0.3-py2.py3-none-any.whl (96 kB)
Requirement already up-to-date: babel!=2.0,>=1.3 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (2.8.0)
Requirement already up-to-date: snowballstemmer>=1.1 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (2.0.0)
Requirement already up-to-date: packaging in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (20.3)
Requirement already up-to-date: imagesize in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (1.2.0)
Requirement already up-to-date: six>=1.5 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from python-dateutil->arrow<0.16.0,>=0.15.5->multiconsumers-queue==0.1.4) (1.14.0)
Requirement already up-to-date: MarkupSafe>=0.23 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from Jinja2>=2.3->sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (1.1.1)
Requirement already up-to-date: certifi>=2017.4.17 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from requests>=2.5.0->sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (2020.4.5.1)
Requirement already up-to-date: chardet<4,>=3.0.2 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from requests>=2.5.0->sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (3.0.4)
Requirement already up-to-date: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from requests>=2.5.0->sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (1.25.9)
Requirement already up-to-date: idna<3,>=2.5 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from requests>=2.5.0->sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (2.9)
Requirement already up-to-date: pytz>=2015.7 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from babel!=2.0,>=1.3->sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (2019.3)
Requirement already up-to-date: pyparsing>=2.0.2 in /home/docs/checkouts/readthedocs.org/user_builds/multiconsumers-queue/envs/stable/lib/python3.7/site-packages (from packaging->sphinx==2.3.1; extra == "docs"->multiconsumers-queue==0.1.4) (2.4.7)
Building wheels for collected packages: multiconsumers-queue, m2r
  Building wheel for multiconsumers-queue (PEP 517): started
  Building wheel for multiconsumers-queue (PEP 517): finished with status 'done'
  Created wheel for multiconsumers-queue: filename=multiconsumers_queue-0.1.4-py3-none-any.whl size=6392 sha256=6587f695831f380ca7037a45ea084155ed84fb0fc05dc205e60908b334ec9027
  Stored in directory: /tmp/pip-ephem-wheel-cache-o22648kv/wheels/0f/ff/96/13f72ca46f66c60bdc0fd096b28d7c93d5ba1b51adec72aab5
  Building wheel for m2r (setup.py): started
  Building wheel for m2r (setup.py): finished with status 'done'
  Created wheel for m2r: filename=m2r-0.2.1-py3-none-any.whl size=10466 sha256=6be74e69404760dbde46648fb6c0f385c87d33014abfa40a18d35a700c625897
  Stored in directory: /tmp/pip-ephem-wheel-cache-o22648kv/wheels/02/47/3a/e1c46c2cca442c8781612542397c9559a579f10e2dd87e7c9f
Successfully built multiconsumers-queue m2r
Installing collected packages: loguru, attrs, python-dateutil, arrow, toml, sphinxcontrib-serializinghtml, sphinxcontrib-applehelp, Pygments, sphinxcontrib-qthelp, setuptools, sphinxcontrib-devhelp, sphinxcontrib-jsmath, sphinxcontrib-htmlhelp, docutils, sphinx, sphinx-autodoc-typehints, mistune, m2r, multiconsumers-queue
  Attempting uninstall: Pygments
    Found existing installation: Pygments 2.3.1
    Uninstalling Pygments-2.3.1:
      Successfully uninstalled Pygments-2.3.1
  Attempting uninstall: setuptools
    Found existing installation: setuptools 41.0.1
    Uninstalling setuptools-41.0.1:
      Successfully uninstalled setuptools-41.0.1
  Attempting uninstall: docutils
    Found existing installation: docutils 0.14
    Uninstalling docutils-0.14:
      Successfully uninstalled docutils-0.14
  Attempting uninstall: sphinx
    Found existing installation: Sphinx 1.8.5
    Uninstalling Sphinx-1.8.5:
      Successfully uninstalled Sphinx-1.8.5
Successfully installed Pygments-2.6.1 arrow-0.15.5 attrs-19.3.0 docutils-0.16 loguru-0.4.1 m2r-0.2.1 mistune-0.8.4 multiconsumers-queue-0.1.4 python-dateutil-2.8.1 setuptools-46.1.3 sphinx-2.3.1 sphinx-autodoc-typehints-1.10.3 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 toml-0.10.0

[IaroslavR]

Yep, I see. By idea we can install it from conf.py this way, with the help of install_with_constraints trick from your noxfile.py, but IMO it's a overkill for the most of doc build tasks :)

[aryarm]

Hi @cjolowicz ,

Thanks for creating this resource! I've learned a lot from reading your blog posts and the hypermodern-python website.

I recently stumbled upon this comment describing another strategy for integrating poetry dependencies into a readthedocs build. I haven't tried it myself yet, but I think it might read the versions from poetry's lock file? Would this be a viable option?