Make Sphinx autodoc work

[amotl]

We found at #153 (comment) that sphinx.ext.autodoc does not work on RTD yet, see Module library > API.

From the past, I know this is mostly a matter of getting the package itself installed properly into the Python environment where Sphinx will be building the documentation. When doing that, there's usually no way around also installing all its dependencies, which occasionally has problems within (restricted) CI environments.

[amotl]

I've tried to follow the guidelines at readthedocs/readthedocs.org#4912 (comment) and also had a look at python-poetry/poetry#754.

[amotl]

Investigation

TLDR; There might be an OOM issue when installing the project and its dependencies, which is required for a Autodoc environment to work out of the box. Apparently, there is also Sphinx AutoAPI, which does not have these requirements.

Details

There's a comment by @stsewd at readthedocs/readthedocs.org#5847 (comment):

So, the autodoc feature of sphinx requires the project to be installed to extract the docs (because it actually executes the code). Autoapi extracts the docs in a static way (no need to run the code). So, when using autoapi, you can skip the dependencies installation for the project.

After @tlvu commented at readthedocs/readthedocs.org#5847 (comment):

I guess I am not the only project with OOM problem because of the awsome sphinx autodoc feature. I solved my issue by mocking import (using autodoc_mock_imports in Sphinx's conf.py file, see my PR Ouranosinc/raven#167) of all the 3rd party dependencies so they do not have to be installed for real and by adding my own sources into the sys.path for autodoc to parse for docstrings and auto generate the doc. I basically work-around the requirement to install my project and all its dependencies, resulting in less memory usage and faster build.

You might want to document this somewhere on ReadTheDocs docs so other users using sphinx autodoc will benefit and prevent other similar OOM support call in the future.

Also, @humitos chimed in at readthedocs/readthedocs.org#5847 (comment):

On the other hand, we recommend to use autoapi if what you want is to auto document the modules. It's static and does not need to install anything to get the modules documentation 😉

[amotl]

However, in our case https://readthedocs.org/projects/wetterdienst/builds/ currently does not show any build errors at all, right?

[amotl]

When investigating the latest build at https://readthedocs.org/projects/wetterdienst/builds/11887538/ more thoroughly, these things can be observed:

• It looks like pip install . or an equivalent is not invoked at all.
• sphinx-material 0.0.30 requires sphinx>=2.0, but you'll have sphinx 1.8.5 which is incompatible.. However, this part actually seems to work in general.
• WARNING: autodoc: failed to import function 'discover_climate_observations' from module 'wetterdienst'; the following exception was raised: No module named 'pandas'.

[amotl]

I've tried to reproduce this on my machine by using a fresh Poetry environment for this project after wiping the old one using poetry env remove wetterdienst-EkOFQaO8-py3.8.

Then, I tried to go for the module mocking thing using autodoc_mock_imports, see also FAQ: I get import errors on libraries that depend on C modules.

After some complaints leading to f71ed0e and 34433d9, things started to work. Enjoy https://wetterdienst.readthedocs.io/en/latest/pages/library/api.html.

[amotl]

I've tried to follow the guidelines at readthedocs/readthedocs.org#4912 (comment) and also had a look at python-poetry/poetry#754.

[amotl]

Investigation

TLDR; There might be an OOM issue when installing the project and its dependencies, which is required for a Autodoc environment to work out of the box. Apparently, there is also Sphinx AutoAPI, which does not have these requirements.

Details

There's a comment by @stsewd at readthedocs/readthedocs.org#5847 (comment):

So, the autodoc feature of sphinx requires the project to be installed to extract the docs (because it actually executes the code). Autoapi extracts the docs in a static way (no need to run the code). So, when using autoapi, you can skip the dependencies installation for the project.

After @tlvu commented at readthedocs/readthedocs.org#5847 (comment):

I guess I am not the only project with OOM problem because of the awsome sphinx autodoc feature. I solved my issue by mocking import (using autodoc_mock_imports in Sphinx's conf.py file, see my PR Ouranosinc/raven#167) of all the 3rd party dependencies so they do not have to be installed for real and by adding my own sources into the sys.path for autodoc to parse for docstrings and auto generate the doc. I basically work-around the requirement to install my project and all its dependencies, resulting in less memory usage and faster build.

You might want to document this somewhere on ReadTheDocs docs so other users using sphinx autodoc will benefit and prevent other similar OOM support call in the future.

Also, @humitos chimed in at readthedocs/readthedocs.org#5847 (comment):

On the other hand, we recommend to use autoapi if what you want is to auto document the modules. It's static and does not need to install anything to get the modules documentation 😉

[amotl]

However, in our case https://readthedocs.org/projects/wetterdienst/builds/ currently does not show any build errors at all, right?

[amotl]

When investigating the latest build at https://readthedocs.org/projects/wetterdienst/builds/11887538/ more thoroughly, these things can be observed:

• It looks like pip install . or an equivalent is not invoked at all.
• sphinx-material 0.0.30 requires sphinx>=2.0, but you'll have sphinx 1.8.5 which is incompatible.. However, this part actually seems to work in general.
• WARNING: autodoc: failed to import function 'discover_climate_observations' from module 'wetterdienst'; the following exception was raised: No module named 'pandas'.

[amotl]

I've tried to reproduce this on my machine by using a fresh Poetry environment for this project after wiping the old one using poetry env remove wetterdienst-EkOFQaO8-py3.8.

Then, I tried to go for the module mocking thing using autodoc_mock_imports, see also FAQ: I get import errors on libraries that depend on C modules.

After some complaints leading to f71ed0e and 34433d9, things started to work. Enjoy https://wetterdienst.readthedocs.io/en/latest/pages/library/api.html.