Sphinx 5.1.0 exception: pop from an empty deque

[eivindjahren]

Describe the bug

As of 5.1.0 our doc build at github.com/equinor/ert stopped working and gives us the following unhandled exception:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/events.py", line 94, in emit
    results.append(listener.handler(self.app, *args))
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/napoleon/__init__.py", line 385, in _process_docstring
    docstring = NumpyDocstring(result_lines, app.config, app, what, name,
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/napoleon/docstring.py", line 1147, in __init__
    super().__init__(docstring, config, app, what, name, obj, options)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/napoleon/docstring.py", line 214, in __init__
    self._parse()
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/napoleon/docstring.py", line 594, in _parse
    res = self._parse_attribute_docstring()
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/napoleon/docstring.py", line 626, in _parse_attribute_docstring
    _type, _desc = self._consume_inline_attribute()
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/napoleon/docstring.py", line 303, in _consume_inline_attribute
    line = self._lines.popleft()
IndexError: pop from an empty deque
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/cmd/build.py", line [27](https://github.com/equinor/ert/runs/7488615143?check_suite_focus=true#step:8:28)7, in build_main
    app.build(args.force_all, filenames)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/application.py", line 349, in build
    self.builder.build_update()
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 300, in build_update
    self.build(to_build,
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 316, in build
    updated_docnames = set(self.read())
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 423, in read
    self._read_serial(docnames)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 444, in _read_serial
    self.read_doc(docname)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/builders/__init__.py", line 497, in read_doc
    publisher.publish()
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/core.py", line 217, in publish
    self.document = self.reader.read(self.source, self.parser,
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/io.py", line 104, in read
    self.parse()
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/readers/__init__.py", line 78, in parse
    self.parser.parse(self.input, document)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/parsers.py", line 78, in parse
    self.statemachine.run(inputlines, document, inliner=self.inliner)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 170, in run
    results = StateMachineWS.run(self, input_lines, input_offset,
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/statemachine.py", line 239, in run
    context, next_state, result = self.check_line(
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/statemachine.py", line 451, in check_line
    return method(match, context, next_state)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2769, in underline
    self.section(title, source, style, lineno - 1, messages)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 327, in section
    self.new_subsection(title, lineno, messages)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 393, in new_subsection
    newabsoffset = self.nested_parse(
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line [28](https://github.com/equinor/ert/runs/7488615143?check_suite_focus=true#step:8:29)1, in nested_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/statemachine.py", line 239, in run
    context, next_state, result = self.check_line(
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/statemachine.py", line 451, in check_line
    return method(match, context, next_state)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2344, in explicit_markup
    self.explicit_list(blank_finish)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2369, in explicit_list
    newline_offset, blank_finish = self.nested_list_parse(
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 318, in nested_list_parse
    state_machine.run(block, input_offset, memo=self.memo,
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 196, in run
    results = StateMachineWS.run(self, input_lines, input_offset)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/statemachine.py", line 239, in run
    context, next_state, result = self.check_line(
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/statemachine.py", line 451, in check_line
    return method(match, context, next_state)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2647, in explicit_markup
    nodelist, blank_finish = self.explicit_construct(match)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2354, in explicit_construct
    return method(self, expmatch)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2096, in directive
    return self.run_directive(
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/docutils/parsers/rst/states.py", line 2146, in run_directive
    result = directive_instance.run()
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/autodoc/directive.py", line 148, in run
    documenter.generate(more_content=self.content)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 1788, in generate
    return super().generate(more_content=more_content,
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 954, in generate
    self.document_members(all_members)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 1779, in document_members
    super().document_members(all_members)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 8[30](https://github.com/equinor/ert/runs/7488615143?check_suite_focus=true#step:8:31), in document_members
    documenter.generate(
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 951, in generate
    self.add_content(more_content)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 602, in add_content
    for i, line in enumerate(self.process_doc(docstrings)):
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/ext/autodoc/__init__.py", line 547, in process_doc
    self.env.app.emit('autodoc-process-docstring',
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/application.py", line [46](https://github.com/equinor/ert/runs/7488615143?check_suite_focus=true#step:8:47)0, in emit
    return self.events.emit(event, *args, allowed_exceptions=allowed_exceptions)
  File "/opt/hostedtoolcache/Python/3.8.13/x64/lib/python3.8/site-packages/sphinx/events.py", line 105, in emit
    raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function _process_docstring at 0x7fe6a9[57](https://github.com/equinor/ert/runs/7488615143?check_suite_focus=true#step:8:58)2280> for event 'autodoc-process-docstring' threw an exception (exception: pop from an empty deque)
Extension error (sphinx.ext.napoleon):
Handler <function _process_docstring at 0x7fe6a95[72](https://github.com/equinor/ert/runs/7488615143?check_suite_focus=true#step:8:73)2[80](https://github.com/equinor/ert/runs/7488615143?check_suite_focus=true#step:8:81)> for event 'autodoc-process-docstring' threw an exception (exception: pop from an empty deque)
Error: Process completed with exit code 2.

How to Reproduce

Following works in my environment, if there are issues, see https://github.com/equinor/ert/blob/ec015e87ef511f45826ae53c0aae1d6e1a99b9f1/.github/workflows/build.yml#L195 for how its
done in CI.

$ git clone https://github.com/equinor/ert
$ cd ert
$ pip install .
$ pip install -r dev_requirements.txt
$ sphinx-build -n -v -E -W ./docs/rst/manual ./tmp/ert_docs

Expected behavior

Documentation is built

Your project

github.com/equinor/ert

Screenshots

No response

OS

Ubuntu-latest

Python version

3.8

Sphinx version

5.1.0

Sphinx extensions

"sphinx.ext.intersphinx", "sphinx.ext.mathjax", "sphinx.ext.ifconfig", "sphinx.ext.githubpages", "sphinx.ext.autodoc", "sphinx.ext.napoleon", "sphinxarg.ext", "sphinx.ext.todo", "sphinxcontrib.datatemplates",

Extra tools

No response

Additional context

Our project uses some custom plugins, and we are trying to figure out if they are acting up.

[AA-Turner]

This is related to the napolean.iterators change (#9856) it seems, cc @anntzer

A

[AA-Turner]

@eivindjahren I can't install ERT on Windows to reproduce (Python 3.9; pip 22.2):

(venv) I:\Development\sphinx-bugfixing\ert>pip install .
Processing i:\development\sphinx-bugfixing\ert
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [18 lines of output]
      Collecting setuptools
        Using cached setuptools-63.2.0-py3-none-any.whl (1.2 MB)
      Collecting setuptools_scm
        Downloading setuptools_scm-7.0.5-py3-none-any.whl (42 kB)
           -------------------------------------- 42.9/42.9 kB 695.1 kB/s eta 0:00:00
      Collecting wheel
        Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
      Collecting scikit-build
        Downloading scikit_build-0.15.0-py2.py3-none-any.whl (77 kB)
           ---------------------------------------- 77.5/77.5 kB 1.1 MB/s eta 0:00:00
      Collecting cmake
        Downloading cmake-3.22.5-py2.py3-none-win_amd64.whl (38.1 MB)
           ---------------------------------------- 38.1/38.1 MB 1.2 MB/s eta 0:00:00
      Collecting ninja
        Downloading ninja-1.10.2.3-py2.py3-none-win_amd64.whl (292 kB)
           ------------------------------------ 292.6/292.6 kB 950.7 kB/s eta 0:00:00
      ERROR: Could not find a version that satisfies the requirement ecl (from versions: none)
      ERROR: No matching distribution found for ecl
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Any suggestions?

A

[eivindjahren]

@AA-Turner unfortunately we don't support windows. I am guessing the problem is that it's not finding a windows wheel for ecl.

[yarikoptic]

seems due to @propertyes having no docstring, e.g. in our case:

  File "/home/yoh/proj/datalad/datalad-maint/venvs/dev3/lib/python3.10/site-packages/sphinx/ext/napoleon/docstring.py", line 626, in _parse_attribute_docstring
    _type, _desc = self._consume_inline_attribute()
  File "/home/yoh/proj/datalad/datalad-maint/venvs/dev3/lib/python3.10/site-packages/sphinx/ext/napoleon/docstring.py", line 303, in _consume_inline_attribute
    line = self._lines.popleft()
IndexError: pop from an empty deque
> /home/yoh/proj/datalad/datalad-maint/venvs/dev3/lib/python3.10/site-packages/sphinx/ext/napoleon/docstring.py(303)_consume_inline_attribute()
-> line = self._lines.popleft()
(Pdb) up
> /home/yoh/proj/datalad/datalad-maint/venvs/dev3/lib/python3.10/site-packages/sphinx/ext/napoleon/docstring.py(626)_parse_attribute_docstring()
-> _type, _desc = self._consume_inline_attribute()
(Pdb) 
> /home/yoh/proj/datalad/datalad-maint/venvs/dev3/lib/python3.10/site-packages/sphinx/ext/napoleon/docstring.py(594)_parse()
-> res = self._parse_attribute_docstring()
(Pdb) 
> /home/yoh/proj/datalad/datalad-maint/venvs/dev3/lib/python3.10/site-packages/sphinx/ext/napoleon/docstring.py(214)__init__()
-> self._parse()
(Pdb) 
> /home/yoh/proj/datalad/datalad-maint/venvs/dev3/lib/python3.10/site-packages/sphinx/ext/napoleon/docstring.py(1147)__init__()
-> super().__init__(docstring, config, app, what, name, obj, options)
(Pdb) p name
'datalad.support.archives.ExtractedArchive.is_extracted'
(Pdb) 
[2]  + 1098197 suspended  DATALAD_SPHINX_RUN=1 sphinx-build -b html -d build/doctrees -W source -v -P 
(dev3) (datalad-py3.7) 1 39129 ->148 [2].....................................:Sun 24 Jul 2022 01:56:21 PM EDT:.
(git)lena:~datalad/datalad-maint[maint]docs
$> git grep -4 'def is_extracted'
(dev3) (datalad-py3.7) 1 39130 ->1 [2].....................................:Sun 24 Jul 2022 01:56:32 PM EDT:.
(git)lena:~datalad/datalad-maint[maint]docs
$> git -C .. grep -4 'def is_extracted'
datalad/support/archives.py-    def stamp_path(self):
datalad/support/archives.py-        return self._path + self.STAMP_SUFFIX
datalad/support/archives.py-
datalad/support/archives.py-    @property
datalad/support/archives.py:    def is_extracted(self):
datalad/support/archives.py-        return exists(self.path) and exists(self.stamp_path) \
datalad/support/archives.py-            and os.stat(self.stamp_path).st_mtime >= os.stat(self.path).st_mtime
datalad/support/archives.py-
datalad/support/archives.py-    def assure_extracted(self):

so adding dummy docstring to those properties worksaround the issue

edit: a quick dirty monkey patching workaround https://github.com/datalad/datalad/pull/6883/files#diff-008dcb3426febd767787b1521f1fe33086313b927ea37eaab86df5fa88a51698R356 with hope that 5.1.1 will soon fix this issue properly.

[adamjstewart]

Same issue with Spack: https://readthedocs.org/projects/spack/builds/17523257/

[PeterBaker0]

Similar error on essentially blank project following guide from https://py-pkgs.org/