Putting version-switcher in Primary Sidebar results in Extension Error

[adam-grant-hendry]

Following the Version Switcher Dropdowns instructions, I'm getting the following error when trying to place the version switcher in the primary sidebar:

Extension error (pydata_sphinx_theme):
Handler <function update_and_remove_templates at 0x000001A18C5BF700> for event 'html-page-context' threw an exception (exception: toggle-primary-sidebar.html)

The template toggle-primary-sidebar.html is defined in sphinx-book-theme, which inherits from pydata-sphinx-theme. It appears the method _remove_empty_templates is intended for pydata-sphinx-theme templates, but will check other templates that derive from it, which is causing this error (toggle-primary-sidebar.html is not a pydata-sphinx-theme template).

I'm not sure how to proceed. I've started an initial discussion (see #1473), but I'm still lost.

See also:

• sphinx-book-theme: Autosummary and templates not working with versions > 1
• sphinx-book-theme: jinja2.exceptions.TemplateNotFound: toggle-primary-sidebar.html

The full traceback is:

Traceback (most recent call last):
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\events.py", line 96, in emit
    results.append(listener.handler(self.app, *args))
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pydata_sphinx_theme\__init__.py", line 224, in update_and_remove_templates
    context[section] = list(filter(_remove_empty_templates, context[section]))
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\pydata_sphinx_theme\__init__.py", line 219, in _remove_empty_templates    
    rendered = app.builder.templates.render(tname, context)
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\jinja2glue.py", line 196, in render
    return self.environment.get_template(template).render(context)
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\jinja2\environment.py", line 1010, in get_template
    return self._load_template(name, globals)
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\jinja2\environment.py", line 969, in _load_template
    template = self.loader.load(self, name, self.make_globals(globals))
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\jinja2\loaders.py", line 126, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\jinja2glue.py", line 217, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: toggle-primary-sidebar.html

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\cmd\build.py", line 285, in build_main
    app.build(args.force_all, args.filenames)
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\application.py", line 353, in build
    self.builder.build_update()
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\builders\__init__.py", line 311, in build_update
    self.build(to_build,
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\builders\__init__.py", line 378, in build
    self.write(docnames, list(updated_docnames), method)
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\builders\__init__.py", line 586, in write
    self._write_serial(sorted(docnames))
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\builders\__init__.py", line 596, in _write_serial
    self.write_doc(docname, doctree)
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\builders\html\__init__.py", line 672, in write_doc
    self.handle_page(docname, ctx, event_arg=doctree)
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\builders\html\__init__.py", line 1076, in handle_page
    newtmpl = self.app.emit_firstresult('html-page-context', pagename,
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\application.py", line 481, in emit_firstresult
    return self.events.emit_firstresult(event, *args,
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\events.py", line 117, in emit_firstresult
    for result in self.emit(name, *args, allowed_exceptions=allowed_exceptions):
  File "C:\Users\hendra11\Code\external\poetry_plugin_constrain\.venv\lib\site-packages\sphinx\events.py", line 107, in emit
    raise ExtensionError(__("Handler %r for event %r threw an exception") %
sphinx.errors.ExtensionError: Handler <function update_and_remove_templates at 0x000001B8B89F7AF0> for event 'html-page-context' threw an exception (exception: toggle-primary-sidebar.html)

Extension error (pydata_sphinx_theme):
Handler <function update_and_remove_templates at 0x000001B8B89F7AF0> for event 'html-page-context' threw an exception (exception: toggle-primary-sidebar.html)

[adam-grant-hendry]

The pydata-sphinx-theme manually adds their components folder to the config.templates_path in their __init__.py/setup() method:

app.config.templates_path.append(str(theme_path / "components"))

which alters the default search path order for templates, calculated in sphinx/jinja2glue.py/BuiltinTemplateLoader.init().

Per the sphinx docs, the default search order is:

• user's template_path
• selected theme
• base theme (i.e. what is specified in theme.conf by inherit =)

However, debugging sphinx-build with the sphinx-book-theme, we see pathchain is initially set properly by sphinx/theming.py/get_theme_dirs():

but is messed up when it gets to the point of prepending explicit template paths:

NOTE: I debug sphinx-build in VSCode on Windows 10 using the following launch.json configuration:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Sphinx Debugger",
            "type": "python",
            "request": "launch",
            "program": "${workspaceFolder}/.venv/Scripts/sphinx-build.exe",
            "args": ["-b", "html", "src", "build"],
            "console": "integratedTerminal",
            "cwd": "${workspaceFolder}/docs",
            "justMyCode": false,
        }
    ]
}

and then running from the Run and Debug view:

and have the following package versions:

python: 3.8.10
Sphinx: 6.2.1
jinja2: 3.1.2
sphinx-book-theme: 1.0.1
pydata-sphinx-theme: 0.14.1

[adam-grant-hendry]

but is messed up when it gets to the point of prepending explicit template paths:

Actually, it appears sphinx-book-theme also appends to the templates_path, but what is odd is that the "components" portion of the path is missing.

[adam-grant-hendry]

Just wanted to capture what's going on during a build, specifically this is run:

sphinx-build -b html src build

In this case, conf.py has sphinx_book_theme in extensions, but not pydata_sphinx_theme. i.e.:

#conf.py

extensions - [
    'sphinx_book_theme',
    # not 'pydata_sphinx_theme',
    ...
]

Starting from the sphinx-build entry point (i.e. sphinx.cmd.build:main; see pyproject.toml), the following happens in sphinx/application.py/Sphinx.__init__():

1. The conf.py file is read
2. The sphinx_book_theme setup() method is run, after which templates_path is:

['project\\.venv\\Lib\\site-packages\\sphinx_book_theme\\theme\\sphinx_book_theme\\components']

3. config.init_values() is run, which overrides templates_path to:

['_templates']

4. _init_builder() is run, calling sphinx/builds/html/__init__.py/StandaloneHTMLBuilder.init(), which calls StandaloneHTMLBuilder.init_templates().

5. An HTMLThemeFactory creates 'sphinx_book_theme', which reloads pydata_sphinx_theme by calling its setup(), after which templates_path is:

['_templates', 'C:\\Users\\hendra11\\Code\\external\\poetry_plugin_constrain\\.venv\\Lib\\site-packages\\pydata_sphinx_theme\\theme\\pydata_sphinx_theme\\components']

The sphinx_book_theme setup() is never re-run, so sphinx is unaware of the sphinx_book_theme\\components path, so sphinx cannot find toggle-primary-sidebar.html.

[adam-grant-hendry]

It looks like this issue was fixed in sphinx-book-theme PR #566 precisely how I would have though to fix it: create a config-inited event:

app.connect("config-inited", update_general_config)

but it looks like it was changed to a 'builder-inited' event in PR #691:

app.connect("builder-inited", update_general_config)

This looks like it was a mistake.

[adam-grant-hendry]

I can confirm reverting to:

app.connect("config-inited", update_general_config)

and

def update_general_config(app, config):  # add config argument back
    ...

fixes the problem.

[drammock]

thanks for documenting your detective work @adam-grant-hendry! Is there a TL;DR for what we should do differently in this repo? Or in the end is the problem all on the sphinx-book-theme side?

[adam-grant-hendry]

thanks for documenting your detective work @adam-grant-hendry! Is there a TL;DR for what we should do differently in this repo? Or in the end is the problem all on the sphinx-book-theme side?

Thanks! (And welcome back!) I like to be as explicit as possible for myself and the community in case we ever need to reread this.

TL;DR

sphinx-book-theme accidentally "undid" work from a previous PR in a recent one, but you should add a note to your docs for downstream theme developers.

NB

In the explanation below, selected theme means the subclassee/downstream theme (e.g. sphinx-book-theme) and base theme the inherited theme (e.g. pydata-sphinx-theme), in alignment with sphinx's terminology

1. sphinx (at least 6.2.1) performs config.init_values() AFTER it runs theme setup()'s
2. This is important is because init_values() overrides config with defaults. If a selected theme needs to modify config (e.g. to add components to the template search path), it should do so upon a config-inited event (not a builder-inited event).

In my case, the only reason pydata-sphinx-theme still modified config.templates_path was because pydata-sphinx-theme wasn't part of my extensions in my conf.py, so sphinx considers it an external theme and loads it when the builder is inited:

• In order, sphinx runs the builtin and selected theme setup()s, then inits config, and lastly inits the builder
• During builder initialization, when templates are inited, sphinx pulls the inherited theme from theme.conf, and if it's not in the sphinx registry, it rightly thinks the theme hasn't been loaded and runs its setup()
• Since sphinx sees the selected theme as already loaded, its setup() isn't run again

So that the selected theme modifies config and has its values placed before those of the base theme's to ensure the proper search path order order of config.templates_path -> selected theme -> base theme, selected themes must modify their config upon config-inited events.