sometimes searchindex.js.tmp is not generated

[informatimago]

Describe the bug

When generating our doc, with make html, I get this error:

dumping search index in English (code: en)... indexer_dumps_unicode
failed

Exception occurred:
  File "/home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/sphinx/util/osutil.py", line 112, in movefile
    os.rename(source, dest)
FileNotFoundError: [Errno 2] No such file or directory: '/home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/_build/html/searchindex.js.tmp' -> '/home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/_build/html/searchindex.js'
The full traceback has been saved in /tmp/sphinx-err-62za4hmz.log, if you want to report the issue to the developers.

Note: I added a print in the if to know whether it was unicode or non-unicode that failed:

    def dump_search_index(self) -> None:
        with progress_message(__('dumping search index in %s') % self.indexer.label()):
            self.indexer.prune(self.env.all_docs)
            searchindexfn = path.join(self.outdir, self.searchindex_filename)
            # first write to a temporary file, so that if dumping fails,
            # the existing index won't be overwritten
            if self.indexer_dumps_unicode:
                print("indexer_dumps_unicode")
                with open(searchindexfn + '.tmp', 'w', encoding='utf-8') as ft:
                    self.indexer.dump(ft, self.indexer_format)
            else:
                print("not indexer_dumps_unicode")
                with open(searchindexfn + '.tmp', 'wb') as fb:
                    self.indexer.dump(fb, self.indexer_format)
            movefile(searchindexfn + '.tmp', searchindexfn)

But I don't know how to catch an error from self.indexer.dump or if it's silent.

cf. logs:

sphinx-err-62za4hmz.log
sphinx.log

How to Reproduce

I prepare the environment with:

    sudo apt-get install  -y  python3-venv python3-pip texlive-font-utils librsvg2-bin plantuml python3-sphinxcontrib.apidoc linuxdoc-tools
    python3 -m venv sphinx-venv
    source sphinx-venv/bin/activate
    pip install -r sphinx/requirements.txt

requirements.txt

In addition, the following sphinx debian packages are installed:

(sphinx-venv) 
~/works/qorvo/src/uwb-stack/uwb-stack/doc
09:34[pjb@qorvo despina.lan.informatimago.com:0 doc 126G]$ apt list --installed|grep -e 'sphinx\|rst'

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libjs-sphinxdoc/stable,now 3.4.3-2 all [installed,automatic]
libpocketsphinx3/stable,now 0.8+5prealpha+1-13 amd64 [installed,automatic]
libsphinxbase3/stable,now 0.8+5prealpha+1-12 amd64 [installed,automatic]
pocketsphinx-en-us/stable,now 0.8+5prealpha+1-13 all [installed,automatic]
python3-sphinx-rst-builder/stable,now 0.0.3-2 all [installed]
python3-sphinx/stable,now 3.4.3-2 all [installed,automatic]
python3-sphinxcontrib.apidoc/stable,now 0.3.0-2 all [installed]
rst2pdf/stable,now 0.98-1 all [installed]
sphinx-common/stable,now 3.4.3-2 all [installed,automatic]

Expected behavior

I'd expect the searchindex.js.tmp file to be generated, or its absence to be ignored.

Your project

Proprietary.

Screenshots

No response

OS

linux

Python version

3.9.2

Sphinx version

2.4.4

Sphinx extensions

# Loaded extensions:
#   sphinx.ext.mathjax            (2.4.4)  from     /home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/sphinx/ext/mathjax.py
#   sphinxcontrib.applehelp       (1.0.2)  from     /home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/sphinxcontrib/applehelp/__init__.py
#   sphinxcontrib.devhelp         (1.0.2)  from     /home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/sphinxcontrib/devhelp/__init__.py
#   sphinxcontrib.htmlhelp        (2.0.0)  from     /home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/sphinxcontrib/htmlhelp/__init__.py
#   sphinxcontrib.serializinghtml (1.1.5)  from     /home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/sphinxcontrib/serializinghtml/__init__.py
#   sphinxcontrib.qthelp          (1.0.3)  from     /home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/sphinxcontrib/qthelp/__init__.py
#   alabaster                     (0.7.12) from     /home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/alabaster/__init__.py
#   linuxdoc.rstKernelDoc         (3.0)    from     /home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/linuxdoc/rstKernelDoc.py
#   linuxdoc.cdomain              (3.0)    from     /home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/linuxdoc/cdomain.py
#   sphinxcontrib.plantuml        (unknown version) from /home/pjb/works/qorvo/src/uwb-stack/uwb-stack/doc/sphinx-venv/lib/python3.9/site-packages/sphinxcontrib/plantuml.py

Extra tools

No response

Additional context

NA

[barbaricyawps]

I also experienced this bug. The workaround solution seems to be to copy the missing file to the tmp directory. Here's the Linux command:

cp docs/_build/html/searchindex.js docs/_build/html/searchindex.js.tmp

[tk0miya]

Hmm... it's strange. Python interpreter surely closes the .tmp file after the with block. So I'm not sure why the file was vanished...

[dave-fitzgerald-bc]

I'm experiencing this bug too but only when the docs are being built in a pre-commit hook.

I can confirm that @barbaricyawps's workaround works for my case too, but it is not a long-term solution.

[Shoeboxam]

I think this may be an issue when multiple sphinx commands are running at the same time-- specifically, I observe this when running sphinx-build doctest and make html, even though they are pointed at different directories. Error goes away if I make html first once (or even sleep for 1 second).