Building the docs creates temporary files

[gwgundersen]

MCVE

Apologies if this has already been noted, but I can't find an open issue. When I create the docs from scratch, I see a couple temporary files and directories that appear to have been created by accident. If this is a legitimate issue, I'm happy to put together a PR.

Here is an MCVE:

(xarray-docs) gwg:doc gwg$ make clean
rm -rf _build/*
rm -rf generated/*
rm -rf auto_gallery/
(xarray-docs) gwg:doc gwg$ git status
On branch master
Your branch is up to date with 'origin/master'.

nothing to commit, working tree clean
(xarray-docs) gwg:doc gwg$ make html
sphinx-build -b html -d _build/doctrees   . _build/html'

...

Build finished. The HTML pages are in _build/html.
(xarray-docs) gwg:doc gwg$ git status
On branch master
Your branch is up to date with 'origin/master'.

Untracked files:
  (use "git add <file>..." to include in what will be committed)

	example-no-leap.nc
	foo.zarr/
	manipulated-example-data.nc
	path/

nothing added to commit but untracked files present (use "git add" to track)

Expected Output

I would expect no files to be created by make html in the doc directory.

Problem Description

It's not clear this will happen and may result in some people (me!) accidentally committing these files with git commit -A.

[max-sixty]

Yes, at the least we should add those to .gitignore without a more general solution (like a build path). I don't know the doc building well - anyone know if there's a general solution to these?

[DocOtak]

The files and directories that were not cleaned up by the make clean command are all artifacts of the code examples which run in the docs themselves. For example, the manipulated-example-data.nc is created in this section.

At leas one of these files is cleaned up at the end, see the ipython block.

I'd probably look into something like a temporary directory rather than trying to track down all the "example artifacts" created during a run. I'm not sure what sort of configuration the IPython blocks have, but there are also some tempdir utilities in IPython.

[gwgundersen]

After doing some more research, I can't find an ipythonic way of doing this—even have a question with a bounty on StackOverflow. However, here are some options. My vote is for using the Makefile:

Remove via Makefile

It's not ideal, but it ensures the files get removed immediately after they get created or upon clean. Near the top of the file, put something like

...
TMP_FILES = example-no-leap.nc foo.zarr/ manipulated-example-data.nc path/
...

Then the clean and html commands are

.PHONY: clean
clean:
	rm -rf $(BUILDDIR)/*
	rm -rf generated/*
	rm -rf auto_gallery/
	rm -rf $(TMP_FILES)

.PHONY: html
html:
	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
	rm -rf $(TMP_FILES)
	@echo
	@echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

Remove via more documentation

As @DocOtak points out, some of the documentation already cleans up after itself. Personally, I dislike this as it's not clear to the reader why these extra snippets exist.

Remove via .gitignore

Easy to do, but I dislike this as it leaves a bunch of temporary junk in my working directory. I'd rather have it cleaned up somehow.

Remove by writing to and ignoring temp directory

Every ipython directive that creates a file should place that file in tmp/ directory. Then we .gitignore this. This is a bit cleaner than the other .gitignore option. This is my second choice.

Remove using tempdir utilities

I dislike this for the same reason I dislike "Remove via more documentation": it clutters the documentation with utility snippets/code that are opaque to the reader.

---

Happy to create a PR for any of these, but does anyone have any preferences or a better way of doing this?

[DocOtak]

Hi @gwgundersen some clarification on those "extra snippets", github is not aware of the ipython directive so it prints those out like code snippets. In the actual built docs, these don't appear (the :suppress: in that block does this).

I personally feel that the code that makes these temporary files should be responsible for cleaning it up, especially since it already tries, and they aren't build artifacts needed in other steps. I'd probably reach for the tempfile.TemporaryDirectory in the standard library and bracket the dask docs in a create, cd in, cd out, cleanup type flow. There is already a suppressed setup ipython block at the top of the dask docs too.

@max-sixty Any opinions on which option we should go for?

[fmaussion]

If I understand well, this is really just about the temporary files, right? In this case, a cleaning block with a :suppress: directive should be enough, yes.

[max-sixty]

Great, thanks for the clear options @gwgundersen

I don't have a strong view. I think either the makefile or the documentation is viable. Would be interesting to know what other similar projects do (numpy / dask / pandas).

Agree with @fmaussion re adding the :suppress: block in the case of documentation – then no cluttering. More difficult is ensuring that every test has cleaned up (maybe there's an option to run a git diff at the end of the docs and ensure there are no changes?)

[gwgundersen]

Thanks for the explanation, @DocOtak. I didn't initially understand :suppress:, but I now agree that a cleaning block with :suppress: is straightforward. It also follows the current pattern in the docs—I noticed this other places once I knew what to look for.