New Sphinx tutorial for beginners

[astrojuanlu]

tl;dr: We are proposing a new Sphinx tutorial for beginners (no rush! we know that you're busy with the upcoming release) and would like to know people's thoughts.

Hello Sphinx developers! As part of the Chan Zuckerberg Initiative grant that Read the Docs received last year, one of the key tasks we are focusing on is doing "advocacy work around documentation in the scientific community", which is one of my duties as Developer Advocate.

In particular, after talking to lots of users from a variety of backgrounds, we identified a gap in the Sphinx documentation: although it is super detailed and works great as reference material, many people seemed to agree that it would be useful to have a good beginners tutorial. As a result, we have been iterating for a few days and we would like to propose adding a new beginners tutorial to Sphinx.

We commit to writing it incrementally (instead of sending one huge pull request after 6 months), maintaining it, and keeping it up to date, while using this opportunity to contribute other documentation fixes and new features along the way, if needed. We would like this tutorial to be The Best™ resource to direct people willing to learn how to author their technical documentation with Sphinx.

Here you can find the rendered version of the contents we are proposing https://docs--8106.org.readthedocs.build/en/8106/development/design/new-sphinx-guides.html#sphinx-tutorial (and here is the originating pull request in case you want to add comments there directly - the fact that it's on the RTD repository was only a way so we could easily iterate over it "internally"). Summarized, these would be:

1. The Sphinx way
2. About this tutorial
3. Getting started
4. First steps to document our project using Sphinx
5. Customizing Sphinx configuration
6. Writing narrative documentation with Sphinx
7. Describing code in Sphinx
8. Autogenerating documentation from code in Sphinx
9. Deploying a Sphinx project online
10. Appendix: Using Jupyter notebooks inside Sphinx
11. Appendix: Understanding the docutils document tree
12. Appendix: Where to go from here

In particular, we'd like to:

• Be as pedagogical as possible, to best serve beginners.
• Conceive a linear "learning path" that people can follow so they can learn the fundamentals step by step.
• Write the tutorial with a generic technical user in mind, while including some appendixes that can be useful for scientists (like integrating math formulas and Jupyter notebooks).
• Use Markdown all throughout the tutorial, thanks to MyST-Parser.

Pinging @ericholscher from RTD, @choldgraf and @chrisjsewell from MyST, @melissawm who is doing a fantastic job with the NumPy tutorials, @pradyunsg who has been working on Sphinx a lot recently, @laisbsc and @emilopez who gave me initial feedback about some of these ideas, and @lornajane who also expressed interest about this in the Write the Docs Slack workspace.

What are your thoughts @tk0miya @shimizukawa?

[chrisjsewell]

Quick initial thought is yeh that all sounds great thanks, and definitely on-board 👍

With our jupyter-book documentation, @choldgraf has recently been working to improve the initial tutorial: https://jupyterbook.org/start/your-first-book.html, which is pretty much "how to create a sphinx project, without knowing you are using sphinx" and, as such, has a lot of synergy with your proposal.

I'd also not that, in parallel to that, we have been looking to extract the "bespoke" code in jupyter-book, to the point that it is basically a pre-defined set of sphinx extensions and configurations.
Two of the key aspects are:

• https://github.com/executablebooks/sphinx-external-toc, which is now out of beta and allows the option to define your site map in a single YAML file (rather than via toctree directives).
• Add static configuration (Sphinx.toml) #9040 to allow configuration to be defined in a YAML file.

Both of these we have found to be very helpful for beginners and make it possible, for example to generate a basic sphinx site from a set of Markdown files without requiring any modification to them

[melissawm]

Hi @astrojuanlu - this looks really good! I will also ping @rossbar who has been working on the NumPy docs team and might want to share his thoughts as well.

I'd be happy to help too - this is something that would definitely help our contributors. Thank you for working on this!

[jakobandersen]

Improving the documentation is always good, and the overall plan I think looks good.
As my primary interest is in

7. Describing code in Sphinx

the following is from that perspective.

• Use Markdown all throughout the tutorial, thanks to MyST-Parser.

I haven't followed the markdown side, but I find this point a non-trivial decision. If new users read this tutorial and get used to MyST flavoured Markdown syntax, but then search for help online, wouldn't they find almost only reST code? While reST has some problems it seems icky to have two different formats in the Sphinx documentation (unless everything gets rewritten to MyST).

As for MyST it self, I looked at the intro but I didn't find an explicit answer to how one nests directives. Though in the syntax section there is some more info.
For the example

.. cpp:class:: template<typename T> Vector

   .. cpp:class:: iterator

      .. cpp:type:: value_type = T

here the nesting has semantic meaning, and the (forced) indentation in source code makes it readable.
For MyST it looks like this nesting can not be preserved in the source code and the best will be something like

`````cpp:class:: template<typename T> Vector

   ````cpp:class:: iterator

   ```cpp:type:: value_type = T
   >```
   >````
>`````

(with the > in the last three lines removed, otherwise Github would get confused)
Is that correct? I'm not sure I understand how to do arbitrary nesting, and I would strongly prefer something were the nesting is indicated clearly in the source, even when you not fit the complete source of the outer block in a single screen (e.g., indentation for each level).
I probably don't understand how directives work in MyST, but it would be nice to see non-trivial examples.

[pradyunsg]

https://pradyunsg.me/furo/reference/ has a bunch of examples of MyST vs reStructuredText.

I've personally also recommended MyST in https://pradyunsg.me/furo/recommendations/, since Markdown is more popular as a markup language. That said, I think for the Sphinx tutorial, it might make sense to either keep it as only reStructuredText. Or to have reStructuredText as default with a tab to switch to MyST like the reference documentation linked above.

The way to do nesting in MyST is increasing backticks. So, basically your example is almost correct -- there's no indentation necessary.

[jdillard]

Use Markdown all throughout the tutorial, thanks to MyST-Parser.

I strongly prefer a particular extension not be required for beginners to learn how the core of Sphinx works. I understand your grant is for advocacy work around documentation in the scientific community and that extension is more heavily used there, but I would prefer that preference not be forced on all new users.

[pradyunsg]

I guess the specific pages I wanna point to for showcasing the differences between the two markup languages would be:

• Nesting example: https://pradyunsg.me/furo/reference/admonitions/#nesting-admonitions
• MyST has a much easier to mentally model approach to hyperlinks: https://pradyunsg.me/furo/reference/hyperlinks/

[pradyunsg]

Stepping away from the markup language choice, which feels a bit flame-war-like, I do like how the tutorial is structured.

A couple of things that look weird though:

Appendix: Understanding the docutils document tree

Hmm, is this about what the toctree produces? I feel like this should immediately follow the section on narrative documentation, instead of being delegated to an optional appendix. It's fairly important to understand this, to be able to write or work with Sphinx documentation IMO.

---

Intersphinx is noticeably missing from this, and AFAIK, that's basically a big reason that folks still continue to choose Sphinx for documentation despite all the alternative documentation genrrators available. In other words, it's a unique selling point for Sphinx, and IMO we should at least have an appendix about it to introduce it to someone who hasn't heard of it.

[jakobandersen]

@chrisjsewell, Indeed, without indentation it is clear, but it is mentioned that one can use a single level of indentation. @pradyunsg admonition example shows the same. While for that case multi-level nesting seems dubious, it is inherently a part code documentation, at least for C++ and C.
I'm not at all convinced this is a good syntax.

@pradyunsg, it's a very nice site to see the differences between the two input formats! (though I noticed that for tables the MyST example is missing tables with row and column spanning).

[pradyunsg]

@panhaoyu,

Oops, wrong mention. ;)

(though I noticed that for tables the MyST example is missing tables with row and column spanning)

It probably has something for that. I had gotten lazy by the time I wrote that, so the later pages of that whole section aren't fully polished. 😅

[astrojuanlu]

Whoa! Didn't expect so many comments :) Quickly answering a couple of easy ones:

Appendix: Understanding the docutils document tree

I referred to this: https://docutils.sourceforge.io/docs/ref/doctree.html so, more like the internals of docutils rather than the toctree itself. Which perhaps we should treat more in depth?

Intersphinx is noticeably missing from this, and AFAIK, ...

It's a super good point, thanks @pradyunsg!

[astrojuanlu]

About the markup choice: regardless of which one we use, Sphinx is much more than that. I would happily discuss the merits of MyST vs reST, perhaps at a later stage, since I don't think it affects the contents of the tutorial itself.

[ericholscher]

Also of note, we aren't trying to write tutorials for every feature of Sphinx. We want to keep this sized to something that people will actually do, instead of adding every possible feature. We should include more tutorial-style content in the Sphinx docs, but we should be pretty focused and concise on what actually makes it into the official tutorial.

I love this section from Jacob's blog post long ago that covers what to include in the tutorial: https://jacobian.org/2009/nov/10/what-to-write/#tutorials -- something to keep in mind here.

I think a way to think about it might be "What are the three most unique and valuable Sphinx features that we must be sure to include, to make sure users get the most 'this is awesome' from the tutorial"

I think my list is probably:

• Cross-references & Intersphinx
• Includes (including literal-include)
• Mixing of auto-generated & narrative content
• csv-table (mostly kidding, but its huge)

[chrisjsewell]

I love this section from Jacob's blog post long ago that covers what to include in the tutorial

FYI I find this a very helpful guide for documentation writing: https://documentation.divio.com/

[astrojuanlu]

It has a new home and a classier name :) https://diataxis.fr/