🔌 Plugin: Sphinx documentation support

[igor-petrik-invitae]

Summary

It would be great to have a plugin that can compile an publish Sphinx documentation to a Backstage server. A lot of python code is already documented with Sphinx, which also is well integrated with code.

Project website

https://www.sphinx-doc.org/en/master/

[emmaindal]

Hi @igor-petrik-invitae thanks a lot for opening this issue with a suggested plugin for Sphinx documentation. To understand the use case a bit better, is your experience that Sphinx is mainly used for API documentation? Or other types of documentation too? I'm curious as we do have the API plugin, as well as the TechDocs plugin and if Sphinx documentation can be integrated in any of those or if you think its more valuable as a separate plugin?

[warsaw]

is your experience that Sphinx is mainly used for API documentation?

I've been exploring Backstage and TechDocs and landed here. Although Sphinx and RST are not Python specific, it's probably much more popular in the Python community than elsewhere. There are tons of documentation written in RST and AFAIK, Sphinx is the only static site builder that consumes RST. It can consume MD too with plugins. Probably one of the most popular open source documentation sites using Sphinx+RST is ReadTheDocs.

Sphinx is used for both prose and API documentation and it has many extensions (both built-in and third party) to automatically extract API docs from Python code, especially when type hinting is added and docstrings are written in a standard format.

That said, in all honesty MD is way more popular of a source format and in my own doc solution we're debating whether to continue to support RST. The biggest unanswered question atm is what to do about Python API docs. We'd need both that functionality and the high quality docs that current solutions provide.

(I'm personally a fan of Sphinx and RST, but I've been a user of them for a very long time, and it's also a maintenance burden to support multiple formats and builders.)

[emmaindal]

@warsaw Thanks for providing that perspective, very valuable!

what to do about Python API docs

Would something like a build time extension like https://github.com/mkdocstrings/mkdocstrings or alternatively a render time "addon" on top of TechDocs that let you embed code from any source code file, help you solve your needs do you think?

Would you still prefer having the python API docs as a part of the rest of your technical documentation (for example by using TechDocs) or as a separate thing but still close to the entity (for example by using the API Docs plugin)?

[garyniemen]

It's tricky with APIs I always think. Overview information about the API fits fine in a tool like TechDocs. But the actual reference information for the API - I feel - belongs close to the API commands. But you are asking about a separate plugin. Perhaps I am missing something, but I can see the sense of that in this case. To make everything homogenous, you would want to surface the documentation under Docs and, if relevant, in the software catalog. And then link the output up to search. So you'd be using the same framework as TechDocs - but making docs in another way. Am I correct here @emmaindal or confused?

[jjedele]

I also landed here after a while of search. Following the "documentation as code" mantra, I also feel like it's preferable to have as much as the documentation as possible inside actual in-code docstrings in the source rather than in some orthogonal markdown file structure. This transcends public APIs as e.g. Sphinx documentation is also nice to document the high-level structure of code and responsibilities of classes, etc.

I was not even looking for backstage integrating Sphinx, it would probably be enough for me to just host and link the static HTML docs produced by Sphinx. Is something like this possible? I browsed the documentation and did some experiments, but I don't seem to get it to work.

The tipp with mkdocstrings of @emmaindal seems pretty helpful. I will try that when I have some time.

[igor-petrik-invitae]

My original thought was an implementation was something along the lines of what @jjedele proposed.

To give perspective that may answer some of your other questions, Sphinx builds static pages from RST files that exist in a docs folder. These contain general documentation that I would say is analogous to the purpose of TechDocs. The convenient thing is, though, that integrates tightly with inline API documentation within the code itself, and can "import" these chunks of documentation throughout other parts of the code. This makes it convenient to not repeat yourself, while still having API documentation in both the docs and inline in the code.

Since Sphinx can already do all this work and generate static HTML files, I think it would be convenient for Backstage to provide a way to host these sites built by Sphinx in a way that is integrated into the rest of the platform.

One solution I had considered was using a custom Sphinx renderer to transpile Sphinx docs into markdown, and feed those into TechDocs, but this is less than ideal, since markdown lacks many of the features of RST, so a lot of the formatting would be lost.

[cblegare]

Also, RST has build-time extension builtin with specific syntax for arbitrary logic in documentation (directives and roles), while Markdown is designed to be extended via HTML. The recommended implementation for writing Markdown for Sphinx (MyST parser) added directives using the code-block Markdown syntax.

There are a few very large documentation built with Sphinx, namely parts of AWS and GCP docs, Linux kernel, MongoDB, Openshift, and more.

Also, Sphinx has a extremely powerful crossreference engine spanning over foreign documentation roots (see intersphinx), also making the crossreference way easier to maintain than anything built with Markdown-powered documentation engines. How Sphinx handles this is basically what is suggested in backstage/backstage#14844. Sphinx also nicely integrates with Doxygen (C/C++,Java,C#,PHP,...).

I don't know much about techdocs' architecture, but along with the issue about enabling dbt, allowing different documentation engines could be interesting.

[warsaw]

There are a few very large documentation built with Sphinx, namely parts of AWS and GCP docs, Linux kernel, MongoDB, Openshift, and more.

Don't forget, CPython's full documentation suite as well!

[aseppala]

This would be a very welcomed addition to Backstage. Having Sphinx docsites generated, hosted and indexed for search by Backstage would be very useful.

[JimNero009]

Adding my voice to this too. I actually think the problem is more acute -- since Sphinx is creating a bunch of static files, I see the problem better phrased as 'TechDocs plugin should be publisher agnostic'. That is, I feel there should be a very basic client that can be pointed to, say, an S3 bucket and just told 'there's some static content there -- serve it!'. The implementation as it stands right now seems to be heavily reliant on the presence of the mkdocs.yml file in the remote storage location, but I don't understand why that should be a necessary component.

[drodil]

One way to possibly make this work is to build the sphinx docs to markdown and serve those as techdocs. Not sure if this works for all sphinx plugins etc. but might be worth a try. See https://github.com/clayrisser/sphinx-markdown-builder

[cblegare]

@JimNero009 That is also my opinion, except one thing: the TechDoc Search plugin. By decoupling TechDoc from MkDocs we could certainly write a Sphinx-compatible collator with the existing documentation.

@drodil That would make using Sphinx and RestructuredText completly pointless. Mardown was designed for single page documents, extensible only using HTML (and CSS), and documentation system such as MkDocs need to find clever holes in the Markdown language specification to add features to it. On the other hand, RestructuredText was designed for extendability using roles and directives, which is clearly the feature we need for non-trivial documentation endeavors.

The MyST parser did a great job at fitting roles and directives in the Markdown language, but this is not usable by a general-purpose Markdown parser.

The only way to go for Backstage, is to decouple it from MkDocs. Sphinx (and other document processor) themes could be nice to have for a consistent user experience, as well as dedicated collator implementations.

[chalbersma]

#267

Just to expand on this. Sphinx also offers plugins for a ton of languages and a ton of things. Trying to cram all that functionality into mkdocs is almost certainly a loosing proposition.

[cblegare]

#9172 (comment)

Just to expand on this. Sphinx also offers plugins for a ton of languages and a ton of things. Trying to cram all that functionality into mkdocs is almost certainly a loosing proposition.

I am the author of a few Sphinx extensions, namely defining mapping for HCL (Terraform) and Gherkin (Cucumber) and I fully agree. Sphinx has also extensions for Doxygen comments and more

[freben]

Sorry, started reclassifying this but changed my mind. Leaving it as a plugin for now

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.