Upload pre-built docs

[marscher]

I wonder if that would be possible? Would be extremely useful, since I'm building and deploying my packages with travis, and could perform the doc building also during this runs.

You may ask why this is necessary, since its not obvious that I can not use your build environment for documentation. I've to convert some ipython notebooks with cabal to rst and include them in my sphinx build. This aint possible on RTD and probably will never be.

Thanks for your effort!

Front conversations

[agjohnson]

We agree this would be an awesome feature, we've already talked about how this might work and it's vaguely on our roadmap already.

[marscher]

what is the status of APIv2.1?

[felixc]

👍 to this. I could really use a "dumb file upload" API, where I can just provide a zip/tar/whatever of HTML+CSS files, along with some basic metadata like what version it's for.

[thedillonb]

I'm also very interested in this. Since I try to generate as much documentation via the code comments it would be very easy to, during CI, generate the pages, and upload to RTD - without the need for checking-in these generated files!

[embray]

IIRC this used to work, or at least, there was a file upload option but it was removed?

What needs to be done to overcome the original problems with this feature? It would be nice to have again, and could even allow us to offload some build overhead from RTD.

[marscher]

Maybe it has been removed, so people don't misuse rtfd as a free hosting service?

[agjohnson]

I believe it was once a feature of RTD, but it was removed. We'd like to have this supported with an overhaul of the API -- a task we haven't added to our roadmap yet.

The more pressing issue we need to take a stance on is what are we accepting via html upload. RTD doesn't aim to be just a file host -- you could output and host documentation almost anywhere on the internet. Our goal is making the documentation authoring and viewing experience great, with Sphinx being our primary focus. The documentation experience would go south if we just took any old html and hosted it. We don't have the resources or community support to improve the experience of al the various documentation output folks use.

We're still shaping our opinion here, but we look to support some form of integration with externally built sphinx documentation, via some local command wrappers that you could run on external CI services/etc. We don't have an ETA on when we might drop that on our roadmap yet, but it is on our radar.

[embray]

It seems the biggest issue is then just ensuring a certain level of quality, and now allowing spam and or non-docs to be hosted. Perhaps the feature could be enabled only for validated users. Or maybe even paying (?) users? I can see how it would be difficult to handle that resource-wise though.

[luketunmer]

This feature would be great, even it was turned off on the public RTD server, but available for corporate-hosted ones in which the concerns over hosting generic HTML would not be an issue. The builds of our repos are just to contorted to expect a RTD server to check out and build, whereas our Jenkins CI server does all of that already.

[mistercrunch]

I can't get RTD to compile my project because of external dependencies that require OS modules. I'd love to be able to upload my compiled docs.

Since this issue isn't moving I have to rely on pythonhosted and kill the RTD entry for my project, loosing some SEO love....

[agjohnson]

@mistercrunch if you don't use these dependencies within your documentation at all, you can mock them out or use conda.

[cpascual]

I would love to see this too. We are having hard time mocking our deps in RTD and we already build on Travis. Conda may help here, but still it seems like duplicating efforts given that we already have everything in Travis...

Alternatively, if we could run in RTD the same docker containers that we already use in Travis, that would also be great.

[humitos]

Hi all! This topic was raised again and we started discussing about it and having a brainstorming of ideas. Although, we don't have a clear direction on what to support and/or how and/or where (community/corporate site).

We have some things to consider for anything that we implement:

• how to deal with spam/non-docs projects
• upload PDF/epub/singlehtml versions
• once a project upload docs what to do with "building docs on RTD"
• output is valid for RTD (contains our custom JS files that serve ads, or the flyout menu, for example)
• generate search index
• support multiple versions for uploaded docs
• support translations

On the other hand, we need to consider other things around the workflow,

• write the logic as a Sphinx extension to be able to get metadata from inside and rely on the user to install all the dependencies and deal with setup, latex, etc or,
• give users a Docker image (dealing with root permissions of output files) with everything pre-installed to build the docs and a guide to extend it or,
• ... another option

Please, if you have some ideas to share about this topic, write them down here to help us to take the right direction on this. We still don't know where this feature fits better, though.

[marscher]

The docker option also sounds very interesting as it would make mocking c-extensions completely obsolete.

[stsewd]

I have sphinx-click in my dev dependencies,

Do you mean you have them in your setup.py file? Then you should use the setup method to install your package https://docs.readthedocs.io/en/stable/config-file/v2.html#packages

[sumanthratna]

sphinx-click is in the dev dependencies section of my pyproject.toml (I don't have a setup.py). I think RTD detects if it's installing from a pyproject.toml, and if so, install dev dependencies that depend on sphinx.

[stsewd]

@sumanthratna I think a current workaround is using extras #4912 (comment) https://python-poetry.org/docs/pyproject/#extras

[humitos]

Hi @sumanthratna! If you have any suggestion that it's not related to "Upload pre-built docs", please open a new issue. There are people subscribed here waiting for this issue and receiving notification of an un-related feature right now. Appreciate :)

[bbannier]

We are generating documentation input from executables built from C++ sources in an expensive build process. Since we are a subproject of a larger project using readthedocs being, able to use readthedocs ourself would be great. It looks like our only option with this unimplemented is to redirect to some externally hosted documentation output.

[humitos]

We are generating documentation input from executables built from C++ sources in an expensive build process.

@bbannier If it's only a problem about resources, please open a new issue so we can take a deeper look and maybe we can assign the resources your project needs. Thanks!

[drewcassidy]

My project requires building the docs from a C++ library that requires GCC-10 or llvm+libc++ to build, so it cant be built on the RTD docker images. this would be nice to see

[marscher]

What about allowing certain docker images for building (on RTD of course), validating the build output outside of that and then, if a "stack"-reviewer accepts it, release it into the wild wild internet.
I guess there must be a duplicate already suggesting this.

[agjohnson]

👋 So, just an update here for anyone following along:

This is loosely going on our roadmap for sometime this year. However, we have a number of technical pieces we need to handle first. There are some big question marks here still.

---

That said, a number of the issues in comments here already have solutions available, without requiring upload of pre-built documentation. In the better part of a decade this issue has been open, we've addressed a number of build and dependency issues with new features:

My project requires special OS packages

You can use build.apt_packages to hopefully install what you need now:

• https://docs.readthedocs.io/en/stable/config-file/v2.html#build-apt-packages

My project needs to execute special commands before build

We'll have a beta in the next couple weeks for build.jobs configuration option, enabling commands pre/post build steps. This should solve a number of issues users have with building their documentation outside their local dev environment.

• Build: implement build.jobs config file key #9016

My project executes different commands than Read the Docs

We are gathering feedback for this feature. Perhaps build.jobs implemented in #9016 works for most projects already.

• Build: implement build.commands override of build commands #9062

---

If you have other use cases that are not well covered by our build infrastructure now, feel free to open a separate issue.

I'll keep this issue open, but we'll be putting discrete issues on our roadmap. I'll remove the design decision designation, not to communicate we have a concrete plan here, but we'll have discussion on individual issues as they come up. There are certainly a lot of questions here still.

[agjohnson]

We haven't had any serious progress here yet this year, but I am currently experimenting with a solution for this using build.jobs and magic-wormhole for file transfer towards RTD. So far, a script starts a new build and then sends files to the build process directly, while the build process waits for these files.

Noting for later, my proof of concept is at:
https://github.com/readthedocs/test-builds/tree/wormhole

I wouldn't recommend this for use yet, there are a few things that will likely change here. I think a one-time codeword would be a good addition and wormhole relays seem unstable. A drop in replacement would be croc or maybe even running a private relay

[stefanseefeld]

I'm also very interested in this feature. Our build processes are quite complex and involve a substantial amount of work to complete. Documentation is extracted partly from the compiled code (notably Python extension modules built using C++ and Boost.Python), so running those build processes within RTD is not really an option.
The best solution would really be to have a (REST ?) API to push the generated docs directly into RTD, rather than building them on-site.

[humitos]

@stefanseefeld

I'm also very interested in this feature.

Have you tested the solution proposed by @agjohnson in #1083 (comment). We would appreciate your feedback on that.

[wrinch]

Hi, I'm interested in this as well, but from a slightly different Standpoint. We are using OctopusDeploy to release our python modules to multiple environments. I don't want to release the Documentation until the code hits a specific environment.

So I want to build the docs once, then deploy them to each environment as the release proceeds though out testing process.

Also, we have a complicated build pipeline with a bunch of C++ dependencies and CMake etc. So I would really just like a HTTP REST API to upload a documentation package ideally...

Just putting this here to add a little different perspective. GitHub branches in this case aren't our ideal workflow. I could probably get around it with tagging... But not ideal. And then there is the whole build steps...

[stefanseefeld]

Just FYI, I have found https://github.com/Polyconseil/dokang which provides a simple documentation hub to which one can push any documentation html tree. This works reasonably well for me, but isn't actually maintained (and much less developed), so I would very much welcome any effort to offer this reduced feature set as part of the readthedocs project.

[g3n35i5]

Is there any update on this topic?

[humitos]

Unfortunately, not too many relevant updates here. However, we are going to discuss about this topic again in the near future and figure it out if there is a potential way to integrate this idea into Read the Docs without breaking the versions pattern we use.