Documentation overhaul roadmap

[darsnack]

This issue is a WIP. I am starting from the template discussed on call, we need to break this into smaller steps.

Flux documentation restructure

It should surprise no one that Flux's current documentation is not very good. It might surprise people that the "whole suite" of docs does have reasonably good coverage—that speaks to the poor organization of the documentation. Despite being written down, most information is an ordeal to find.

With this in mind, we consolidated a list of steps to improve this situation. These steps have been discussed by maintainers and contributors on various Julia platforms, but this issue will now serve as a central tracking for the re-haul.

• Restructure the Flux.jl documentation
  The current doc model was designed at a time when Flux's ethos was to demonstrate how simple neural networks in Julia can be. The current package offers a lot more than a simple list of layers and tracker-based AD. The proposed organization here is a commonly used format (notably in Python fast.ai and our own SciML). See below for a description of this format.
• Deprecate website tutorials and model zoo in favor of documentation
• Detailed documentation for NNlib.jl, FluxTraining.jl, etc.

New documentation structure

Home / Getting Started: Briefly describe the package and quick installation instructions. Provide links to (a) the entry point for an absolute beginner, (b) the ecosystem page, (c) contributing and bug reporting resources.

Ecosystem: This page is currently way down at the bottom, and it needs to be prominently featured. This should immediately make clear what it is the Flux provides (layers + simple utilities) and what it does not / what it borrows (datasets, data loading, optimizers, schedules, etc.). A diagram like MLJ's would be useful.

Tutorials: These should be complete, runnable examples that users can use to get started quickly and learn by doing. The model zoo examples should be tutorials. Literate.jl offers the ability to generate markdown for consumption by Documenter.jl. Tutorials should leverage this feature.

• Beginner: These should be basic tutorials like "taking your first gradient" or "running MNIST." They should rely heavily on the ecosystem packages that a typical beginner would use. For example, use MLDatasets.jl to fetch MNIST instead of rolling your own dataset downloader + type. Writing a data pipeline from scratch provides no value to someone who may not even know what a CNN does. Some existing parts of the Flux docs that fit this area are:
  • "Overview" which should be renamed to "First steps" or "What is machine learning?"
  • "Basics" which should be renamed to "Getting started with neural networks" or "What is a layer?"
  • "Training" which probably should be re-written to be more like a tutorial on understanding the basics of a training loop (i.e. remove the docstrings, etc.)
• Intermediate: These should be where most model zoo examples should end up. Existing pages that belong here are:
  • "Regularisation"
• Advanced: These should be where complex examples with unconventional training schemes end up.

How-tos: Unlike tutorials, which are focused on end-to-end examples, how-tos are focused on small bite-sized examples. They should demostrate doing a single thing only. Existing pages here could be:

• "GPU Support"
• "Saving & Loading"
• "Custom model example" which could be "Writing custom layers"

Reference documentation: These pages explain how certain concepts are designed in Flux from first principles. They are pedagogical like tutorials, but they are not focused on end-to-end examples. Instead of bringing together multiple concepts, reference docs bring explain a single concept in the context of FluxML.

• "Recurrence"
• "Performance tips"

API documentation: Flux likes to weave docstrings in and out of tutorials which is horrible for finding API references. We should have a single category that is nothing but docstrings organized by package and type.

Developer documentation: Flux currently lacks explanations for how things work under the hood. This is a place to explain how all the packages (Functors.jl, NNlib.jl, Zygote.jl, etc.) come together in a gradient call.

[Saransh-cpp]

Maybe we can start by segregating the current getting started and tutorials sections into separate folders? We can also add a new "how-to" section with the Dataloader example to start things up!

[mcabbott]

"Overview" which should be renamed to "First steps" or "What is machine learning?"
"Basics" which should be renamed to "Getting started with neural networks" or "What is a layer?"

These have already been re-titled, to "Fitting a Straight Line" and "Gradients and Layers".

"Training" which probably should be re-written to be more like a tutorial on understanding the basics of a training loop (i.e. remove the docstrings, etc.)

Yes. Doing this is a big job which I didn't want to try in #2069. It's also tangled up with the Optimisers section, which IMO should become a pure API listing of rules, and the other text moved into Training.

These changes are all a bit tangled up with #2082 etc. They probably want to be done at the same time as changing to explicit parameters in that section.

API documentation: Flux likes to weave docstrings in and out of tutorials which is horrible for finding API references. We should have a single category that is nothing but docstrings organized by package and type.

Yes. The book emoji from #2069 are an attempt to do this, without moving too much text in individual files. They aren't great, and a folder separation would probably be better.

One vision for folder structure is:

#2069 (comment)

This rolls many existing pages into "Guide".

The proposal above is to group those into many more different categories, according to whether they are for beginners / experts / developers / how-to / reference. I'm not so sure this will help make things easier to find, IMO too many top-level titles (with one or two children) make for a guessing game as to which title the topic of interest is likely to be filed under. If we can get away with a fairly flat structure (10 pages, starting with the beginner ones) then maybe page titles don't need a lot of further grouping.

The separation which does seem clear to me is that "Guide" is documentation about some part of Flux, and "Tutorials" are worked examples of using Flux. Almost every item in https://fluxml.ai/tutorials/ except 1 goes data, architecture, fit... which is great, and clearly a different animal. Learn-by-doing, not by being told. It's fine if several cover the same ground. Whereas "Guide" should try to say each thing once, so that you can be sure you haven't missed the place where it said X.

The model zoo examples should be tutorials. Literate.jl offers the ability to generate markdown for consumption by Documenter.jl. Tutorials should leverage this feature.

I think moving the website's tutorials to the manual would be a great start here. I think we want this regardless of other organisation questions. It could prototype whatever the pipeline with Literate.jl etc. looks like.

IIRC Literate wants to run the code while generating docs. Making sure that tutorials run would be great.

The ones there have quite a lot of text interspersed with code. The model zoo is just code. Is the vision that the model zoo examples should gain a lot of new prose? It also seems OK to have "easy" tutorials first (lots of hand-holding) and "harder" ones later.

place to explain how all the packages (Functors.jl, NNlib.jl, Zygote.jl, etc.) come together

This would be nice to have. I don't know where it ends up living exactly, but a page explaining a bit why things are scattered like this, and what to look for in which package, would be nice.

[dorn-gerhard]

I could contribute on the tutorial side. I am fan of interactive Pluto notebooks, so you could directly play around with some neural networks. I think for a cool tutorial example you should be able to explore the data, see the learning process (training and validation) and check the result (testing)

Are embedded Pluto Notebooks an option for the documentation?
(like on the computational thinking course)

[mcabbott]

Pluto is great. Do you have in mind something small enough that if it re-runs training repeatedly, waiting for it isn't too much of a pain?

Not quite sure how it could be set up. Hosting live notebooks may be hard, but perhaps something pre-computed like this could work:

https://github.com/JuliaPluto/PlutoSliderServer.jl#3-wip-precomputed-slider-server

Finally, at the moment the docs use Documenter which wants individual pages in as plain markdown files. Do you know if anyone has made Pluto-generated HTML fit in nicely with that?

[ToucheSir]

If fonsp/Pluto.jl#985 can be leveraged to be relatively ergonomic for new users (i.e. it's clear why certain cells aren't running and how to enable them), then Pluto could be in the running. That and making sure the generated file is legible enough. Last I checked it does a decent job of not cluttering source code, but didn't check how markdown blocks look.

[dorn-gerhard]

regarding disabled cells, it's now way easier to find the reason why a cell is disabled, you can simple jump to that cell.

What I have in mind for a sample notebook is one, where you can draw digits and and the notebook tries to recognize it.
Using some user input, like Connor did.

Not 100% sure how to integrate Pluto notebooks into Documenter but I will ask ;)