Experimental --as-json option for the HTML renderer

[sabine]

This adds an experimental flag --as-json to odoc html to emit HTML fragments (preamble, content) together with JSON metadata (whether the page uses katex, breadcrumbs, table of contents).

--as-json replaces the --omit-toc, --omit-breadcrumbs, and --content-only options of odoc html.

Purpose:
Make it simpler and more robust to embed the HTML output of odoc in websites, such as ocaml.org, which is currently missing all preambles and has no reliable way to distinguish between .mld pages and modules.

Notes:

• In order to abstract breadcrumb rendering, we treat breadcrumbs the same as the table of contents: there is now an intermediate data type breadcrumb used by the HTML renderer, from which either HTML or a JSON representation is rendererd.
• In order to avoid introducing a JSON encoder dependency to odoc, @dbuenzli contributed the JSON encoder contained in this PR. All remaining additions and changes are to be blamed on me.

This PR enables us to fix the following issues on ocaml.org:

• "Missing title and first paragraph" of Some more documentation regression ocaml.org#203
• docs generation: error with mld files in title ocaml.org#583

[jonludlam]

I like this, it's a lot neater than the more general approach I was initially taking. My only objection, as before, is the json encoder. I'm content to live with it though. I also am aware that the compiler is also looking to output structured data in a parsable format for error/warning reporting for tooling, so I hope that encoder ends up exposed in a reasonable way and we should aim to switch to it as soon as we can.

[sabine]

Indeed, it looks to me like the next best alternative to wrapping up the HTML preamble and content block together with the corresponding metadata (what this PR does) would be to emit the structured data of the document and have the embedding site/product perform all of the HTML rendering.

However, it looks to me like the HTML+metadata emitted here is likely to be more convenient to integrate into web-browser-based products.

[dbuenzli]

Is the html-targets command updated accordingly by this PR ? (a very quick look seems to indicate that it's not the case).

[sabine]

Ah, sorry about html-targets, you're correct, it's not updated. I'm fixing that.

[sabine]

I was too quick to assume a problem with html-targets. There seems to be no need to update anything, as the filenames are obtained from the renderer. In the force-push, I added a test that confirms that html-targets returns the .html.json filepaths (https://github.com/ocaml/odoc/pull/908/files#diff-8f34ee7dd9df5690704237b21e8125187f6da349ce8be06a25446635b2c5a36fR24).

[Julow]

That's not the API that I dreamed to see. I think it makes the API more complicated than needed to have a tree of JSON files instead of a single structured output.
I believe a new sub-command that outputs its output on stdout would be more convenient. It would also allow to output the list of subpages, parent-child, canonical path, whether it has source code (coming soon) in a structured way.

With that said, I think the approach here is fine and I don't want to block it.

I've submitted a PR to your PR to add a test about subpages: sabine#1

[sabine]

Thank you @Julow, your PR that adds better tests is merged.

I believe the issue of wanting to have one single structured output is orthogonal to the HTML+metadata output and could be made as a separate PR, improving on this one.

[jonludlam]

That's not the API that I dreamed to see. I think it makes the API more complicated than needed to have a tree of JSON files instead of a single structured output.
I believe a new sub-command that outputs its output on stdout would be more convenient. It would also allow to output the list of subpages, parent-child, canonical path, whether it has source code (coming soon) in a structured way.

Though a benefit to the approach in this PR is that dynamically loading pages becomes an awful lot quicker, especially for large libraries that contain a lot of modules. A build of core produces 51Mb of html files, the largest of which is 400k.

[jonludlam]

It's important to note that this interface is experimental, and we should feel we can change the output format without preserving any kind of backward compatibility. I've seen the warnings in the CHANGES and in the help, but I'm quite tempted to output something to stderr when running html-generate too.

[dbuenzli]

but I'm quite tempted to output something to stderr when running html-generate too.

Please don't, that will just annoy people who are using the stuff anyways.

[Julow]

could be made as a separate PR, improving on this one.

I'm not sure. This API will be used I suppose so such a change would either be breaking or add a new command without removing the --as-json option.

Though a benefit to the approach in this PR is that dynamically loading pages becomes an awful lot quicker, especially for large libraries that contain a lot of modules

A different format could be used. For example, parsexp allows reading concatenated sexp one by one. There would be a first object containing the list of output, then one per output.

[jonludlam]

but I'm quite tempted to output something to stderr when running html-generate too.

Please don't, that will just annoy people who are using the stuff anyways.

I just really don't want people to be upset when we inevitably change something here. Maybe I'm being paranoid? Perhaps --void-warranty to enable the feature :-)

[dbuenzli]

I just really don't want people to be upset when we inevitably change something here.

People get upset for all sorts of valid and invalid reasons. The only thing you can do is have your own peace of mind. Marking the feature as unstable both in the release notes and in the docs seems enough to counter any future anger.

[sabine]

To clarify about the output format: For ocaml.org, having these individual .html.json files (that directly correspond to the .html files odoc would otherwise output) is convenient to work with: every URL can be served by looking up one .html.json file and rendering the full page. So if there is a different output format, it is highly likely that our documentation pipeline would break up the output to store it in individual files.

[Julow]

The pipeline has all the time it needs to present the data like it wants, Odoc shouldn't worry about that.
However, it is useful to consider very large outputs, the tree of json doesn't look so bad after all.

[jonludlam]

The tests were broken on 4.02/3 due to ordering issues on the command line. Have pushed a fix.