Write inventory files

[goerz]

This automatically writes inventory files objects.inv and inventory.toml.gz to the build folder, which should get deployed together with the rest of the documentation. The presence of these files allows the DocumenterInterLinks plugin to link into the documentation from any other project with an @extref link.

The PR adds a single new dependency: CodecZlib. This is necessary because inventory files should definitely be compressed (they potentially get pulled a lot, every time someone builds a documentation that has an @extref into the project). The CodecZlib is maintained by the JuliaIO org and should be quite stable.

Also see the discussion in #2366 (comment)

Closes #2366

[goerz]

Pending tests, additional documentation, and a decision on whether it's okay to write objects.inv and inventory.toml.gz, or one or the other.

Also, #2385 should be resolved before merging this (preferably via #2389)

[goerz]

I've marked this as "Ready for review" since it now has tests and documentation (no CHANGELOG yet).

I agree with #2366 (comment), though: we should probably choose one format. I'd also lean towards the Sphinx objects.inv, if that's palatable to everyone. The major pros and cons are outlined in #2366 (comment). On top of that, not writing the toml.gz would cut a significant number of lines from this PR, so that might be another pro.

The documentation is already written to only mention objects.inv, although the code still generates both objects.inv and inventory.toml.gz. I'll clean up and adapt after we make a decision.

[goerz]

There's also the follow-up #2433 which adds the option for the version in the inventory.

There should be some way to specify a version, and ideally, it would be a solution so that when the new version of Documenter is released and starts to be used, existing projects end up with the correct version in their inventory, without having to put a lot of work in.

That's why in #2433, I also added the fallback of searching for a Project.toml that specifies a version.

[goerz]

Community call decision was to only write the Sphinx-format objects.inv. So I'll remove the code for the toml-writing and update everything, and let you know when that's done.

[goerz]

@mortenpi Okay, should be ready for merging or further comments.

There was actually still a bug: .md filenames in the documentation with spaces weren't handled correctly (not that I would ever recommend them, but they do work, and now they also have a properly escaped URI in the inventory)

#2433 is also ready. If you like #2433, you could just change the base branch for that PR to master and merge it instead of this PR. Or let me know, and I can merge it into this PR before we merge this PR into master.

Edit: I forgot to update the CHANGELOG. I'll do it tomorrow, or maybe we can decide first how we want to handle #2433 so I can do one changelog for the combined changes.

[mortenpi]

What's the fallback for Version: when you can't determine it from Project.toml? Empty string? It's still something that's relevant when you're generating the HTML for something that does not have a well-defined version number (like https://github.com/JuliaDocs/juliadocs.github.io).

I know you don't like this idea, but I still feel that the "right way" to do it right now is to patch the inventory file in the deploydocs phase. It should be a reasonably simple patch of the file: just readlines through the inventory, find the ^Version: line, and if the version number is empty, add the correct version then?

Using Project.toml would be automatic for the user, but it introduces two sources of truth for the version number, and I'm really allergic to that.

[mortenpi]

Also, for main/master, what do you think the Version: value should be? In practice, packages are not particularly consistent with whether they switch to vX.Y.Z-DEV after a version was tagged. In deploydocs, we could enforce something. E.g. I would be okay with deploydocs going into Project.toml, figuring out the version number, and then making sure there's a -DEV at the end, if it's a "dev" version.

[goerz]

What's the fallback for Version: when you can't determine it from Project.toml? Empty string?

Yes, empty string and a warning to set a manual version as a parameter to HTML.

I know you don't like this idea

Indeed :-)

but I still feel that the "right way" to do it right now is to patch the inventory file in the deploydocs phase

Alright, fair enough. Let me have a stab at that. I'll make an alternative to #2433.

You're okay with the new version parameter, in HTML, though, right? So I'll just remove the fallback to Project.toml, and add something in deploydocs that patches in a version.

[goerz]

Also, for main/master, what do you think the Version: value should be?

I actually have very strong opinions on that, but you're right, most people aren't very strict or consistent about that.

In deploydocs, we could enforce something. E.g. I would be okay with deploydocs going into Project.toml, figuring out the version number, and then making sure there's a -DEV at the end, if it's a "dev" version.

It's probably not that critical what exactly the version inside the inventory is for a dev-version. But I guess deploydocs right now only really knows that it's "dev", without any related version number? I'll have to have a look at the code in deploydocs...

[mortenpi]

You're okay with the new version parameter, in HTML, though, right?

Actually... I was kinda hoping we wouldn't need it 😅 But I would be okay with version or inventory_version, if you think we need the ability to set the value in cases where deploydocs can't help us (it seems like a reasonable thing you want to do sometimes).

One alternative possibility: we can have a public API function that you can use to patch the inventory file, which deploydocs will use, but the user can also use if they do need to set the version, and they don't use deploydocs. But.. now that I typed this out.. probably better to have just an argument to HTML.

On version vs inventory_version: I would prefer not to make it quite an official to set a version right now, due to the multiple-source-of-truth issue. For better or for worse, I think we need deploydocs to be in charge of the version numbers right now. The only user now would be the inventory writing, so let's make the option specific to that?

[goerz]

I’m fine with inventory_version, and I’ll have a look at deploydocs

[goerz]

Just to keep the discussion up-to-date here, we now have three competing PRs for completing this (regarding what is the best way to set the inventory version), see #2443 (comment) for details:

• Obtain inventory version from Project.toml #2442 – makedocs gets the version for the inventory from the closest Project.toml/JuliaProject.toml/VERSION that defines a version. Then deploydocs checks whether that version matches the tag for the release that's being deployed (for non-dev deployments). There is no need for a manual inventory_version option. I'd be extremely happy with this option!
• Overwrite inventory version in deploydocs #2449 – Initially set the inventory version in makedocs based on the closest Project.toml/JuliaProject.toml/VERSION file that defines a version on the Project.toml file one level up from the docs folder. This automatic version is only for standard setups. For anything else (monorepos?), makedocs will not set a version beyond what is manually given via the inventory_version option. Finally, unconditionally overwrite the version in deploydocs for a tagged release (where deploydocs can know the correct version). This overwrite shouldn't actually change the version for all normal setups, but it'll still let deploydocs have the last word. I'd be happy with this option as well.
• Patch inventory version in deploydocs with fallback to Project.toml #2448 – The inventory version is set by deploydocs. For tagged releases, it uses the version from the tag, and for dev-releases, it falls back to the same logic as Obtain inventory version from Project.toml #2442 to get the version from the closest Project.toml. The ultimate result is identical to Obtain inventory version from Project.toml #2442 (at least that's the intent), but the code is a bit more convoluted than necessary, and I'm not enthusiastic that the objects.inv file isn't generally complete after the makedocs step.
• Patch in inventory version in deploydocs #2443 – Set the version in deploydocs without a fallback. As a result, dev-deployments would typically have an empty version in the inventory, and we'd have to push people to modify their setup and specify a manual inventory_version. I'd find this one hard to swallow.

[goerz]

This has been updated by squash-merging #2449 as the solution for setting the version metadata that we reached consensus on.

I've also added a CHANGELOG entry.

[goerz]

@mortenpi This should be ready now for a final review (if there's any more bikeschedding) and merging.

There is one issue we might still have to think about: Since we've settled on auto-detecting the version only for "standard setups", that no longer includes the VERSION file that Julia itself uses. I'd put a high priority on the inventory file for Julia (including the nightly version) being in good shape, ideally without them having to tweak their setup. I think it will work out of the box, since they seem to have a custom setup that deploys nightly to https://docs.julialang.org/en/v1.12-dev/, i.e., a folder that includes a proper version, not just the standard dev.

On the other hand, they might also pin a specific Documenter version, and in that case we'll have to make a PR to Julia to update that to the new release anyway, which then might as well include setting inventory_version manually. I'll try to look into this or find someone on Slack with a better understanding of Julia's documentation setup.

I've rebased everything onto the current master and rewritten the history to be somewhat sane. If possible, I would prefer to merge this with a non-fast-forward merge commit (as opposed to a squash-merge) in order to preserve some of the history, especially the commit messages referencing the various alternative PRs about how to set the version. I'm actually not quite sure if there's a button in the Github UI to do that kind of merge (I think it would be "Create a merge commit"), but on the command line, it would be git merge --no-ff mg/write-inventory from master.

I've assumed that this will make it into a v1.3.0 release eventually (and set version = "1.3.0-dev" in the Project.toml accordingly). I still have to do some tweaking of the DocInventories/DocumenterInterLinks documentations to adapt to the new reality of inventory-writing now being built in to Documenter ("As of Documenter 1.3.0…"). For the same reason, I also wouldn't mind terribly if we waited at least until this coming Monday to actually tag a new Documenter release.

After we make the release, it might make sense to make some kind of announcement on Discourse about the new inventory feature. I'll also want to make a separate announcement of the DocumenterInterLinks plugin, but I want to polish that just a little more before I remove the official "experimental, use at your own risk" warning.

[goerz]

I'd put a high priority on the inventory file for Julia (including the nightly version) being in good shape, ideally without them having to tweak their setup. […] On the other hand, they might also pin a specific Documenter version

Thanks to @vchuravy's help on Slack, I was able to determine that the Julia documentation builds pretty much just by running doc/make.jl, like any other package. Just for future reference, the building and deployment of Julia's documentation is set up at https://github.com/JuliaCI/julia-buildkite, specifically in the files
doctest.yml and deploy_docs.yml.

There is a custom DeployConfig in doc/make.jl. Just from looking at that, it seems to me that deploydocs would set the correct inventory-version automatically during deployment.

However, there is also a docs/Manifest.toml file that pins the Documenter version, so we'll have to submit a pull request to update that. The precedent for that is JuliaLang/julia#47105. So, @mortenpi has done this before, and we should be fine :-)

I would still include adding an inventory_version=VERSION to the HTML() options for the makedocs in that PR as well, though. It would be good that if people build the Julia documentation locally, they still get an objects.inv file that has the correct version set (and, with the design we have now, makedocs would not be able to auto-detect the version with Julia's setup).

Bottom line: Julia's peculiarities are not a blocker for merging this PR, but we will have to follow up in order to ensure that future deployments of Julia's documentation have an inventory.

[goerz]

I also wouldn't mind terribly if we waited at least until this coming Monday to actually tag a new Documenter release.

Actually, we can tag it right away, but maybe hold off a few days with any kind of Discourse announcement. That might even give us a chance to

follow up in order to ensure that future deployments of Julia's documentation have an inventory.

and for a handful of other packages to make a release/build with the newest Documenter before we go public; so it's not quite "you can link to any other project, but there's basically nothing yet that actually has an inventory" (despite the handful of inventories I've collected in the wiki)

[goerz]

It turns out the inventory-writing-code is more or less compatible all the way back to Documenter v0.25, so I put that at https://github.com/JuliaDocs/DocumenterInventoryWritingBackport.jl (currently waiting for registration). So, no excuses for any package not to have an inventory, even if they didn't like dealing with the Documenter 1.0 update.

[mortenpi]

Sorry for the delay, but LGTM!

[goerz]

@mortenpi Can I merge this now (manually)? Ideally, before we merge too many other PRs, since otherwise I'll have to go through the rebase and conflict resolution again.

[mortenpi]

Yeah, let's merge!

[goerz]

Ah, branch protection!

I'd still kinda like to preserve some of those commit messages with the references to the discussion, so not sure what to do. Maybe temporarily remove the protection, or maybe you can push with higher permissions than me.

(On the other hand, the references are also in the PR comments, so I guess the usual squash would be ok, too. It's up to you.)

[mortenpi]

Let's just squash merge this. The PRs offer a better context for the decisions here I think.