cargo doc should delete search-index.js before invoking rustdoc

[lolbinarycat]

Problem

Unlike the vast majority of build artifacts, search-index.js is both read and written by rustdoc, in order to append to it in a structured way.

This has the unfortunate side effect of persisting some state across cargo doc runs, as rustdoc will be able to see the output of the previous invocation.

This cannot be easily addressed within rustdoc, as it has no way to tell the difference between two invocations of rustdoc within one larger cargo doc invocation, and two different invocations of rustdoc across two different cargo doc invocations.

The weird read+write behavior of rustdoc is required because it does not have any equivalent to the "linking" step that rustc does, in which it can do one-time finalization and collection.

see rust-lang/rust#140698 and rust-lang/rust#133544 for some of the issues this causes

Proposed Solution

target/doc/search-index.js should be deleted by cargo doc before doing a full documentation rebuild (such as when the command-line arguments have changed, like in --document-private-items).

Notes

I believe crates.js also has a similar behavior.

[epage]

Somewhat related: #8487

[epage]

As cargo doc is intended to be additive across runs, I suspect we'd have a hard time deciding when the results should or shouldn't be additive.

[GuillaumeGomez]

Agreed with @epage: if you remove the search index, you need to rebuild the doc of the current crate and of all its dependencies.

[lolbinarycat]

@GuillaumeGomez but, if we're already going to do that, then perhaps cargo should delete things?

it feels quite difficult to fix these problems in rustdoc without support from cargo.

maybe we could add cargo doc --pure or something to ensure deterministic docs without getting rid of precompiled rlibs or the incremental build cache?

[GuillaumeGomez]

I's not something we can fix in rustdoc. We could exit with a specific error or provide information in a specific way to cargo so it knows it needs to rebuild all docs.

[lolbinarycat]

what about the case of duplicating crates in the search index? how can we fix that without breaking the case where there actually are multiple crates with the same name?

[GuillaumeGomez]

Seems very unlikely to happen, no?

[lolbinarycat]

which is unlikely, duplicating search index entries or a situation where multiple crates with the same name coexist in a dependancy tree?

the former is the cause of rust-lang/rust#133544, while the latter is easily caused by transitively depending on multiple semver-incompatible versions of the same crate.