Unexpected test failure on CI for scrape_examples_basic

[ehuss]

The doc::scrape_examples_basic test unexpectedly failed in CI for PR #10116:

---- doc::scrape_examples_basic stdout ----
running `D:\a\cargo\cargo\target\debug\cargo.exe doc -Zunstable-options -Z rustdoc-scrape-examples=all`
thread 'doc::scrape_examples_basic' panicked at 'assertion failed: p.build_dir().join(\"doc/src/ex/ex.rs.html\").exists()', tests\testsuite\doc.rs:2187:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    doc::scrape_examples_basic

I cannot reproduce this locally using macOS or nightly-x86_64-pc-windows-gnu.

Perhaps the CI machine was having a bad day? Opening this to track to see if it ever happens again.

nightly-x86_64-pc-windows-gnu
rustc 1.58.0-nightly (b426445c6 2021-11-24)

[ehuss]

This continues to happen with a low frequency.

@willcrichton is this something you could take a look at? I've been doing some testing on CI, and it seems to happen rarely (maybe once out of a thousand loops). The behavior is that everything is on disk except for the target/doc/src/ex/ex.rs.html file. Here's some raw output of the test with some debug information:

---- docscrape::basic434 stdout ----
running `D:\a\cargo\cargo\target\debug\cargo.exe doc -Zunstable-options -Zrustdoc-scrape-examples -v`
    Checking foo v0.0.1 (D:\a\cargo\cargo\target\tmp\cit\t374\foo)
     Running `rustc --crate-name foo src\lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat --crate-type lib --emit=dep-info,metadata -C embed-bitcode=no -C debuginfo=2 -C metadata=77403d1681517c41 -C extra-filename=-77403d1681517c41 --out-dir D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\debug\deps -L dependency=D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\debug\deps`
    Scraping foo v0.0.1 (D:\a\cargo\cargo\target\tmp\cit\t374\foo)
     Running `rustdoc --crate-type bin --crate-name ex examples\ex.rs -o D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat -C metadata=b82abb4116288145 -Zunstable-options --scrape-examples-output-path D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\debug\deps\foo-b82abb4116288145.examples --scrape-examples-target-crate ex --scrape-examples-target-crate foo -L dependency=D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\debug\deps --extern foo=D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\debug\deps\libfoo-77403d1681517c41.rmeta --crate-version 0.0.1`
 Documenting foo v0.0.1 (D:\a\cargo\cargo\target\tmp\cit\t374\foo)
     Running `rustdoc --crate-type lib --crate-name foo src\lib.rs -o D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc --error-format=json --json=diagnostic-rendered-ansi,artifacts,future-incompat -C metadata=77403d1681517c41 -Zunstable-options -L dependency=D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\debug\deps --crate-version 0.0.1 --with-examples D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\debug\deps\foo-b82abb4116288145.examples`
    Finished dev [unoptimized + debuginfo] target(s) in 1.39s
running `D:\a\cargo\cargo\target\debug\cargo.exe doc -Zunstable-options -Z rustdoc-scrape-examples`
    Finished dev [unoptimized + debuginfo] target(s) in 0.04s
D:\a\cargo\cargo\target\tmp\cit\t374\foo 4096
D:\a\cargo\cargo\target\tmp\cit\t374\foo\Cargo.lock 147
D:\a\cargo\cargo\target\tmp\cit\t374\foo\Cargo.toml 131
D:\a\cargo\cargo\target\tmp\cit\t374\foo\examples 0
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\COPYRIGHT-23e9bde6c69aea69.txt 1831
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\dark-cf923f49f397b216.css 3723
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\favicon-16x16-8b506e7a72182f1c.png 715
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\favicon-2c020d218678b618.svg 4298
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\favicon-32x32-422f7d1d52889060.png 1125
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\FiraSans-LICENSE-db4b642586e02d97.txt 4461
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\FiraSans-Medium-8f9a781e4970d388.woff2 132780
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\FiraSans-Regular-018c141bf0843ffd.woff2 129188
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\LICENSE-APACHE-b91fa81cba47b86a.txt 10847
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\LICENSE-MIT-65090b722b3f6c56.txt 1023
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\light-db279b6232be9c13.css 3598
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\main-f5a2577c5297a973.js 25376
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\NanumBarunGothic-0f09457c7a19b7c6.ttf.woff2 399468
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\NanumBarunGothic-LICENSE-18c5adf4b52b4041.txt 4787
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\normalize-76eba96aa4d2e634.css 1853
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\noscript-13285aec31fa243e.css 161
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\rust-logo-151179464ae7ed46.svg 3297
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\rustdoc-c4dbdcde0fbd8430.css 29547
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\scrape-examples-ef1e698c1d417c0c.js 2430
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\search-ab3e9174d42dd6c2.js 36974
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\settings-0bcba95ff279c1db.css 1440
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\settings-f0c5c39777a9a2f6.js 5749
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\source-script-1b95b7cca98b26e5.js 4554
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\SourceCodePro-It-1cc31594bf4f1f79.ttf.woff2 44896
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\SourceCodePro-LICENSE-d180d465a756484a.txt 4570
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\SourceCodePro-Regular-562dcc5011b6de7d.ttf.woff2 52228
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\SourceCodePro-Semibold-d899c5a5c4aeb14a.ttf.woff2 52348
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\SourceSerif4-Bold-a2c9cd1067f8b328.ttf.woff2 81540
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\SourceSerif4-It-acdfaf1a8af734b1.ttf.woff2 59716
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\SourceSerif4-LICENSE-3bb119e13b1258b7.md 4727
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\SourceSerif4-Regular-46f98efaafac5295.ttf.woff2 76260
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\storage-9184409068f70b79.js 3557
D:\a\cargo\cargo\target\tmp\cit\t374\foo\target\doc\static.files\wheel-7b819b6101059cd0.svg 2972
foo:
<!DOCTYPE html><html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1.0"><meta name="generator" content="rustdoc"><meta name="description" content="API documentation for the Rust `foo` fn in crate `foo`."><title>foo in foo - Rust</title><link rel="preload" as="font" type="font/woff2" crossorigin href="../static.files/SourceSerif4-Regular-46f98efaafac5295.ttf.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../static.files/FiraSans-Regular-018c141bf0843ffd.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../static.files/FiraSans-Medium-8f9a781e4970d388.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../static.files/SourceCodePro-Regular-562dcc5011b6de7d.ttf.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../static.files/SourceSerif4-Bold-a2c9cd1067f8b328.ttf.woff2"><link rel="preload" as="font" type="font/woff2" crossorigin href="../static.files/SourceCodePro-Semibold-d899c5a5c4aeb14a.ttf.woff2"><link rel="stylesheet" href="../static.files/normalize-76eba96aa4d2e634.css"><link rel="stylesheet" href="../static.files/rustdoc-c4dbdcde0fbd8430.css" id="mainThemeStyle"><div id="rustdoc-vars" data-root-path="../" data-static-root-path="../static.files/" data-current-crate="foo" data-themes="" data-resource-suffix="" data-rustdoc-version="1.70.0-nightly (8a73f50d8 2023-03-11)" data-search-js="search-ab3e9174d42dd6c2.js" data-settings-js="settings-f0c5c39777a9a2f6.js" data-settings-css="settings-0bcba95ff279c1db.css" data-theme-light-css="light-db279b6232be9c13.css" data-theme-dark-css="dark-cf923f49f397b216.css" data-theme-ayu-css="ayu-be46fdc453a55015.css" ></div><script src="../static.files/storage-9184409068f70b79.js"></script><script defer src="sidebar-items.js"></script><script defer src="../static.files/main-f5a2577c5297a973.js"></script><script defer src="../static.files/scrape-examples-ef1e698c1d417c0c.js"></script><noscript><link rel="stylesheet" media="(prefers-color-scheme:light)" href="../static.files/light-db279b6232be9c13.css"><link rel="stylesheet" media="(prefers-color-scheme:dark)" href="../static.files/dark-cf923f49f397b216.css"><link rel="stylesheet" href="../static.files/noscript-13285aec31fa243e.css"></noscript><link rel="alternate icon" type="image/png" href="../static.files/favicon-16x16-8b506e7a72182f1c.png"><link rel="alternate icon" type="image/png" href="../static.files/favicon-32x32-422f7d1d52889060.png"><link rel="icon" type="image/svg+xml" href="../static.files/favicon-2c020d218678b618.svg"></head><body class="rustdoc fn"><!--[if lte IE 11]><div class="warning">This old browser is unsupported and will most likely display funky things.</div><![endif]--><nav class="mobile-topbar"><button class="sidebar-menu-toggle">&#9776;</button><a class="logo-container" href="../foo/index.html"><img class="rust-logo" src="../static.files/rust-logo-151179464ae7ed46.svg" alt="logo"></a><h2></h2></nav><nav class="sidebar"><a class="logo-container" href="../foo/index.html"><img class="rust-logo" src="../static.files/rust-logo-151179464ae7ed46.svg" alt="logo"></a><div class="sidebar-elems"><h2><a href="index.html">In foo</a></h2></div></nav><main><div class="width-limiter"><nav class="sub"><form class="search-form"><span></span><input class="search-input" name="search" aria-label="Run search in the documentation" autocomplete="off" spellcheck="false" placeholder="Click or press ‘S’ to search, ‘?’ for more options…" type="search"><div id="help-button" title="help" tabindex="-1"><a href="../help.html">?</a></div><div id="settings-menu" tabindex="-1"><a href="../settings.html" title="settings"><img width="22" height="22" alt="Change settings" src="../static.files/wheel-7b819b6101059cd0.svg"></a></div></form></nav><section id="main-content" class="content"><div class="main-heading"><h1>Function <a href="index.html">foo</a>::<wbr><a class="fn" href="#">foo</a><button id="copy-path" title="Copy item path to clipboard"><img src="../static.files/clipboard-7571035ce49a181d.svg" width="19" height="18" alt="Copy item path"></button></h1><span class="out-of-band"><a class="srclink" href="../src/foo/lib.rs.html#1">source</a> · <button id="toggle-all-docs" title="collapse all docs">[<span>&#x2212;</span>]</button></span></div><pre class="rust item-decl"><code>pub fn foo()</code></pre><div class="docblock scraped-example-list"><span></span><h5 id="scraped-examples"><a href="#scraped-examples">Examples found in repository</a><a class="scrape-help" href="../scrape-examples-help.html">?</a></h5><div class="scraped-example expanded" data-locs="[[[0,0],&quot;../src/ex/ex.rs.html#1&quot;,&quot;line 1&quot;]]"><div class="scraped-example-title">examples\ex.rs (<a href="../src/ex/ex.rs.html#1">line 1</a>)</div><div class="code-wrapper"><div class="example-wrap"><pre class="src-line-numbers"><span>1</span>
</pre><pre class="rust"><code><span class="kw">fn </span>main() { <span class="highlight focus">foo::foo</span>(); }</code></pre></div>
</div></div></div></section></div></main></body></html>
data:
\x01M6\x85\xfavC\x1b0\r\xe3A*\x8f`\xf7h\x01;\\\\?\\D:\\a\\cargo\\cargo\\target\\tmp\\cit\\t374\\foo\\examples\\ex.rs\xc1\x01\x0c\x16\x00\x00\x0c\x14\x00\x00\x00\x19\x00\x00\x11src/ex/ex.rs.html\xc1\x0eexamples\\ex.rs\xc1\x00\x01
thread 'docscrape::basic434' panicked at 'cannot find "D:\\a\\cargo\\cargo\\target\\tmp\\cit\\t374\\foo\\target\\debug\\deps\\foo-b82abb4116288145.examples"', tests\testsuite\docscrape.rs:1067:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

I took a look at the code that writes the source file to disk, and found this which looks extremely suspicious: https://github.com/rust-lang/rust/blob/f41927f30943e4d57af62cfcedc9f07b819013e7/src/librustdoc/docfs.rs#L62-L68

That is spawning a thread to write some content to disk. However, I don't see anything that waits for that thread to finish. If rustdoc finishes its work quickly, and that thread got delayed for whatever reason, then I think it will just exit without writing the file.

[willcrichton]

@ehuss that code does indeed look suspicious, thanks for finding it.

cc @rbtcollins @GuillaumeGomez: is there a reason that the DocFS writes aren't eventually synchronized? Seems like Rustdoc might need to wait for all spawned Rayon tasks to complete.

[GuillaumeGomez]

It's because of perf issues on windows. If we do sync writes, the performance drop can drastically.

[willcrichton]

Sorry, I understand why the writes are individually not synchronized. But shouldn't Rustdoc make sure that all the writes are completed before exiting the process?

[GuillaumeGomez]

Sorry missed that part. Yes it definitely should.

[willcrichton]

Ok, I just filed an issue with a proposed solution at rust-lang/rust#109060.

[rbtcollins]

Thats merged, can this be closed? Sorry for not responding earlier, but yes, clearly exiting before writes have been done will break :P. The rustup code this was inspired by does indeed wait for all the aggregate of IOs to complete.

[ehuss]

I've been doing some stress testing on CI, and it seems to be resolved. Thanks @GuillaumeGomez for the quick fix!