[crater only] count crates that have more than five doctests on one item

[notriddle]

Follow up for https://rust-lang.zulipchat.com/#narrow/channel/393423-t-rustdoc.2Fmeetings/topic/2024-12-09/near/487096843

cc @camelid @GuillaumeGomez

[rustbot]

r? @GuillaumeGomez

rustbot has assigned @GuillaumeGomez.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[GuillaumeGomez]

Looks good to me, thanks! Please start the crater run whenever you want. :)

[rust-log-analyzer]

The job x86_64-gnu-llvm-18 failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

#21 exporting to docker image format
#21 sending tarball 27.4s done
#21 DONE 34.9s
##[endgroup]
Setting extra environment values for docker:  --env ENABLE_GCC_CODEGEN=1 --env GCC_EXEC_PREFIX=/usr/lib/gcc/
[CI_JOB_NAME=x86_64-gnu-llvm-18]
debug: `DISABLE_CI_RUSTC_IF_INCOMPATIBLE` configured.
---
sccache: Starting the server...
##[group]Configure the build
configure: processing command line
configure: 
configure: build.configure-args := ['--build=x86_64-unknown-linux-gnu', '--llvm-root=/usr/lib/llvm-18', '--enable-llvm-link-shared', '--set', 'rust.randomize-layout=true', '--set', 'rust.thin-lto-import-instr-limit=10', '--enable-verbose-configure', '--enable-sccache', '--disable-manage-submodules', '--enable-locked-deps', '--enable-cargo-native-static', '--set', 'rust.codegen-units-std=1', '--set', 'dist.compression-profile=balanced', '--dist-compression-formats=xz', '--set', 'rust.lld=false', '--disable-dist-src', '--release-channel=nightly', '--enable-debug-assertions', '--enable-overflow-checks', '--enable-llvm-assertions', '--set', 'rust.verify-llvm-ir', '--set', 'rust.codegen-backends=llvm,cranelift,gcc', '--set', 'llvm.static-libstdcpp', '--enable-new-symbol-mangling']
configure: target.x86_64-unknown-linux-gnu.llvm-config := /usr/lib/llvm-18/bin/llvm-config
configure: llvm.link-shared     := True
configure: rust.randomize-layout := True
configure: rust.thin-lto-import-instr-limit := 10
---
  Downloaded boml v0.3.1
   Compiling boml v0.3.1
   Compiling y v0.1.0 (/checkout/compiler/rustc_codegen_gcc/build_system)
    Finished `release` profile [optimized] target(s) in 3.89s
     Running `/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-codegen/x86_64-unknown-linux-gnu/release/y test --use-system-gcc --use-backend gcc --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools/cg_gcc --release --mini-tests --std-tests`
Using system GCC
[BUILD] example
[AOT] mini_core_hello_world
/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-tools/cg_gcc/mini_core_hello_world
abc
---

test result: ok. 57 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 418.21ms

   Doc-tests alloc
error: DefId(0:160 ~ alloc[2297]::boxed) has more than five doctests
error: aborting due to 1 previous error


error: doctest failed, to rerun pass `-p alloc --doc`
Caused by:
Caused by:
  process didn't exit successfully: `/checkout/obj/build/bootstrap/debug/rustdoc --edition=2021 --crate-type lib --color always --crate-name alloc --test library/alloc/src/lib.rs --test-run-directory /checkout/library/alloc --target x86_64-unknown-linux-gnu -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps -L dependency=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/release/deps -L native=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/build/compiler_builtins-9abe3c2f6120cbbe/out --test-args --quiet --test-args -Z --test-args unstable-options --test-args --format --test-args json --warn=unexpected_cfgs --check-cfg 'cfg(bootstrap)' --check-cfg 'cfg(no_global_oom_handling)' --check-cfg 'cfg(no_rc)' --check-cfg 'cfg(no_sync)' --check-cfg 'cfg(randomized_layouts)' --extern alloc=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps/liballoc-5201ff2e83fd0a74.rlib --extern compiler_builtins=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps/libcompiler_builtins-f4a0e8d793a4eafe.rlib --extern core=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps/libcore-c9690195bf6f5d1b.rlib --extern rand=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps/librand-97b19f81b630f426.rlib --extern rand_xorshift=/checkout/obj/build/x86_64-unknown-linux-gnu/stage1-std/x86_64-unknown-linux-gnu/release/deps/librand_xorshift-48a2f5a0fec54b8d.rlib -C embed-bitcode=no --cfg 'feature="compiler-builtins-c"' --check-cfg 'cfg(docsrs)' --check-cfg 'cfg(feature, values("compiler-builtins-c", "compiler-builtins-mangled-names", "compiler-builtins-mem", "compiler-builtins-no-asm", "compiler-builtins-no-f16-f128", "optimize_for_size", "panic_immediate_abort"))' -Csymbol-mangling-version=v0 -Zrandomize-layout '--check-cfg=cfg(feature,values(any()))' -Zunstable-options '--check-cfg=cfg(bootstrap)' -Dwarnings '-Wrustdoc::invalid_codeblock_attributes' --crate-version '1.85.0-nightly (38764b1fa 2024-12-19)' '-Zcrate-attr=doc(html_root_url="https://doc.rust-lang.org/nightly/")' '-Zcrate-attr=warn(rust_2018_idioms)' --error-format human` (exit status: 1)
note: test exited abnormally; to see the full output pass --nocapture to the harness.
  local time: Thu Dec 19 19:35:44 UTC 2024
  network time: Thu, 19 Dec 2024 19:35:44 GMT
##[error]Process completed with exit code 1.
Post job cleanup.

[notriddle]

@bors try

[bors]

⌛ Trying commit 0cf5ffe with merge ed7f3d7...

[bors]

☀️ Try build successful - checks-actions
Build commit: ed7f3d7 (ed7f3d77595b4831fdf085f2529aa13b5b51ff7d)

[notriddle]

@craterbot run mode=build-and-test

[craterbot]

👌 Experiment pr-134530 created and queued.
🤖 Automatically detected try build ed7f3d7
🔍 You can check out the queue and this experiment's details.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🚧 Experiment pr-134530 is now running

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🚨 Report generation of pr-134530 failed: timed out waiting for connection
🛠️ If the error is fixed use the retry-report command.

🆘 Can someone from the infra team check in on this? @rust-lang/infra
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[notriddle]

@craterbot retry-report

[craterbot]

🛠️ Generation of the report for pr-134530 queued again.

ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[craterbot]

🎉 Experiment pr-134530 is completed!
📊 1953 regressed and 264 fixed (557029 total)
📰 Open the full report.

⚠️ If you notice any spurious failure please add them to the denylist!
ℹ️ Crater is a tool to run experiments across parts of the Rust ecosystem. Learn more

[notriddle]

It looks like 0.64% of all crates known to crater have any items with more than five doctests. That's less than 1%.

$ rg -l 'has more than five doctests' test-fail/ | wc -l
411
$ rg -l '.' test-fail/ | wc -l
63930
$ python3
Python 3.11.2 (main, Sep 14 2024, 03:00:30) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> (411.0 / 63930.0) * 100.0
0.6428906616611919