Installing docs with `x.py install` is very slow.

[vext01]

CC @pietroalbini

On today's git (0cd459f) x.py install seems to get stuck if you are building docs.

In config.toml:

[install]
prefix = "build/ykrustc-stage2-latest"
sysconfdir = "etc"

x.py install will get stuck at:

    Finished release [optimized] target(s) in 8.38s
Dist rust-docs-1.51.0-dev-x86_64-unknown-linux-gnu
        finished in 17.222 seconds
Install docs stage2 (Some(TargetSelection { triple: "x86_64-unknown-linux-gnu", file: None }))
install: creating uninstall script at /home/vext01/source/rust/build/tmp/empty_dir/build/ykrustc-stage2-latest/lib/rustlib/uninstall.sh
install: installing component 'rust-docs'

The underlying command is: sh /home/vext01/source/rust/build/tmp/tarball/rust-docs/x86_64-unknown-linux-gnu/rust-docs-1.51.0-dev-x86_64-unknown-linux-gnu/install.sh --prefix=/home/vext01/source/rust/build/ykrustc-stage2-latest --sysconfdir=build/ykrustc-stage2-latest/etc --datadir=build/ykrustc-stage2-latest/share --docdir=build/ykrustc-stage2-latest/share/doc/rust --bindir=build/ykrustc-stage2-latest/bin --libdir=build/ykrustc-stage2-latest/lib --mandir=build/ykrustc-stage2-latest/share/man --disable-ldconfig

Adding this to the config seems to work around the hang:

[build]
docs = false

Thanks!

[pietroalbini]

How long did you let the command run before killing it? I remember installing the docs taking a long time last time I did it.

[vext01]

I left it for half an hour or so IIRC. I'll run it for longer and see if it goes away.

How long do you think it should take to install on a modern SSD?

[pietroalbini]

I don't remember exactly how long it takes, but it doesn't depend on the speed of the disk. install.sh is just really inefficient. You can see if it's actually hanging by checking the contents of lib/rustlib/install.log this command while it's installing:

tail -f build/ykrustc-stage2-latest/lib/rustlib/install.log

[vext01]

The log file is empty, but I can see new files appearing on the disk (using find and diff) so progress must be occurring. Slowly.

[vext01]

It's been about 40 minutes now. It seems to be adding files to build/tmp/empty_dir/ is that right?

[vext01]

Build completed successfully in 0:57:44

So it's just very slow.

[jyn514]

I took a look into this. The main issue is that the generated install.sh is very inefficient; it takes about 5 minutes to copy 2500 files (there are more than 2500, but I got bored and cancelled it). @pietroalbini @Mark-Simulacrum what do you think about rewriting this inner loop in Rust so it's not launching so many external processes?

https://github.com/rust-lang/rust-installer/blob/5254dbfd25d5284728ab624dca1969d61427a0db/install-template.sh#L548

[daira]

This issue is compounded on macOS, where it seems to be even more inefficient to launch lots of processes from a shell. Sorry I don't have timings, but it takes tens of minutes at least for this step [edit: over an hour]; long enough that I would expect most users who are trying to build our software from source in macOS (it's Zcash, which installs a pinned version of Rust in order to build) are likely to think it's hanging and give up.

[Mark-Simulacrum]

I think the problem may largely be that the manifest.in generated for rust-docs component (at least) is a file: list instead of using the directory: command; directory commands should be much more efficient. This change may be a recent(ish) regression; I suspect that fixing that is likely the right solution here.