cargo fmt --all is slow in a workspace project

[mzabaluev]

Problem
cargo fmt --all runs unexpectedly slow in a medium-sized workspace project.

Steps

1. Run a GitHub workflow that runs cargo fmt --all on the repository.
2. Check execution time.

This run shows rustfmt and/or cargo pausing for more than 20 s before producing any output in verbose mode. Seen with timestamps in the raw log:

2020-06-08T17:05:32.3268959Z [command]/usr/share/rust/.cargo/bin/cargo fmt --all --verbose -- --check --verbose
2020-06-08T17:05:55.8733086Z [bench (2018)] "/home/runner/work/chain-libs/chain-libs/btree/benches/benchmark.rs"

Notes

Output of cargo version:
cargo 1.44.0 (05d080faa 2020-05-06)

Output of rustfmt --version
rustfmt 1.4.14-stable (e417356 2020-04-21)

Virtual environment information from GitHub workflow:
Operating System
Ubuntu
18.04.4
LTS
Virtual Environment
Environment: ubuntu-18.04
Version: 20200518.1
Included Software: https://github.com/actions/virtual-environments/blob/ubuntu18/20200518.1/images/linux/Ubuntu1804-README.md

[mzabaluev]

Originally filed as rust-lang/cargo#8345.

[mzabaluev]

Closing as the problem was tracked down to a cargo issue after all.

[topecongiro]

This is because rustfmt uses cargo-metadata to find the root file of each project.

By default, rustfmt uses the offline mode of cargo-metadata and falls back to the online mode when it fails. I assume the slowdown you observed is caused by the connection delay; when running cargo fmt --all, we need the cargo index file to be present.

[mzabaluev]

As noted by @ehuss in rust-lang/cargo#8345 (comment):

Sorry, I should have specified in my comment. The issue is here where cargo-fmt calls cargo metadata for every package, twice. It should only call it once for the entire workspace. I was going to make some recommendations over on the rustfmt tracker.

[calebcartwright]

@mzabaluev - does your project have non-workspace member local/path-based dependencies that you also want to format? Running cargo fmt (without the --all) in the root of a workspace will format all the crates in a workspace by default. The --all flag isn't necessary in most situations.

From the cargo fmt help text:

--all                          Format all packages, and also their local path-based dependencies

FYI the multiple cargo metadata calls rustfmt is making is due to rust-lang/cargo#7483 😄

[mzabaluev]

@calebcartwright Thanks for the tip! One of our larger projects does use path-based dependencies pulled in by a git submodule, but these can be format-checked in a CI workflow of that submodule's repository.

[calebcartwright]

Great! In that case I'd recommend just running cargo fmt and dropping the --all flag so you won't have to worry about the registry index at all

[ehuss]

@calebcartwright I can help rework the logic for the --all case if you'd like. I don't see that it is necessary to re-fetch the metadata for each package, so it should be able to do it just once. I'm not too familiar with rustfmt development, but I can try to make a PR.