Help users discover overly broad rerun-if-changed directives

[bradzacher]

Maintainers notes

• Root problem was an overly broad rerun-if-changed, see Help users discover overly broad rerun-if-changed directives #14240 (comment)

---

Problem

At work we have a rather large monorepo - of which rust (currently) makes up a very small part.
Some quick numbers using cloc --vcs git -- ~305k files (~45m LOC) of which ~400 files (~80k LOC) are rust.

We've noticed that the time for cargo check --workspace --manifest-path ./Cargo.toml is quite slow - disproportionately slow compared to the sum of running check on each.

Here are the examples from running the commands locally

$ time cargo check -p crate1
$ time cargo check -p crate2
...
Sum: 32.377 total

$ time cargo check --workspace --manifest-path ./Cargo.toml
4.26s user 43.64s system 59% cpu 1:21.14 total

$ time cargo check --manifest-path ./Cargo.toml -p crate1 -p crate2 ...
3.89s user 43.02s system 58% cpu 1:20.16 total

Doing some investigation - in the last two cases it looks like cargo is scanning the entire repo. When inspecting what files that cargo is reading I can see it accessing files and folders not included in those those defined within the workspace's members array that contain no rust code.

Running sudo fs_usage | grep cargo I can see it accessing:

• our "frontend" folder and scanning through all of its node_modules, and
• bazel's output folders (which are symlinked by bazel into the repo root) which are all build artifacts.

These two cases were actually what first triggered us to investigate we had reports from several engineers about slowness - we noticed two things:

1. the more bazel builds they do, the slower cargo got - and it looks like it's because more builds = more files in the bazel output folders = more crawl time.
2. the more npm packages that were installed, the slower cargo got - similarly more installs = more node_modules = more crawl time.

This is not great for us because it means that certain operations can take 3 times longer than they should - and it impacts IDE startup times too as rust-analyzer runs cargo check --workspace --message-format=json-diagnostic-rendered-ansi --manifest-path /absolute/path/to/Cargo.toml as part of its startup.

Steps

No response

Possible Solution(s)

I'd love a way to tell cargo "please don't scan anything - all you should care about is located right here in the members list" so that we can avoid scanning irrelevant folders.

Alternately - making cargo respect .gitignore during its scan would also be good as it would mean that folders like node_modules and the bazel folders would be skipped entirely.

Notes

All of our dependencies are sourced from crates.io
We do not use any git links or patches
We do have workspace interdependencies

Please let me know if there are any log files, etc that you would like to help investigate - happy to provide whatever info I can to you.

Version

cargo 1.78.0 (54d8815d0 2024-03-26)
release: 1.78.0
commit-hash: 54d8815d04fa3816edc207bbc4dd36bf18014dbc
commit-date: 2024-03-26
host: aarch64-apple-darwin
libgit2: 1.7.2 (sys:0.18.2 vendored)
libcurl: 8.6.0 (sys:0.4.72+curl-8.6.0 system ssl:(SecureTransport) LibreSSL/3.3.6)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Mac OS 14.5.0 [64-bit]

[epage]

Huh, not too sure what this would be. cargo check--manifest-path Cargo.toml vs cargo check shouldn't be too different. When looking for Cargo.toml, we only walk up the directories, not back down again.

Could you give us an idea of

• Your general directory hierarchy (where node_modules, workspace Cargo.toml, package Cargo.toml, bazel build directory, etc live relative to each other)
• What your workspace.members looks like

Even better if you could also run cargo check && CARGO_LOG_PROFILE=true cargo check and open the traces in chrome://tracing and let us know what the hot section is (docs). If possible, this would be best done with 1.79 as we tweak the traces for better information with each release.

[bradzacher]

Your general directory hierarchy

it's all over the place, really - the repo is a mess that's grown organically over time.
here's a rough idea of the mess

- /
  - Cargo.toml (workspace)
  - (bazel folders symlinked in the repo root)
    - bazel_out/
    - bazel_bin/
    - bazel_canva/
  - web/ (the frontend codebase
    - node_modules/
    - crate1/ [[rust]] (some WASM rust)
      - Cargo.toml
  - tools/ (a lot of internal tooling)
    - crate2/ [[rust]]
      - Cargo.toml
    - crate3/ [[rust]]
      - Cargo.toml
    - ...
    - js1/
      - node_modules/
    - js2/
      - node_modules/
    - ...
  - backend_service1/ (a backend microservice)
  - backend_service_rust1/ [[rust]] (a backend microservice, written in rust)
    - Cargo.toml

What your workspace.members looks like

it's just a list of folders (i.e. example based on the above)

members = [
  "web/crate1",
  "tools/crate2",
  "tools/crate3",
  "backend_service_rust1",
  ...
  "path/to/crate12", # there are only 12 crates in our monorepo thus far
]

[bradzacher]

Trace (with 1.78):
trace-1720750556589970.json

I'm not sure how to interpret the trace so here it is in its raw form.
The bulk of the time is spent in "Name prepare_target, Category cargo::core::compiler::fingerprint"

[bradzacher]

Trace (with 1.79):
trace-1720751045123202.json

Looks to be much the same

[Xuanwo]

How about making cargo respect .gitignore, or alternatively, introducing .cargoignore when managing workspaces?

[epage]

@bradzacher Thanks for that context! It looks like only one specific prepare_target call is taking up all of that time. If you re-run with cargo check && CARGO_LOG_PROFILE=true CARGO_LOG_PROFILE_CAPTURE_ARGS=true cargo check then the traces will include the Package ID of the package that is causing this. From that we can try to understand what is different about this package that it is hitting this case in cargo. In particular, if there are is a build script and what directives it emits, symlinks inside its directory, an odd location on disk, etc.

@Xuanwo I don't even know what file walk this is to evaluate an answer like that. One theory is a build.rs that gives a bad path for for re-run-if-changed or we have some bug with how we are walking the directory structure for one of the rebuild-detection checks we do.

[weihanglo]

How about making cargo respect .gitignore, or alternatively, introducing .cargoignore when managing workspaces?

@Xuanwo it already does. The presence of exclude and include fields would affect whether .gitignore is respected.

[weihanglo]

Some more questions:

• Is there any Cargo.toml containing odd package.include that refer to relative parent directory like ..?
• Is there any symlink under a member package pointing to directories outside the package root, for example to the workspace root?
• Any builds script rerun-if-changed has relative paths?

[bradzacher]

Is there any Cargo.toml containing odd package.include that refer to relative parent directory like ..?

No. There is no package.include defined in any Cargo.toml

Is there any symlink under a member package pointing to directories outside the package root, for example to the workspace root?

Two crates contain a node_modules which have a symlinks (as pnpm does symlink installs)
The symlink is from the node_modules back to the root node_modules.
I.e. from tools/crate2/node_modules/typescript to ../../../node_modules/.pnpm/typescript@5.4.5/node_modules/typescript (note ../../.. resolves to the repo root)

Other than that - no links at all

Any builds script rerun-if-changed has relative paths?

There are two:
one outputs cargo:rerun-if-changed=tools/crate2/folder
the other outputs this (where ../../ resolves to the repo root)

cargo:rerun-if-changed=../../tools/crate3/some/file.foo
cargo:rerun-if-changed=../../

[bradzacher]

If you re-run with cargo check && CARGO_LOG_PROFILE=true CARGO_LOG_PROFILE_CAPTURE_ARGS=true cargo check then the traces will include the Package ID of the package that is causing this

I reran it and the profile says that tools/crate3 (the one with the relative rerun-if-changed path) is the one that's responsible for that massive block.

[bradzacher]

And it only just clicked that OFC the fact that it's dumping cargo:rerun-if-changed=../../ is what's causing cargo to watch the entire repo and crawl everything.