Add more profiling traces #14238
Merged
Add more profiling traces #14238
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Someone suggested I look at [zed](https://github.com/zed-industries/zed). It has some interesting performance characteristics compared to projects I looked at before: - A lot of workspace members - git dependencies - patches This adds traces that help provide more context for those scenarios when looking at them. Some of these areas might be of interest for optimizing: - On every invocation, we do a git2 `copy_to` call for each git checkout - On every invocation, we do a `du` on each git checkout - It'd be great to avoid parsing every manifest in a git checkout but we need to do a full parse to make sure we find all packages (see https://rust-lang.zulipchat.com/#narrow/stream/246057-t-cargo/topic/Redundant.20code.20in.20.60GitSouce.60.3F) - Its suspicious how much time we spend in the "poison" step of resolving when its a no-op
rustbot
added
A-git
arlosi
reviewed
Jul 11, 2024
|
Thanks @bors r+ |
bors
added
S-waiting-on-bors
|
☀️ Test successful - checks-actions |
1 similar comment
|
☀️ Test successful - checks-actions |
|
👀 Test was successful, but fast-forwarding failed: 422 Changes must be made through a pull request. |
epage
added a commit
to epage/cargo
that referenced
this pull request
Jul 15, 2024
When profiling Zed (rust-lang#14238), a major factor in their no-op run times is git patches and git dependencies. The slowest operation for each git source is running `du`. This is extraneous for a couple of reasons - GC isn't stable, slowing people down for a feature they aren't using - Size tracking was expected to be lazy, only reading sizes when the GC is configured for size, while this was eager - Git checkouts are immutable but we check on every load - This optimized for "while filesystem caches are warm" from a checkout operation when checkout operations are rare compared to all of the other commands run on a working directory. This removes the `du`, relying on the lazy loading that happens in `update_null_sizes`.
bors
added a commit
that referenced
this pull request
Jul 15, 2024
perf(source): Don't `du` on every git source load ### What does this PR try to resolve? When profiling Zed (#14238), a major factor in their no-op run times is git patches and git dependencies. The slowest operation for each git source is running `du`. This is extraneous for a couple of reasons - GC isn't stable, slowing people down for a feature they aren't using - Size tracking was expected to be lazy, only reading sizes when the GC is configured for size, while this was eager - Git checkouts are immutable but we check on every load - This optimized for "while filesystem caches are warm" from a checkout operation when checkout operations are rare compared to all of the other commands run on a working directory. This removes the `du`, relying on the lazy loading that happens in `update_null_sizes`. For Zed, this removed about 40ms total from the runtime. While by itself, this is below the threshold of being noticed, - It adds up if any editor integrations are calling `cargo metadata` a lot - Over time, small gains like this will add up ### How should we test and review this PR? ### Additional information cc `@ehuss`
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Jul 17, 2024
Update cargo 31 commits in 154fdac39ae9629954e19e9986fd2cf2cdd8d964..a2b58c3dad4d554ba01ed6c45c41ff85390560f2 2024-07-07 01:28:23 +0000 to 2024-07-16 00:52:02 +0000 - chore(ci): bump CI tools (rust-lang/cargo#14257) - test: migrate fetch and list_availables to snapbox (rust-lang/cargo#14214) - chore: downgrade to jobserver@0.1.28 (rust-lang/cargo#14254) - perf(source): Don't `du` on every git source load (rust-lang/cargo#14252) - fix(source): Don't warn about unreferenced duplicate packages (rust-lang/cargo#14239) - feat(test): Add cargo_test to test-support prelude (rust-lang/cargo#14243) - Add workflow to publish Cargo automatically (rust-lang/cargo#14202) - test: migrate implicit_features to snapbox (rust-lang/cargo#14245) - test: migrate build-std/main to snapbox (rust-lang/cargo#14241) - test: migrate check_cfg to snapbox (rust-lang/cargo#14235) - refactor(source): More RecursivePathSource clean up (rust-lang/cargo#14231) - Add more profiling traces (rust-lang/cargo#14238) - fix(overrides): Don't warn on duplicate packages from using '..' (rust-lang/cargo#14234) - fix(test): Redact elapsed time in the minutes time frame (rust-lang/cargo#14233) - test: Migrate lto tests to snapbox (rust-lang/cargo#14209) - fix: Ensure dep/feature activates the dependency on 2024 (rust-lang/cargo#14221) - chore(docs): update index of reference (rust-lang/cargo#14228) - test: migrate test to snapbox (rust-lang/cargo#14226) - chore: remove duplicate words (rust-lang/cargo#14229) - docs(contrib): Document things I look for in RFCs (rust-lang/cargo#14222) - docs(ref): Note MSRV for features in the docs (rust-lang/cargo#14224) - test(progress): Resolve flakiness (rust-lang/cargo#14223) - fix(test): Reduce over-prescription to the caller (rust-lang/cargo#14217) - refactor: move get_source_id out of registry (rust-lang/cargo#14218) - fix: rename to `rustdoc::broken_intra_doc_links` (rust-lang/cargo#14215) - test: migrate member_errors, multitarget and new to snapbox (rust-lang/cargo#14210) - test: migrate generate_lockfile and glob_targets to snapbox (rust-lang/cargo#14200) - test: Ensure --list test works with cargo-bloat (rust-lang/cargo#14213) - dont make new constant InternedString in hot path (rust-lang/cargo#14211) - Fix compatible_with_older_cargo test. (rust-lang/cargo#14212) - test: migrate metabuild, metadata and net_config to snapbox (rust-lang/cargo#14162)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR try to resolve?
Someone suggested I look at
zed.
It has some interesting performance characteristics compared to projects I looked at before:
This adds traces that help provide more context for those scenarios when looking at them.
How should we test and review this PR?
Additional information
Some of these areas might be of interest for
optimizing:
copy_tocall for each git checkoutduon each git checkout