This repository was archived by the owner on Sep 17, 2023. It is now read-only.
refactor: read monorepo manifest files in parallel #154
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The `lint-dependency-version` subcommand takes a list of dependencies, and for each dependency will test that there is at most one version of that dependency in use throughout the target monorepo. If this condition is violated, the command will exit non-zero exit code. The plan is to use this in CI to enforce monorepo invariants for using only a single version of a given external dependency; for example, `typescript`. Ideally we separate out the library and binary code in this crate, and later we can write multiple binaries that use this library, one of which will invoke this new code and can function as a stand-alone Drone plugin. Thinking about it now, the coupling to Drone is not a goal of this crate but importing the library portion of this codebase from a different crate will be a prerequisite for the plugin.
- use anyhow for error handling
I collected the following benchmarks on my desktop machine. Using no paralleliation ``` $ hyperfine --warmup 10 --min-runs 250 "./target/release/monorepo link --root ~/workspace/my-monorepo" Benchmark 1: ./target/release/monorepo link --root ~/workspace/my-monorepo Time (mean ± σ): 63.1 ms ± 23.6 ms [User: 50.8 ms, System: 55.9 ms] Range (min … max): 30.4 ms … 150.8 ms 250 runs ``` Using a parallel iterator ``` $ hyperfine --warmup 10 --min-runs 250 "./target/release/monorepo link --root ~/workspace/stratos" Benchmark 1: ./target/release/monorepo link --root ~/workspace/stratos Time (mean ± σ): 39.5 ms ± 12.5 ms [User: 26.5 ms, System: 24.2 ms] Range (min … max): 30.0 ms … 90.3 ms 250 runs ``` This looks like a speedup of almost 50%.
ocornoc
approved these changes
May 3, 2022
dpc
reviewed
May 3, 2022
dpc
reviewed
May 3, 2022
dpc
reviewed
May 3, 2022
| @@ -47,34 +51,37 @@ where | |||
| // ignore paths to speed up file-system walk | |||
| package_manifests.push(String::from("!node_modules/")); | |||
|
|
|||
| let monorepo_root = directory.as_ref().to_owned(); | |||
|
|
|||
| GlobWalkerBuilder::from_patterns(&directory, &package_manifests) | |||
| .file_type(FileType::FILE) | |||
| .min_depth(1) | |||
| .build() | |||
| .expect("Unable to create glob") | |||
| .into_iter() | |||
| .filter_map(Result::ok) | |||
There was a problem hiding this comment.
Ignoring errors - not a best strategy. For simplicity, I'd just .map(|r| r.unwrap()) to cause a panic on any error.
There was a problem hiding this comment.
Properly handling it is a bit of a PITA, if you don't want to collect to Result<Vec<_>> as an intermediate step (which you don't here). Though it is possibly. you just have to handle&pass Result everywhere. 🤷
There was a problem hiding this comment.
This is a fair point. I'm going to punt on this for now as it would be a breaking change
|
|
Superseded by #156 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I collected the following benchmarks on my desktop machine.
Using no paralleliation
Using a parallel iterator
This looks like a speedup of almost 50%.