Skip to content

chore(NODE-6921): perf tests default to cwd and errexit #791

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Apr 28, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions .evergreen/run-granular-benchmarks.sh
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
#!/usr/bin/env bash

source $DRIVERS_TOOLS/.evergreen/init-node-and-npm-env.sh
set -o xtrace
WARMUP=$WARMUP
ITERATIONS=$ITERATIONS

WARMUP=$WARMUP ITERATIONS=$ITERATIONS LIBRARY=$(pwd) npm run check:granular-bench
set -euxo pipefail

export WARMUP=$WARMUP
export ITERATIONS=$ITERATIONS

npm run check:granular-bench
2 changes: 2 additions & 0 deletions .evergreen/run-spec-benchmarks.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,6 @@

source $DRIVERS_TOOLS/.evergreen/init-node-and-npm-env.sh

set -euxo pipefail

npm run check:spec-bench
22 changes: 12 additions & 10 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,15 +83,16 @@ WARMUP=<warmup iterations> ITERATIONS=<measured iterations> npm run check:granul
This will build the granular tests and run them with `test/bench/etc/run_granular_benchmarks.js`. The `WARMUP` and `ITERATIONS` environment variables can be optionally provided to configure how these granular benchmarks
are run. `WARMUP` changes the number of iterations run before results are collected to give v8's
optimizing compiler time to reach steady state. `ITERATIONS` changes the number of iterations that
are measured and used to calculate summary statistics. Note also that the test can be configured to
make use of the local copy of bson when testing performance changes locally by setting the `LIBRARY`
variable to the root directory of the bson library to be tested.
are measured and used to calculate summary statistics.

```bash
WARMUP=100 ITERATIONS=1000 LIBRARY=$(pwd) npm run check:granular-bench
WARMUP=100 ITERATIONS=1000 npm run check:granular-bench
```
When the `LIBRARY` environment variable is unset, the benchmark clones and runs against the main
branch of this repository.

When the `LIBRARY` environment variable is unset, the benchmark clones and runs against the current working directory.

> [!NOTE]
> The test can be configured to make use of a different copy of bson when testing performance changes locally by setting the `LIBRARY` variable to the root directory of a copy of the bson library.

When the script is complete, results will be output to `test/bench/etc/resultsCollectedMeans.json`. These results will
be in a format compatible with evergreen's perf.send command. To convert these results to CSV, run
Expand All @@ -109,14 +110,15 @@ npm run check:spec-bench

This will run the spec benchmarks in `test/bench/spec/bsonBench.ts` which also makes use of the
`bson-bench` library. Results will be written to `bsonBench`. The warmup and iterations are not
configurable as these are determined by the common driver benchmarking specification, but similar
to the granular benchmarks, the spec benchmarks can be run against the local copy of bson by setting
the `LIBRARY` environment variable appropriately.
configurable as these are determined by the common driver benchmarking specification.

```bash
LIBRARY=$(pwd) npm run check:spec-bench
npm run check:spec-bench
```

> [!NOTE]
> The test can be configured to make use of a different copy of bson when testing performance changes locally by setting the `LIBRARY` variable to the root directory of a copy of the bson library.

### Commit messages

Please follow the [Conventional Commits specification][conventional-commit-style].
Expand Down
2 changes: 1 addition & 1 deletion test/bench/etc/run_granular_benchmarks.js
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ const DOCUMENT_ROOT = path.resolve(`${__dirname}/../documents`);
}
}

console.log('No duplcate testName:Option pairs found. Now merging files...');
console.log('No duplicate testName:Option pairs found. Now merging files...');

const meansFile = `${__dirname}/resultsCollectedMeans.json`;
// Iterate over all result files and merge into one file
Expand Down
3 changes: 2 additions & 1 deletion test/bench/granular/common.ts
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ export function getMixedTestTags(documentPath: string) {
export async function runSuiteAndWriteResults(suite: Suite) {
const targetDirectory = path.resolve(`${__dirname}/../../etc`);
await suite.run();
if (suite.errors.length) throw suite.errors[0].error;
await suite.writeResults(`${targetDirectory}/${suite.name.toLowerCase()}Results.json`);
}

Expand All @@ -71,7 +72,7 @@ export function readEnvVars(): { warmup: number; iterations: number; library: st
const rv = {
warmup: Number.isSafeInteger(envWarmup) && envWarmup > 0 ? envWarmup : 100_000,
iterations: Number.isSafeInteger(envIterations) && envIterations > 0 ? envIterations : 10_000,
library: libraryPath ? `bson:${libraryPath}` : 'bson#main'
library: libraryPath ? `bson:${libraryPath}` : `bson:${process.cwd()}`
};

console.log(
Expand Down
2 changes: 2 additions & 0 deletions test/bench/spec/bsonBench.ts
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,8 @@ suite.task({
});

suite.run().then(async () => {
if (suite.errors.length) throw suite.errors[0].error;

const cpuBaseline = await readFile(
join(__dirname, '..', '..', 'etc', 'cpuBaseline.json'),
'utf8'
Expand Down