This repository has been archived by the owner on Nov 15, 2023. It is now read-only.
Reset benchmarks by removing post-genesis changes #5435
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I tested this PR by generating a large chain spec (~85 mb, 100k accounts). I then ran the same benchmark before and after this commit: Before: After: So the PR is a huge improvement. From 50 min to 3 min to run the same benchmark. However, The results from the benchmark has changed significantly. Before: Median Slopes: 568 microseconds Raw DataAfter: Raw DataMedian: 1925 microseconds |
|
@shawntabrizi Does the time measurement include calls to |
|
@arkpar it does not. Only the extrinsic itself. Let me test end to end again. |
shawntabrizi
reviewed
Apr 1, 2020
There was a problem hiding this comment.
This PR makes it significantly faster to run benchmarks with a large chain specification.
Tested again with a 19 mb chain spec containing 100K account keys:
Before:
time ./target/release/substrate benchmark --chain ./substrate-dev-a-bench-revert-final.json --execution=wasm --wasm-execution=compiled --pallet balances --extrinsic "*" --repeat 20 --steps 20 --raw > balances.txt
./target/release/substrate benchmark --chain --execution=wasm --pallet "* 1578.32s user 184.09s system 100% cpu 29:22.07 total
After:
time ./target/release/substrate benchmark --chain ./substrate-dev-a-bench-revert-final.json --execution=wasm --wasm-execution=compiled --pallet balances --extrinsic "*" --repeat 20 --steps 20 --raw > balances.txt
./target/release/substrate benchmark --chain --execution=wasm --pallet "* 148.41s user 77.64s system 149% cpu 2:30.88 total
So I am seeing a 10x improvement here.
However, i will still note that the results of the benchmarks have slowed a bit, but talking with @arkpar it makes sense that if deleting content from the DB causes slowdown, then this is an even more accurate model of what a real DB might be doing.
So this is really great then! Lets merge asap
shawntabrizi
approved these changes
Apr 1, 2020
shawntabrizi
added
A8-mergeoncegreen
and removed
A0-please_review
NikVolf
reviewed
Apr 1, 2020
NikVolf
approved these changes
Apr 1, 2020
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.