Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fresh benchmarks #13

Conversation

bkontur
Copy link

@bkontur bkontur commented Jul 1, 2024

subweight compare commits \
>          --path-pattern "./system-parachains/bridge-hubs/**/weights/**/*.rs" \
>          --format markdown --no-color  \
>          --change added changed removed  \
>         --method asymptotic --ignore-errors  \
>         --strip-path-prefix="system-parachains/bridge-hubs/" \
>         --threshold 0.5 \
>          remotes/Snowfork/ethereum-headers-on-demand \
>          origin/bko-snowbridge-benchmarks
File Extrinsic Old New Change [%]
bridge-hub-polkadot/src/weights/snowbridge_pallet_ethereum_client.rs submit 25.40ms 34.32ms +35.11
bridge-hub-polkadot/src/weights/snowbridge_pallet_ethereum_client.rs submit_with_sync_committee 127.21ms 167.58ms +31.73
bridge-hub-polkadot/src/weights/snowbridge_pallet_ethereum_client.rs force_checkpoint 102.44ms 133.80ms +30.61
bridge-hub-kusama/src/weights/snowbridge_pallet_ethereum_client.rs submit 26.22ms 33.01ms +25.89
bridge-hub-kusama/src/weights/snowbridge_pallet_ethereum_client.rs submit_with_sync_committee 131.31ms 162.24ms +23.56
bridge-hub-kusama/src/weights/snowbridge_pallet_ethereum_client.rs force_checkpoint 105.80ms 129.70ms +22.58
bridge-hub-kusama/src/weights/snowbridge_pallet_inbound_queue.rs submit 481.28us 584.43us +21.43
bridge-hub-polkadot/src/weights/snowbridge_pallet_inbound_queue.rs submit 482.86us 585.13us +21.18

Copy link

github-actions bot commented Jul 1, 2024

Review required! Latest push from author must always be reviewed

@claravanstaden
Copy link
Collaborator

claravanstaden commented Jul 2, 2024

Thanks for the fresh benchmarks @bkontur! I am concerned about the increase in weight, especially for all the ethereum client extrinsics. They should start more or less the same.

The diff shows that there were mostly minor removals in bridges/snowbridge/parachain/pallets/ethereum-client/src/lib.rs. Have you seen weight increased like this before, for code that didn't really change?

@acatangiu
Copy link

@claravanstaden can you run short benchmark for just one of them with old version and new, on your local machine and see if you are getting any change between them?

if so, someone on your team can also "debug" the benchmarking code for said extrinsics and see if it is actual real-world cost change or benchmarking issue

@bkontur
Copy link
Author

bkontur commented Jul 3, 2024

Thanks for the fresh benchmarks @bkontur! I am concerned about the increase in weight, especially for all the ethereum client extrinsics. They should start more or less the same.

The diff shows that there were mostly minor removals in bridges/snowbridge/parachain/pallets/ethereum-client/src/lib.rs. Have you seen weight increased like this before, for code that didn't really change?

I pushed another commit from different machines, results are a little bit better, but still.

bparity@bkontur-ThinkPad-P14s-Gen-2i:~/parity/polkadot-fellows/runtimes$ subweight compare commits \
>           --path-pattern "./system-parachains/bridge-hubs/**/weights/**/*.rs" \
>           --format markdown --no-color  \
>           --change added changed removed  \
>          --method asymptotic --ignore-errors  \
>          --strip-path-prefix="system-parachains/bridge-hubs/" \
>          --threshold 0.5 \
>           remotes/Snowfork/ethereum-headers-on-demand \
>           origin/bko-snowbridge-benchmarks
File Extrinsic Old New Change [%]
bridge-hub-polkadot/src/weights/snowbridge_pallet_ethereum_client.rs submit 25.40ms 31.51ms +24.03
bridge-hub-kusama/src/weights/snowbridge_pallet_ethereum_client.rs submit 26.22ms 31.82ms +21.36
bridge-hub-polkadot/src/weights/snowbridge_pallet_ethereum_client.rs submit_with_sync_committee 127.21ms 152.93ms +20.22
bridge-hub-kusama/src/weights/snowbridge_pallet_inbound_queue.rs submit 481.28us 577.03us +19.90
bridge-hub-polkadot/src/weights/snowbridge_pallet_inbound_queue.rs submit 482.86us 578.29us +19.76
bridge-hub-polkadot/src/weights/snowbridge_pallet_ethereum_client.rs force_checkpoint 102.44ms 122.11ms +19.20
bridge-hub-kusama/src/weights/snowbridge_pallet_ethereum_client.rs submit_with_sync_committee 131.31ms 154.03ms +17.31
bridge-hub-kusama/src/weights/snowbridge_pallet_ethereum_client.rs force_checkpoint 105.80ms 123.00ms +16.25

I see that your benchmarks were regenerated the last time here:

-//! DATE: 2024-03-25, STEPS: `50`, REPEAT: `20`, LOW RANGE: `[]`, HIGH RANGE: `[]`
+//! DATE: 2024-07-01, STEPS: `50`, REPEAT: `20`, LOW RANGE: `[]`, HIGH RANGE: `[]`
 //! WORST CASE MAP SIZE: `1000000`
-//! HOSTNAME: `ggwpez-ref-hw`, CPU: `Intel(R) Xeon(R) CPU @ 2.60GHz`
+//! HOSTNAME: `bkontur-ref-hw`, CPU: `AMD EPYC 7232P 8-Core Processor`

Previously we used some google cloud VM instance, but we don't have them anymore. We have new baremetal servers.

@claravanstaden
Copy link
Collaborator

@bkontur @acatangiu thanks for running the benchmarks again! It looks like the diff might be from moving the hash tree root of the finalized header: https://github.com/paritytech/polkadot-sdk/compare/e046d3b2caaca6c43a60f26594052c5c806b980e..6dba07c0bc0c14c2caa33dc8fc385c849175f6d5#diff-98c747bd18857fe76701570ede9bc3bef5350ed0f9808daea2af1e769c1a8effR502

I'm going to merge the PR.

@claravanstaden claravanstaden merged commit b818954 into Snowfork:ethereum-headers-on-demand Jul 3, 2024
39 of 41 checks passed
@bkontur bkontur deleted the bko-snowbridge-benchmarks branch July 3, 2024 10:33
@yrong
Copy link

yrong commented Jul 3, 2024

can you run short benchmark for just one of them with old version and new, on your local machine and see if you are getting any change between them?

Run against old version and the new on same machine, seems no difference.

File Extrinsic Old New Change [%]
bridge-hub-polkadot/src/weights/snowbridge_pallet_ethereum_client.rs submit 35.886ms 35.813ms -0.2
bridge-hub-polkadot/src/weights/snowbridge_pallet_ethereum_client.rs submit_with_sync_committee 181.850ms 181.573ms -0.1

@claravanstaden
Copy link
Collaborator

@yrong you're right, I saw the same when I tested locally. So might be the machine difference. Thanks for checking. 🙏🏻

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants