Added initial benchmarks #551

Vrixyz · 2024-07-02T16:02:48Z

Objective

bevy_rapier currently has no benchmarks, making it difficult to be confident with refactoring, or consider performance improvements. This PR aims to implement foundation to address this.

objective 1: take inspiration from bevy_xpbd/avian benchmarks, I think it's useful to aim for comparable benchmarks with other bevy physics crates. That being said, bevy_xpbd has simplistic tests.
objective 2: implement the same benchmarks as rapier, to have a baseline to compare with.
- Using criterion for that is not adapted because the benches are very heavy and tailored for 1 run, to check the update rapier step timings. So I dropped criterion:
  - Divan is used for quick benchmarking, running the same App multiple times, and outputting statistics on update times.
  - A custom bench is used for more granular benchmarks, it's able to compute difference between rapier inner update and total bevy update.

Avian comparison

I doubt the benchmarks are comparable with avian's, as the time management from avian is customized, it might not be running full speed in avian.

bevy_rapier

results as of 6e61318:

Benchmarking cubes 3x3, 30 steps: Warming up for 3.0000 s

Warning: Unable to complete 100 samples in 20.0s. You may wish to increase target time to 22.0s, enable flat sampling, or reduce sample count to 60.
cubes 3x3, 30 steps     time:   [2.8347 ms 2.8435 ms 2.8534 ms]
Found 11 outliers among 100 measurements (11.00%)
  1 (1.00%) low mild
  3 (3.00%) high mild
  7 (7.00%) high severe

cubes 5x5, 30 steps     time:   [8.0240 ms 8.0784 ms 8.1379 ms]
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe

cubes 10x10, 30 steps   time:   [105.89 ms 107.98 ms 110.40 ms]
Found 8 outliers among 100 measurements (8.00%)
  4 (4.00%) high mild
  4 (4.00%) high severe

Avian

results as of Jondolf/avian@8129042

cubes 3x3, 30 steps     time:   [49.782 ms 50.203 ms 50.708 ms]
                        change: [+1638.8% +1656.7% +1676.8%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe

cubes 5x5, 30 steps     time:   [109.87 ms 110.38 ms 111.03 ms]
                        change: [+1254.3% +1266.3% +1278.4%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
  7 (7.00%) high mild
  5 (5.00%) high severe

Benchmarking cubes 10x10, 30 steps: Warming up for 3.0000 s
Warning: Unable to complete 100 samples in 20.0s. You may wish to increase target time to 39.4s, or reduce sample count to 50.
cubes 10x10, 30 steps   time:   [390.36 ms 391.82 ms 393.67 ms]
                        change: [+254.70% +262.86% +270.08%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 9 outliers among 100 measurements (9.00%)
  3 (3.00%) high mild
  6 (6.00%) high severe

Suspicious bad performance

I looked into pipeline.counters.step_time.time(), originally to compare with the total update time, in order to get an idea of the overhead from bevy/bevy_rapier, but I had surprising results:

on rapier testbed, a rapier step takes 116ms on first iteration then goes steadily towards ~4ms.
on bevy_rapier, I had to tune down the pyramids to 3 (opposed to 40 ⚠️), and still got 160ms 😱 for a rapier step.

Flamegraph

flamegraph.zip

^ above numbers reported with f3a3d79

investigating performance is not the objective of this Pull Request so I created another issue for follow ups: Investigate performance overhead #553

…ypyramid

bevy_rapier3d/Cargo.toml

bevy_rapier3d/benches/cubes.rs

bevy_rapier3d/benches/many_pyramids3.rs

sebcrozet · 2024-07-05T09:11:31Z

Both approaches make sense: using divan for smaller benchmarks to quickly detect performance regression; and the custom bench for larger simulations and per-step analysis (ideally outputting some sort of graph).

Is there any way to initialize the scene, and then have divan benchmark each frame of the bevy world update?

benches_common/src/lib.rs

Vrixyz · 2024-07-10T09:38:04Z

Is there any way to initialize the scene, and then have divan benchmark each frame of the bevy world update?

I did implement that with 2cda622

sebcrozet · 2024-07-11T08:41:56Z

benches_common/src/lib.rs

Instead of benches_common be its own create, it should just be a subdirectory in the bevy_rapier3d/benches folder.

I followed the same hierarchy of avian : https://github.com/Jondolf/avian/tree/main/crates ; I'd have preferred a way to enable feature on benches_common to support either 2d or 3d, but I didn't try that.

Either way, it's not too much code, and we can probably focus on 3d for now.

That actually allows me to share some logic with the custom benchmark.

I did add this logic in the custom benchmark, and call into this custom benchmark library from the "simpler" benches.

custom_benches/src/main.rs

sebcrozet · 2024-07-11T08:48:57Z

custom_benches/README.md

The cargo bench section should be removed to the last position in this file. The Custom benches section should be a bit more wordy by saying that it runs benchmarks manually, without bench harnesses, to obtain more detailed results on long-running simulations.

The cargo bench section should say something like "For short-lived benchmarks based on statistical analysis, benchmarks can be run through the divan bench harness.`

side note: I imagine we could use the bench approach (with our own custom bench harness) for better control over custom benchmarks, i.e. run specific custom benches, but I'd need to research that a bit, that can probably be a follow up.

Vrixyz · 2024-07-11T10:03:03Z

custom_benches/src/main.rs

+//! Translated from rapier benchmark.
+//!
+//! <https://github.com/dimforge/rapier/blob/87ada34008f4a1a313ccf8c3040040bab4f10e69/benchmarks3d/many_pyramids3.rs>
+
+use benches_common::default_app;
+use benches_common::wait_app_start;
+use bevy::prelude::*;
+use bevy_rapier3d::dynamics::RigidBody;
+use bevy_rapier3d::geometry::Collider;
+use bevy_rapier3d::math::Vect;
+use bevy_rapier3d::plugin::RapierContext;
+
+pub fn create_pyramid(commands: &mut Commands, offset: Vect, stack_height: usize, rad: f32) {


I'm not a fan of code duplication with bevy_rapier3d/benches ; but I'm not sure how to solve it.

I added shared setup code to the custom benchmark bevy_rapier_benches3d

bevy_rapier_benches3d/Cargo.toml

sebcrozet

Looking good, thanks!

Vrixyz added 6 commits July 2, 2024 18:02

added initial benchmarks, similar to bevy_xpbd

6e61318

fix ci ?

89186b6

use divan + try a custom bench strategy

35c6630

bigger test for pyramid to have comparable performances to rapier man…

f3a3d79

…ypyramid

Merge branch 'master' into benchmarks

08df777

custom benches

c0be019

Vrixyz force-pushed the benchmarks branch from 710e899 to c0be019 Compare July 4, 2024 07:59

Vrixyz commented Jul 4, 2024

View reviewed changes

bevy_rapier3d/Cargo.toml Outdated Show resolved Hide resolved

Vrixyz mentioned this pull request Jul 4, 2024

Investigate performance overhead #553

Open

Vrixyz added 4 commits July 4, 2024 11:06

cleanup duplicate test; add a readme for benches

64e2e1e

simpler cargo.toml for custom_benches

52b3a35

licence for benches_common

82aed44

custom bench version

aecf7ea

Vrixyz mentioned this pull request Jul 4, 2024

Rapier context as a Component #545

Merged

minor fixes

dd19b7e

Vrixyz force-pushed the benchmarks branch from 7b848ca to 4032208 Compare July 4, 2024 12:27

please cargo deny

775ab1f

Vrixyz force-pushed the benchmarks branch from 4032208 to 775ab1f Compare July 4, 2024 12:53

Vrixyz marked this pull request as ready for review July 4, 2024 13:12

Vrixyz requested a review from sebcrozet July 4, 2024 13:18

sebcrozet reviewed Jul 5, 2024

View reviewed changes

bevy_rapier3d/benches/cubes.rs Show resolved Hide resolved

bevy_rapier3d/benches/many_pyramids3.rs Show resolved Hide resolved

Vrixyz commented Jul 5, 2024

View reviewed changes

benches_common/src/lib.rs Outdated Show resolved Hide resolved

Vrixyz added 2 commits July 10, 2024 09:18

added permalinks to benches

4e6b753

divan benchmarks update timings independently

2cda622

Vrixyz added 2 commits July 10, 2024 11:38

Merge branch 'master' into benchmarks

2e7266f

fix cargo doc

74315eb

Vrixyz requested a review from sebcrozet July 10, 2024 14:15

sebcrozet requested changes Jul 11, 2024

View reviewed changes

Vrixyz commented Jul 11, 2024

View reviewed changes

rearrange hierarchy and readme

3bb2b1a

Vrixyz requested a review from sebcrozet July 12, 2024 09:19

Vrixyz commented Jul 12, 2024

View reviewed changes

bevy_rapier_benches3d/Cargo.toml Outdated Show resolved Hide resolved

Vrixyz removed the request for review from sebcrozet July 12, 2024 09:21

fix compilation ; removed some duplicate code

7bf425b

Vrixyz requested a review from sebcrozet July 12, 2024 12:38

sebcrozet approved these changes Jul 18, 2024

View reviewed changes

Vrixyz merged commit 188dcc2 into dimforge:master Jul 18, 2024
5 checks passed

Vrixyz changed the title ~~added initial benchmarks, similar to bevy_xpbd~~ added initial benchmarks Jul 18, 2024

Vrixyz changed the title ~~added initial benchmarks~~ Added initial benchmarks Jul 18, 2024

Vrixyz added a commit to Vrixyz/bevy_rapier that referenced this pull request Aug 8, 2024

added initial benchmarks, similar to bevy_xpbd (dimforge#551)

3f266fe

Vrixyz mentioned this pull request Aug 13, 2024

Fix benchmark #577

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added initial benchmarks #551

Added initial benchmarks #551

Vrixyz commented Jul 2, 2024 •

edited

Loading

sebcrozet commented Jul 5, 2024

Vrixyz commented Jul 10, 2024

sebcrozet Jul 11, 2024

Vrixyz Jul 11, 2024

Vrixyz Jul 12, 2024 •

edited

Loading

sebcrozet Jul 11, 2024

Vrixyz Jul 11, 2024

Vrixyz Jul 11, 2024

Vrixyz Jul 12, 2024

sebcrozet left a comment

Added initial benchmarks #551

Added initial benchmarks #551

Conversation

Vrixyz commented Jul 2, 2024 • edited Loading

Objective

Avian comparison

Suspicious bad performance

sebcrozet commented Jul 5, 2024

Vrixyz commented Jul 10, 2024

sebcrozet Jul 11, 2024

Choose a reason for hiding this comment

Vrixyz Jul 11, 2024

Choose a reason for hiding this comment

Vrixyz Jul 12, 2024 • edited Loading

Choose a reason for hiding this comment

sebcrozet Jul 11, 2024

Choose a reason for hiding this comment

Vrixyz Jul 11, 2024

Choose a reason for hiding this comment

Vrixyz Jul 11, 2024

Choose a reason for hiding this comment

Vrixyz Jul 12, 2024

Choose a reason for hiding this comment

sebcrozet left a comment

Choose a reason for hiding this comment

Vrixyz commented Jul 2, 2024 •

edited

Loading

Vrixyz Jul 12, 2024 •

edited

Loading