Skip to content

Commit

Permalink
test: State benchmarks (FuelLabs#1226)
Browse files Browse the repository at this point in the history
Related issues:
- Ref FuelLabs#1134

These benches demonstrate the performance of:

- Inserting 1 key/value pair into the RocksDB table that contains `n`
state entries for a single contracts. `n = [0, 1, 10, 100, 1,000,
10,000, 100,000, 1,000,000]`
- Inserting 1 key/value pair into the RocksDB table that contains 1
state entries with `n` existing contracts. `n = [0, 1, 10, 100, 1,000,
10,000, 100,000, 1,000,000]`

Bench results:
```
state single contract/insert state with 0 preexisting entries
                        time:   [11.991 µs 12.132 µs 12.310 µs]
                        change: [-4.7172% -1.6523% +1.0884%] (p = 0.29 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe
state single contract/insert state with 1 preexisting entries
                        time:   [25.009 µs 28.694 µs 32.962 µs]
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe
state single contract/insert state with 10 preexisting entries
                        time:   [39.002 µs 41.617 µs 44.324 µs]
Found 1 outliers among 100 measurements (1.00%)
  1 (1.00%) high severe
state single contract/insert state with 100 preexisting entries
                        time:   [60.968 µs 63.985 µs 67.259 µs]
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
state single contract/insert state with 1,000 preexisting entries
                        time:   [87.465 µs 91.207 µs 95.394 µs]
                        change: [+0.4239% +4.5684% +8.9542%] (p = 0.04 < 0.05)
                        Change within noise threshold.
Found 7 outliers among 100 measurements (7.00%)
  6 (6.00%) high mild
  1 (1.00%) high severe
state single contract/insert state with 10,000 preexisting entries
                        time:   [117.22 µs 121.38 µs 125.93 µs]
Found 6 outliers among 100 measurements (6.00%)
  6 (6.00%) high mild
state single contract/insert state with 100,000 preexisting entries
                        time:   [145.05 µs 148.30 µs 151.60 µs]
                        change: [-3.4471% -0.8145% +1.9661%] (p = 0.56 > 0.05)
                        No change in performance detected.
state single contract/insert state with 1,000,000 preexisting entries
                        time:   [186.36 µs 190.05 µs 193.80 µs]
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high mild

state multiple contracts/insert state with 0 preexisting entries
                        time:   [11.957 µs 12.080 µs 12.237 µs]
                        change: [-19.431% -7.6642% +0.8636%] (p = 0.27 > 0.05)
                        No change in performance detected.
Found 9 outliers among 100 measurements (9.00%)
  5 (5.00%) high mild
  4 (4.00%) high severe
state multiple contracts/insert state with 1 preexisting entries
                        time:   [11.872 µs 12.008 µs 12.199 µs]
Found 5 outliers among 100 measurements (5.00%)
  2 (2.00%) high mild
  3 (3.00%) high severe
state multiple contracts/insert state with 10 preexisting entries
                        time:   [12.189 µs 12.252 µs 12.320 µs]
Found 4 outliers among 100 measurements (4.00%)
  1 (1.00%) high mild
  3 (3.00%) high severe
state multiple contracts/insert state with 100 preexisting entries
                        time:   [12.538 µs 12.691 µs 12.867 µs]
Found 8 outliers among 100 measurements (8.00%)
  5 (5.00%) high mild
  3 (3.00%) high severe
state multiple contracts/insert state with 1,000 preexisting entries
                        time:   [13.041 µs 13.278 µs 13.640 µs]
                        change: [-1.9317% +1.1895% +4.0288%] (p = 0.46 > 0.05)
                        No change in performance detected.
Found 2 outliers among 100 measurements (2.00%)
  2 (2.00%) high severe
state multiple contracts/insert state with 10,000 preexisting entries
                        time:   [13.350 µs 13.529 µs 13.729 µs]
Found 3 outliers among 100 measurements (3.00%)
  2 (2.00%) high mild
  1 (1.00%) high severe
state multiple contracts/insert state with 100,000 preexisting entries
                        time:   [13.973 µs 14.312 µs 14.719 µs]
                        change: [+4.3928% +9.0688% +14.995%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 6 outliers among 100 measurements (6.00%)
  2 (2.00%) high mild
  4 (4.00%) high severe
state multiple contracts/insert state with 1,000,000 preexisting entries
                        time:   [13.891 µs 14.096 µs 14.314 µs]
Found 4 outliers among 100 measurements (4.00%)
  3 (3.00%) high mild
  1 (1.00%) high severe
```

**Conclusion:**

For inserting `n` state entries for the same contract, the performance
appears to be logarithmic with `n`.

For inserting 1 state entry for a contract, with `n` existing contracts,
the performance appears to be fairly constant, i.e., the number of
existing contracts does not have an impact on the performance of
inserting state for a contract.

Note that these benchmarks were taken after the change to use RocksDB in
benches (FuelLabs#1227).
  • Loading branch information
Brandon Vrooman authored Jun 29, 2023
1 parent 56cc33d commit 633de7c
Show file tree
Hide file tree
Showing 2 changed files with 173 additions and 0 deletions.
4 changes: 4 additions & 0 deletions benches/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,10 @@ serde_json = { workspace = true }
serde_yaml = "0.9.13"
tikv-jemallocator = { workspace = true }

[[bench]]
harness = false
name = "state"

[[bench]]
harness = false
name = "vm"
169 changes: 169 additions & 0 deletions benches/benches/state.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,169 @@
use criterion::{
criterion_group,
criterion_main,
measurement::WallTime,
BenchmarkGroup,
Criterion,
};
use fuel_core::database::vm_database::VmDatabase;
use fuel_core_storage::InterpreterStorage;
use fuel_core_types::{
blockchain::header::GeneratedConsensusFields,
fuel_tx::Bytes32,
fuel_types::ContractId,
};
use rand::{
rngs::StdRng,
thread_rng,
Rng,
SeedableRng,
};
use std::{
iter,
time::Duration,
};

fn setup(db: &mut VmDatabase, contract: &ContractId, n: usize) {
let mut rng_keys = thread_rng();
let gen_keys = || -> Bytes32 { rng_keys.gen() };
let state_keys = iter::repeat_with(gen_keys).take(n);

let mut rng_values = thread_rng();
let gen_values = || -> Bytes32 { rng_values.gen() };
let state_values = iter::repeat_with(gen_values).take(n);

// State key-values
let state_key_values = state_keys.zip(state_values);

db.database_mut()
.init_contract_state(contract, state_key_values)
.expect("Failed to initialize contract state");
}

fn state_single_contract(c: &mut Criterion) {
let mut rng = StdRng::seed_from_u64(0xF00DF00D);
let state: Bytes32 = rng.gen();
let value: Bytes32 = rng.gen();

let mut bench_state = |group: &mut BenchmarkGroup<WallTime>, name: &str, n: usize| {
group.bench_function(name, |b| {
let mut db = VmDatabase::default();
let contract: ContractId = rng.gen();
setup(&mut db, &contract, n);
let outer = db.database_mut().transaction();
b.iter_custom(|iters| {
let mut elapsed_time = Duration::default();
for _ in 0..iters {
let mut inner = outer.transaction();
let mut inner_db = VmDatabase::new::<GeneratedConsensusFields>(
inner.as_mut().clone(),
&Default::default(),
Default::default(),
);
let start = std::time::Instant::now();
inner_db
.merkle_contract_state_insert(&contract, &state, &value)
.expect("failed to insert state into transaction");
elapsed_time += start.elapsed();
}
elapsed_time
});
});
};

let mut group = c.benchmark_group("state single contract");

bench_state(&mut group, "insert state with 0 preexisting entries", 0);
bench_state(&mut group, "insert state with 1 preexisting entry", 1);
bench_state(&mut group, "insert state with 10 preexisting entries", 10);
bench_state(&mut group, "insert state with 100 preexisting entries", 100);
bench_state(
&mut group,
"insert state with 1,000 preexisting entries",
1_000,
);
bench_state(
&mut group,
"insert state with 10,000 preexisting entries",
10_000,
);
bench_state(
&mut group,
"insert state with 100,000 preexisting entries",
100_000,
);
bench_state(
&mut group,
"insert state with 1,000,000 preexisting entries",
1_000_000,
);

group.finish();
}

fn state_multiple_contracts(c: &mut Criterion) {
let mut rng = StdRng::seed_from_u64(0xF00DF00D);
let state: Bytes32 = rng.gen();
let value: Bytes32 = rng.gen();

let mut bench_state = |group: &mut BenchmarkGroup<WallTime>, name: &str, n: usize| {
group.bench_function(name, |b| {
let mut db = VmDatabase::default();
for _ in 0..n {
let contract: ContractId = rng.gen();
setup(&mut db, &contract, 1);
}
let outer = db.database_mut().transaction();
b.iter_custom(|iters| {
let mut elapsed_time = Duration::default();
let contract: ContractId = rng.gen();
for _ in 0..iters {
let mut inner = outer.transaction();
let mut inner_db = VmDatabase::new::<GeneratedConsensusFields>(
inner.as_mut().clone(),
&Default::default(),
Default::default(),
);
let start = std::time::Instant::now();
inner_db
.merkle_contract_state_insert(&contract, &state, &value)
.expect("failed to insert state into transaction");
elapsed_time += start.elapsed();
}
elapsed_time
})
});
};

let mut group = c.benchmark_group("state multiple contracts");

bench_state(&mut group, "insert state with 0 preexisting entries", 0);
bench_state(&mut group, "insert state with 1 preexisting entry", 1);
bench_state(&mut group, "insert state with 10 preexisting entries", 10);
bench_state(&mut group, "insert state with 100 preexisting entries", 100);
bench_state(
&mut group,
"insert state with 1,000 preexisting entries",
1_000,
);
bench_state(
&mut group,
"insert state with 10,000 preexisting entries",
10_000,
);
bench_state(
&mut group,
"insert state with 100,000 preexisting entries",
100_000,
);
bench_state(
&mut group,
"insert state with 1,000,000 preexisting entries",
1_000_000,
);

group.finish();
}

criterion_group!(benches, state_single_contract, state_multiple_contracts);
criterion_main!(benches);

0 comments on commit 633de7c

Please sign in to comment.