Skip to content

Conversation

@Centril
Copy link
Contributor

@Centril Centril commented Nov 28, 2024

Description of Changes

Basically replace with ::Refs as needed.

The numbers, just to set a baseline for any future work, are as of this PR:

Benchmarking full-scan: Collecting 100 samples in estimated 5.6924 s (200
full-scan               time:   [27.842 ms 28.019 ms 28.249 ms]

Benchmarking full-join: Collecting 100 samples in estimated 5.7658 s (25k
full-join               time:   [222.02 µs 222.63 µs 223.53 µs]

Benchmarking incr-select: Collecting 100 samples in estimated 5.0003 s (3
incr-select             time:   [153.60 ns 159.05 ns 169.53 ns]

Benchmarking incr-join: Collecting 100 samples in estimated 5.0028 s (7.0
incr-join               time:   [700.16 ns 704.12 ns 711.24 ns]

Benchmarking query-indexes-multi: Collecting 100 samples in estimated 5.0
query-indexes-multi     time:   [640.48 ns 643.12 ns 648.56 ns]

Fixes #1993

API and ABI breaking changes

No, this just fixes tests and benches.

@Centril Centril requested a review from kazimuth November 28, 2024 20:47
@Centril Centril force-pushed the centril/fix-sub-benches branch from 68857bc to d3c7265 Compare November 28, 2024 21:14
@bfops bfops added the release-any To be landed in any release window label Dec 2, 2024
@Centril Centril requested a review from RReverser December 4, 2024 18:58
@Centril
Copy link
Contributor Author

Centril commented Dec 5, 2024

benchmarks please

@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2024

Benchmarking failed. Please check the workflow run for details.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2024

Callgrind benchmark results

Callgrind Benchmark Report

These benchmarks were run using callgrind,
an instruction-level profiler. They allow comparisons between sqlite (sqlite), SpacetimeDB running through a module (stdb_module), and the underlying SpacetimeDB data storage engine (stdb_raw). Callgrind emulates a CPU to collect the below estimates.

Measurement changes larger than five percent are in bold.

In-memory benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6432 6396 0.56% 6616 6532 1.29%
sqlite 5609 5609 0.00% 6095 6015 1.33%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 74706 74672 0.05% 75186 75420 -0.31%
stdb_raw u32_u64_str no_index 64 128 2 string 116948 116914 0.03% 117686 117940 -0.22%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25117 25081 0.14% 25767 25771 -0.02%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24084 24048 0.15% 24656 24776 -0.48%
sqlite u32_u64_str no_index 64 128 2 string 144415 144415 0.00% 145823 145837 -0.01%
sqlite u32_u64_str no_index 64 128 1 u64 123763 123763 0.00% 125009 124907 0.08%
sqlite u32_u64_str btree_each_column 64 128 1 u64 131080 131080 0.00% 132514 132504 0.01%
sqlite u32_u64_str btree_each_column 64 128 2 string 134222 134222 0.00% 135818 135832 -0.01%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 880651 881594 -0.11% 931027 927476 0.38%
stdb_raw u32_u64_str btree_each_column 64 128 1029166 1029361 -0.02% 1090016 1053101 3.51%
sqlite u32_u64_str unique_0 64 128 399360 399366 -0.00% 416482 418334 -0.44%
sqlite u32_u64_str btree_each_column 64 128 984611 984611 0.00% 1020973 1023405 -0.24%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 138485 138372 0.08% 138679 138482 0.14%
stdb_raw u32_u64_str unique_0 64 15910 15797 0.72% 16096 15915 1.14%
sqlite u32_u64_str unique_0 1024 1042718 1042718 0.00% 1046074 1046038 0.00%
sqlite u32_u64_str unique_0 64 74704 74710 -0.01% 75842 75806 0.05%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50180 50282 -0.20%
64 bsatn 25509 25509 0.00% 27787 27719 0.25%
16 bsatn 8200 8200 0.00% 9628 9560 0.71%
16 json 12188 12188 0.00% 14092 14160 -0.48%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 20567742 20130299 2.17% 21220382 20588343 3.07%
stdb_raw u32_u64_str unique_0 64 128 1289582 1287061 0.20% 1365650 1320957 3.38%
sqlite u32_u64_str unique_0 1024 1024 1802137 1802137 0.00% 1811311 1811273 0.00%
sqlite u32_u64_str unique_0 64 128 128540 128540 0.00% 131468 131336 0.10%
On-disk benchmarks

callgrind: empty transaction

db total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw 6437 6401 0.56% 6617 6549 1.04%
sqlite 5651 5651 0.00% 6163 6091 1.18%

callgrind: filter

db schema indices count preload _column data_type total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str no_index 64 128 1 u64 74711 74677 0.05% 75155 75349 -0.26%
stdb_raw u32_u64_str no_index 64 128 2 string 116974 116919 0.05% 117684 117861 -0.15%
stdb_raw u32_u64_str btree_each_column 64 128 2 string 25121 25085 0.14% 25747 25743 0.02%
stdb_raw u32_u64_str btree_each_column 64 128 1 u64 24089 24053 0.15% 24649 24709 -0.24%
sqlite u32_u64_str no_index 64 128 1 u64 125702 125684 0.01% 127316 127200 0.09%
sqlite u32_u64_str no_index 64 128 2 string 146336 146336 0.00% 148096 148026 0.05%
sqlite u32_u64_str btree_each_column 64 128 2 string 136418 136418 0.00% 138466 138444 0.02%
sqlite u32_u64_str btree_each_column 64 128 1 u64 133176 133176 0.00% 135034 134988 0.03%

callgrind: insert bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 64 128 830790 830143 0.08% 880736 875167 0.64%
stdb_raw u32_u64_str btree_each_column 64 128 979612 980922 -0.13% 1039548 1037274 0.22%
sqlite u32_u64_str unique_0 64 128 416914 416914 0.00% 433562 435364 -0.41%
sqlite u32_u64_str btree_each_column 64 128 1023158 1023158 0.00% 1058568 1062324 -0.35%

callgrind: iterate

db schema indices count total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 138490 138377 0.08% 138668 138467 0.15%
stdb_raw u32_u64_str unique_0 64 15915 15802 0.72% 16093 15892 1.26%
sqlite u32_u64_str unique_0 1024 1045796 1045796 0.00% 1049640 1049528 0.01%
sqlite u32_u64_str unique_0 64 76486 76486 0.00% 77808 77830 -0.03%

callgrind: serialize_product_value

count format total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
64 json 47528 47528 0.00% 50180 50282 -0.20%
64 bsatn 25509 25509 0.00% 27787 27719 0.25%
16 bsatn 8200 8200 0.00% 9628 9560 0.71%
16 json 12188 12188 0.00% 14092 14160 -0.48%

callgrind: update bulk

db schema indices count preload total reads + writes old total reads + writes Δrw estimated cycles old estimated cycles Δcycles
stdb_raw u32_u64_str unique_0 1024 1024 19056199 19059120 -0.02% 19762863 19627540 0.69%
stdb_raw u32_u64_str unique_0 64 128 1240874 1240513 0.03% 1315652 1306115 0.73%
sqlite u32_u64_str unique_0 1024 1024 1809785 1809791 -0.00% 1818535 1818339 0.01%
sqlite u32_u64_str unique_0 64 128 132687 132687 0.00% 135771 135643 0.09%

@Centril Centril added this pull request to the merge queue Dec 5, 2024
Merged via the queue into master with commit 6f22261 Dec 5, 2024
10 checks passed
@Centril Centril deleted the centril/fix-sub-benches branch December 5, 2024 14:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release-any To be landed in any release window

Projects

None yet

Development

Successfully merging this pull request may close these issues.

subscription benchmark is broken

4 participants