Skip to content

Conversation

@SiarheiFedartsou
Copy link
Member

@SiarheiFedartsou SiarheiFedartsou commented Jun 6, 2024

The goal here is to have more "diversified" benchmarks, i.e.:

  1. Be able to run it against bigger dataset than just Berlin (it is enabled via adding magic "Performance" label to PR): I chose Poland as it was small enough to somehow work on CI machines from GitHub :)
  2. Be able to cover more cases via using random routes, but not fixed ones

It is kind of similar to e2e tests we added in one of recent PRs, but benchmarks are written in C++ and run against libosrm, but not osrm-routed.

Benchmark Results

Benchmark Base PR
alias aliased u32: 1083.64
plain u32: 1079.6
aliased double: 953.734
plain double: 950.187
aliased u32: 1093.13
plain u32: 1093.32
aliased double: 953.474
plain double: 947.916
e2e_match_ch Total: 57224.499225616455ms
Min time: 4.024982452392578ms
Mean time: 59.11621820828146ms
Median time: 41.547536849975586ms
95th percentile: 183.1827044486999ms
99th percentile: 434.13395643234264ms
Max time: 492.4740791320801ms
Total: 54882.668018341064ms
Min time: 4.048824310302734ms
Mean time: 56.69697109332755ms
Median time: 40.22789001464844ms
95th percentile: 168.15186738967896ms
99th percentile: 398.0480027198792ms
Max time: 458.0681324005127ms
e2e_match_mld Total: 38384.233713150024ms
Min time: 3.398895263671875ms
Mean time: 39.65313400118804ms
Median time: 30.88390827178955ms
95th percentile: 119.09809112548822ms
99th percentile: 253.042521476746ms
Max time: 296.2648868560791ms
Total: 38459.699630737305ms
Min time: 3.421306610107422ms
Mean time: 39.73109465985259ms
Median time: 30.67755699157715ms
95th percentile: 119.7221755981445ms
99th percentile: 255.00832557678245ms
Max time: 299.21770095825195ms
e2e_nearest_ch Total: 1356.0309410095215ms
Min time: 1.1560916900634766ms
Mean time: 1.3560309410095215ms
Median time: 1.2714862823486328ms
95th percentile: 1.759326457977295ms
99th percentile: 1.7977166175842285ms
Max time: 2.1240711212158203ms
Total: 1356.16135597229ms
Min time: 1.1622905731201172ms
Mean time: 1.35616135597229ms
Median time: 1.270890235900879ms
95th percentile: 1.7720460891723633ms
99th percentile: 1.8274998664855957ms
Max time: 1.8966197967529297ms
e2e_nearest_mld Total: 1356.9645881652832ms
Min time: 1.1353492736816406ms
Mean time: 1.3569645881652832ms
Median time: 1.2695789337158203ms
95th percentile: 1.7695426940917969ms
99th percentile: 1.8039345741271973ms
Max time: 2.173900604248047ms
Total: 1338.1531238555908ms
Min time: 1.1463165283203125ms
Mean time: 1.3381531238555908ms
Median time: 1.2562274932861328ms
95th percentile: 1.7443299293518066ms
99th percentile: 1.79368257522583ms
Max time: 1.9650459289550781ms
e2e_route_ch Total: 4331.030607223511ms
Min time: 1.3818740844726562ms
Mean time: 4.331030607223511ms
Median time: 4.546999931335449ms
95th percentile: 6.4900994300842285ms
99th percentile: 7.155919075012207ms
Max time: 8.391857147216797ms
Total: 4376.399040222168ms
Min time: 1.4405250549316406ms
Mean time: 4.376399040222168ms
Median time: 4.576206207275391ms
95th percentile: 6.480705738067626ms
99th percentile: 7.098591327667236ms
Max time: 7.613658905029297ms
e2e_route_mld Total: 6212.393283843994ms
Min time: 1.4238357543945312ms
Mean time: 6.212393283843994ms
Median time: 6.524801254272461ms
95th percentile: 9.512269496917725ms
99th percentile: 10.237679481506348ms
Max time: 10.898113250732422ms
Total: 6121.039867401123ms
Min time: 1.4231204986572266ms
Mean time: 6.121039867401123ms
Median time: 6.423234939575195ms
95th percentile: 9.348511695861816ms
99th percentile: 9.903013706207275ms
Max time: 10.885953903198242ms
e2e_table_ch Total: 18127.747297286987ms
Min time: 2.0105838775634766ms
Mean time: 18.127747297286987ms
Median time: 17.17674732208252ms
95th percentile: 34.09578800201415ms
99th percentile: 36.0596227645874ms
Max time: 37.95123100280762ms
Total: 19232.73515701294ms
Min time: 2.2161006927490234ms
Mean time: 19.23273515701294ms
Median time: 18.1657075881958ms
95th percentile: 36.10975742340088ms
99th percentile: 38.077383041381836ms
Max time: 39.65425491333008ms
e2e_table_mld Total: 210515.6626701355ms
Min time: 10.004043579101562ms
Mean time: 210.5156626701355ms
Median time: 199.46730136871338ms
95th percentile: 408.1838369369507ms
99th percentile: 431.2767624855041ms
Max time: 505.50103187561035ms
Total: 211779.85954284668ms
Min time: 9.959936141967773ms
Mean time: 211.77985954284668ms
Median time: 200.26063919067383ms
95th percentile: 411.81390285491943ms
99th percentile: 436.5763211250305ms
Max time: 464.2190933227539ms
e2e_trip_ch Total: 16362.309455871582ms
Min time: 2.31170654296875ms
Mean time: 16.362309455871582ms
Median time: 16.530275344848633ms
95th percentile: 25.089335441589355ms
99th percentile: 26.507506370544434ms
Max time: 34.74164009094238ms
Total: 17066.81752204895ms
Min time: 2.393960952758789ms
Mean time: 17.06681752204895ms
Median time: 17.2579288482666ms
95th percentile: 26.066029071807858ms
99th percentile: 27.829279899597168ms
Max time: 30.863523483276367ms
e2e_trip_mld Total: 45663.48075866699ms
Min time: 6.854772567749023ms
Mean time: 45.66348075866699ms
Median time: 46.689510345458984ms
95th percentile: 68.59872341156006ms
99th percentile: 70.54884195327759ms
Max time: 73.65894317626953ms
Total: 46039.14475440979ms
Min time: 6.834983825683594ms
Mean time: 46.03914475440979ms
Median time: 46.92816734313965ms
95th percentile: 69.15066242218018ms
99th percentile: 71.47207021713257ms
Max time: 76.27081871032715ms
json-render String: 6.76086ms
Stringstream: 8.81286ms
Vector: 6.84986ms
String: 6.63169ms
Stringstream: 9.40745ms
Vector: 6.95798ms
match_ch Default radius:
4.42754ms/req at 82 coordinate
0.0539944ms/coordinate
Radius 5m:
4.38611ms/req at 82 coordinate
0.0534892ms/coordinate
Radius 10m:
14.9625ms/req at 82 coordinate
0.18247ms/coordinate
Radius 15m:
36.6464ms/req at 82 coordinate
0.446907ms/coordinate
Radius 30m:
313.341ms/req at 82 coordinate
3.82123ms/coordinate
Default radius:
4.41412ms/req at 82 coordinate
0.0538307ms/coordinate
Radius 5m:
4.40452ms/req at 82 coordinate
0.0537136ms/coordinate
Radius 10m:
15.0524ms/req at 82 coordinate
0.183566ms/coordinate
Radius 15m:
36.6571ms/req at 82 coordinate
0.447038ms/coordinate
Radius 30m:
311.97ms/req at 82 coordinate
3.80451ms/coordinate
match_mld Default radius:
2.78561ms/req at 82 coordinate
0.0339709ms/coordinate
Radius 5m:
2.76198ms/req at 82 coordinate
0.0336826ms/coordinate
Radius 10m:
10.2232ms/req at 82 coordinate
0.124673ms/coordinate
Radius 15m:
25.8556ms/req at 82 coordinate
0.315313ms/coordinate
Radius 30m:
303.518ms/req at 82 coordinate
3.70144ms/coordinate
Default radius:
2.77916ms/req at 82 coordinate
0.0338922ms/coordinate
Radius 5m:
2.76928ms/req at 82 coordinate
0.0337717ms/coordinate
Radius 10m:
10.1692ms/req at 82 coordinate
0.124015ms/coordinate
Radius 15m:
26.0818ms/req at 82 coordinate
0.31807ms/coordinate
Radius 30m:
306.859ms/req at 82 coordinate
3.74218ms/coordinate
osrm_contract Time: 827.40s Peak RAM: 2967.22MB Time: 830.42s Peak RAM: 2963.85MB
osrm_customize Time: 24.16s Peak RAM: 1943.72MB Time: 24.15s Peak RAM: 1925.04MB
osrm_extract Time: 274.99s Peak RAM: 5227.39MB Time: 277.74s Peak RAM: 5236.08MB
osrm_partition Time: 104.70s Peak RAM: 2179.76MB Time: 106.02s Peak RAM: 2179.88MB
packedvector random write:
std::vector 9810.3 ms
util::packed_vector 81787.4 ms
slowdown: 8.33689
random read:
std::vector 8469.8 ms
util::packed_vector 33199.1 ms
slowdown: 3.91971
random write:
std::vector 9740.47 ms
util::packed_vector 73379 ms
slowdown: 7.53341
random read:
std::vector 8447.72 ms
util::packed_vector 29879.3 ms
slowdown: 3.53696
random_match_ch 1000 matches, default radius
total: 78202.47ms
avg: 78.20ms
min: 0.48ms
max: 503.92ms
p99: 497.28ms

1000 matches, radius=10
total: 276336.46ms
avg: 276.34ms
min: 0.60ms
max: 1780.56ms
p99: 1745.96ms

1000 matches, radius=20
total: 1328821.14ms
avg: 1328.82ms
min: 4.38ms
max: 8125.80ms
p99: 8055.83ms
random_match_mld 1000 matches, default radius
total: 52138.88ms
avg: 52.14ms
min: 0.47ms
max: 320.35ms
p99: 315.54ms

1000 matches, radius=10
total: 184917.09ms
avg: 184.92ms
min: 0.57ms
max: 1187.55ms
p99: 1165.00ms

1000 matches, radius=20
total: 924148.77ms
avg: 924.15ms
min: 2.49ms
max: 5294.74ms
p99: 5272.57ms
random_nearest_ch 10000 nearest, number_of_results=1
total: 520.99ms
avg: 0.05ms
min: 0.02ms
max: 0.39ms
p99: 0.10ms

10000 nearest, number_of_results=5
total: 638.95ms
avg: 0.06ms
min: 0.03ms
max: 0.15ms
p99: 0.12ms

10000 nearest, number_of_results=10
total: 780.73ms
avg: 0.08ms
min: 0.04ms
max: 0.17ms
p99: 0.13ms
random_nearest_mld 10000 nearest, number_of_results=1
total: 519.45ms
avg: 0.05ms
min: 0.02ms
max: 0.44ms
p99: 0.11ms

10000 nearest, number_of_results=5
total: 634.51ms
avg: 0.06ms
min: 0.03ms
max: 0.13ms
p99: 0.12ms

10000 nearest, number_of_results=10
total: 783.69ms
avg: 0.08ms
min: 0.04ms
max: 0.16ms
p99: 0.13ms
random_route_ch 10000 routes, 3 coordinates, no alternatives, overview=full, steps=true
total: 34849.64ms
avg: 3.48ms
min: 0.19ms
max: 15.01ms
p99: 5.89ms

10000 routes, 2 coordinates, no alternatives, overview=full, steps=true
total: 16845.64ms
avg: 1.68ms
min: 0.10ms
max: 4.45ms
p99: 2.95ms

10000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
total: 26965.10ms
avg: 2.70ms
min: 0.12ms
max: 10.96ms
p99: 5.87ms

10000 routes, 3 coordinates, no alternatives, overview=false, steps=false
total: 13132.51ms
avg: 1.31ms
min: 0.15ms
max: 2.82ms
p99: 2.05ms

10000 routes, 2 coordinates, no alternatives, overview=false, steps=false
total: 5823.38ms
avg: 0.58ms
min: 0.06ms
max: 1.21ms
p99: 0.95ms

10000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
total: 11388.02ms
avg: 1.14ms
min: 0.08ms
max: 2.83ms
p99: 2.20ms

10000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
total: 13080.85ms
avg: 1.31ms
min: 0.16ms
max: 2.69ms
p99: 2.05ms

10000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
total: 5845.14ms
avg: 0.58ms
min: 0.06ms
max: 1.22ms
p99: 0.95ms

10000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
total: 11463.23ms
avg: 1.15ms
min: 0.08ms
max: 2.77ms
p99: 2.19ms
random_route_mld 10000 routes, 3 coordinates, no alternatives, overview=full, steps=true
total: 89515.91ms
avg: 8.95ms
min: 0.17ms
max: 29.70ms
p99: 17.76ms

10000 routes, 2 coordinates, no alternatives, overview=full, steps=true
total: 34238.26ms
avg: 3.42ms
min: 0.10ms
max: 8.62ms
p99: 6.05ms

10000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
total: 59592.65ms
avg: 5.96ms
min: 0.10ms
max: 14.70ms
p99: 11.41ms

10000 routes, 3 coordinates, no alternatives, overview=false, steps=false
total: 67406.97ms
avg: 6.74ms
min: 0.15ms
max: 28.41ms
p99: 14.35ms

10000 routes, 2 coordinates, no alternatives, overview=false, steps=false
total: 22742.97ms
avg: 2.27ms
min: 0.06ms
max: 6.09ms
p99: 4.14ms

10000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
total: 44386.31ms
avg: 4.44ms
min: 0.07ms
max: 11.08ms
p99: 8.12ms

10000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
total: 66925.50ms
avg: 6.69ms
min: 0.20ms
max: 25.57ms
p99: 14.38ms

10000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
total: 22920.88ms
avg: 2.29ms
min: 0.06ms
max: 5.90ms
p99: 4.29ms

10000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
total: 44548.52ms
avg: 4.45ms
min: 0.06ms
max: 10.79ms
p99: 8.10ms
random_table_ch 250 tables, 3 coordinates
total: 235.88ms
avg: 0.94ms
min: 0.61ms
max: 3.91ms
p99: 1.91ms

250 tables, 25 coordinates
total: 1874.42ms
avg: 7.50ms
min: 6.85ms
max: 8.21ms
p99: 8.10ms

250 tables, 50 coordinates
total: 3750.36ms
avg: 15.00ms
min: 13.86ms
max: 16.05ms
p99: 15.81ms

250 tables, 100 coordinates
total: 7769.71ms
avg: 31.08ms
min: 29.37ms
max: 34.12ms
p99: 32.69ms
random_table_mld 250 tables, 3 coordinates
total: 2805.73ms
avg: 11.22ms
min: 7.12ms
max: 18.02ms
p99: 15.15ms

250 tables, 25 coordinates
total: 25058.06ms
avg: 100.23ms
min: 88.52ms
max: 118.71ms
p99: 114.25ms

250 tables, 50 coordinates
total: 51528.55ms
avg: 206.11ms
min: 190.15ms
max: 228.51ms
p99: 227.75ms

250 tables, 100 coordinates
total: 109533.81ms
avg: 438.14ms
min: 408.95ms
max: 497.78ms
p99: 486.55ms
random_trip_ch 1000 trips, 3 coordinates
total: 4264.20ms
avg: 4.26ms
min: 1.06ms
max: 13.63ms
p99: 6.38ms

1000 trips, 4 coordinates
total: 5410.67ms
avg: 5.41ms
min: 1.39ms
max: 11.46ms
p99: 7.69ms

1000 trips, 5 coordinates
total: 6199.17ms
avg: 6.20ms
min: 1.59ms
max: 12.47ms
p99: 8.23ms
random_trip_mld 1000 trips, 3 coordinates
total: 20642.97ms
avg: 20.64ms
min: 7.42ms
max: 37.05ms
p99: 27.66ms

1000 trips, 4 coordinates
total: 26384.84ms
avg: 26.38ms
min: 15.36ms
max: 35.37ms
p99: 32.79ms

1000 trips, 5 coordinates
total: 32035.37ms
avg: 32.04ms
min: 19.99ms
max: 42.84ms
p99: 40.03ms
route_ch 1000 routes, 3 coordinates, no alternatives, overview=full, steps=true
510.256ms
0.510256ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
351.894ms
0.351894ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
628.455ms
0.628455ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
151.514ms
0.151514ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
97.5668ms
0.0975668ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
133.414ms
0.133414ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
151.398ms
0.151398ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
97.8421ms
0.0978421ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
133.048ms
0.133048ms/req
1000 routes, 3 coordinates, no alternatives, overview=full, steps=true
508.457ms
0.508457ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
352.12ms
0.35212ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
626.2ms
0.6262ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
150.082ms
0.150082ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
96.9107ms
0.0969107ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
132.324ms
0.132324ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
150.562ms
0.150562ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
97.0503ms
0.0970503ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
132.339ms
0.132339ms/req
route_mld 1000 routes, 3 coordinates, no alternatives, overview=full, steps=true
640.358ms
0.640358ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
437.477ms
0.437477ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
812.91ms
0.81291ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
264.782ms
0.264782ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
161.13ms
0.16113ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
289.467ms
0.289467ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
274.668ms
0.274668ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
161.357ms
0.161357ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
288.555ms
0.288555ms/req
1000 routes, 3 coordinates, no alternatives, overview=full, steps=true
639.497ms
0.639497ms/req
1000 routes, 2 coordinates, no alternatives, overview=full, steps=true
437.242ms
0.437242ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=full, steps=true
816.906ms
0.816906ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false
260.396ms
0.260396ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false
160.175ms
0.160175ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false
285.776ms
0.285776ms/req
1000 routes, 3 coordinates, no alternatives, overview=false, steps=false, radius=750
258.091ms
0.258091ms/req
1000 routes, 2 coordinates, no alternatives, overview=false, steps=false, radius=750
159.67ms
0.15967ms/req
1000 routes, 2 coordinates, 3 alternatives, overview=false, steps=false, radius=750
284.769ms
0.284769ms/req
rtree 1 result:
206.687ms -> 0.0206687 ms/query
10 results:
241.985ms -> 0.0241985 ms/query
1 result:
207.947ms -> 0.0207947 ms/query
10 results:
243.336ms -> 0.0243336 ms/query

@SiarheiFedartsou SiarheiFedartsou marked this pull request as ready for review June 9, 2024 09:12
@SiarheiFedartsou SiarheiFedartsou merged commit 89435aa into master Jun 9, 2024
@SiarheiFedartsou SiarheiFedartsou deleted the sf-random-bench branch June 9, 2024 15:03
eliseier pushed a commit to wanderlog/osrm-backend that referenced this pull request Mar 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants