Benchmark runner #137

martinsik · 2017-02-08T22:33:53Z

I've mentioned it in #134 already. This is a script runner with a couple of benchmarks that follow their RxJS 5 equivalents.

It can run all tests in the /benchmark/**/*.php directory with:

php benchmark/run.php

Or a single script:

php benchmark/run.php benchmark/distinct/distinct.php

This reverts commit 9a85ced.

coveralls · 2017-02-08T22:37:30Z

Coverage remained the same at 95.897% when pulling 1b61dba on martinsik:benchmark-runner into 84bfbd0 on ReactiveX:master.

davidwdan · 2017-02-09T00:54:11Z

@martinsik This looks awesome! I ran it against v2 and I'm getting some unexpected results. I would think that the performance would be about the same, but v1 is way faster for some of the operators!

v1:

Benchmarking 6 file/s (min 5s each)
script_name - total_runs (single_run_mean ±standard_variant)
============================================================
bufferWithCount - 107188 (0.05ms ±0.02ms)
distinct - 94617 (0.05ms ±0.03ms)
filter - 51071 (0.10ms ±0.03ms)
skipLast - 5888 (0.85ms ±0.16ms)
takeLast - 6514 (0.77ms ±0.14ms)
zip - 43674 (0.11ms ±0.03ms)
============================================================
total duration: 30.16s

v2:

Benchmarking 6 file/s (min 5s each)
script_name - total_runs (single_run_mean ±standard_variant)
============================================================
bufferWithCount - 42064 (0.12ms ±2.84ms)
distinct - 17970 (0.29ms ±6.12ms)
filter - 8470 (0.60ms ±10.11ms)
skipLast - 12612 (0.41ms ±9.08ms)
takeLast - 11036 (0.47ms ±10.37ms)
zip - 4856 (1.03ms ±16.54ms)
============================================================
total duration: 30.67s

davidwdan · 2017-02-09T00:55:02Z

benchmark/run.php

+define('MIN_TOTAL_DURATION', 5);
+$start = microtime(true);
+
+if ($_SERVER['argc'] == 1) {


Should that be a ===?

davidwdan · 2017-02-09T00:55:13Z

benchmark/run.php

+} else {
+    // Force absolute path
+    $files = array_map(function($file) {
+        return $file[0] == DIRECTORY_SEPARATOR ? $file : $_SERVER['PWD'] . DIRECTORY_SEPARATOR . $file;


Should that be a ===?

You're right with both.

davidwdan · 2017-02-09T01:40:35Z

Figured it out. It's because we changed the default scheduler in v2 to the EventLoopScheduler. When I set them to Immediate, I get results closer to what I was expecting:

Benchmarking 6 file/s (min 5s each)
script_name - total_runs (single_run_mean ±standard_variant)
============================================================
bufferWithCount - 113451 (0.04ms ±0.02ms)
distinct - 116431 (0.04ms ±0.02ms)
filter - 59151 (0.08ms ±0.03ms)
skipLast - 6476 (0.77ms ±0.13ms)
takeLast - 7155 (0.70ms ±0.12ms)
zip - 47154 (0.11ms ±0.03ms)
============================================================
total duration: 30.16s

I take it that the plan is to eventually add benchmarks for the different schedulers.

coveralls · 2017-02-09T20:03:05Z

Coverage remained the same at 95.897% when pulling 68d7a0b on martinsik:benchmark-runner into 84bfbd0 on ReactiveX:master.

davidwdan · 2017-02-09T20:18:15Z

With the latest changes:

v1:

Benchmarking 12 file/s (min 5s each)
script_name - total_runs (single_run_mean ±standard_deviation)
==============================================================
bufferWithCount - 107068 (0.05ms ±0.01ms)
bufferWithCount_eventloop - 6104 (0.82ms ±2.07ms)
distinct - 90765 (0.06ms ±0.05ms)
distinct_eventloop - 5935 (0.84ms ±2.05ms)
filter - 49457 (0.10ms ±0.03ms)
filter_eventloop - 2788 (1.80ms ±3.27ms)
skipLast - 5347 (0.94ms ±0.41ms)
skipLast_eventloop - 296 (16.89ms ±8.25ms)
takeLast - 6159 (0.81ms ±0.16ms)
takeLast_eventloop - 307 (16.32ms ±8.26ms)
zip - 42594 (0.12ms ±0.03ms)
zip_eventloop - 2846 (1.76ms ±2.42ms)
============================================================
total duration: 60.19s

v2:

Benchmarking 12 file/s (min 5s each)
script_name - total_runs (single_run_mean ±standard_deviation)
==============================================================
bufferWithCount - 118147 (0.04ms ±0.01ms)
bufferWithCount_eventloop - 30381 (0.17ms ±3.47ms)
distinct - 102737 (0.05ms ±0.39ms)
distinct_eventloop - 12915 (0.40ms ±7.58ms)
filter - 56806 (0.09ms ±0.14ms)
filter_eventloop - 6564 (0.77ms ±12.08ms)
skipLast - 6257 (0.80ms ±0.14ms)
skipLast_eventloop - 9151 (0.57ms ±11.23ms)
takeLast - 6927 (0.72ms ±0.13ms)
takeLast_eventloop - 7902 (0.65ms ±12.83ms)
zip - 47426 (0.11ms ±0.02ms)
zip_eventloop - 3609 (1.42ms ±20.08ms)
============================================================
total duration: 60.85s

martinsik · 2017-02-09T20:27:22Z

It didn't occur to me at first but I think we'll have to write scripts for both ImmediateScheduler and EventLoopScheduler similarly to https://github.com/ReactiveX/rxjs/tree/master/perf/micro in RxJS 5.

I updated run.php to also allows running all scripts in a single directory for practical reasons:

php benchmark/run.php benchmark/skipLast/

coveralls · 2017-02-18T03:24:53Z

Coverage increased (+0.01%) to 95.909% when pulling 3518c1c on martinsik:benchmark-runner into 84bfbd0 on ReactiveX:master.

coveralls · 2017-02-18T03:24:53Z

Coverage increased (+0.01%) to 95.909% when pulling 3518c1c on martinsik:benchmark-runner into 84bfbd0 on ReactiveX:master.

coveralls · 2017-02-21T21:15:35Z

Coverage increased (+0.01%) to 95.909% when pulling 0c792a7 on martinsik:benchmark-runner into 84bfbd0 on ReactiveX:master.

mbonneau · 2017-03-14T22:14:17Z

This was closed by github when I moved master to 2.x - I have reopened it against the 1.5 branch

coveralls · 2017-03-14T22:18:27Z

Changes Unknown when pulling 0c792a7 on martinsik:benchmark-runner into ** on ReactiveX:1.5**.

coveralls · 2017-03-14T22:18:27Z

Changes Unknown when pulling 0c792a7 on martinsik:benchmark-runner into ** on ReactiveX:1.5**.

coveralls · 2017-03-14T22:33:43Z

Changes Unknown when pulling 0c792a7 on martinsik:benchmark-runner into ** on ReactiveX:1.x**.

coveralls · 2017-03-14T22:33:43Z

Changes Unknown when pulling 0c792a7 on martinsik:benchmark-runner into ** on ReactiveX:1.x**.

coveralls · 2017-03-18T13:10:30Z

Changes Unknown when pulling 21489a7 on martinsik:benchmark-runner into ** on ReactiveX:1.x**.

martinsik · 2017-03-18T13:12:47Z

I updated the benchmark runner script so when using a custom loop it doesn't staring the loop over and over again. The results should be more accurate now:

Before:

Benchmarking 12 file/s (min 5s each)
script_name - total_runs (single_run_mean ±standard_deviation) - mem_start [mem_100_iter] mem_end
==============================================================
bufferWithCount - 33946 (0.15ms ±0.05ms) - 1.11MB 1.15MB 3.25MB 
bufferWithCount_eventloop - 2523 (1.98ms ±3.14ms) - 1.25MB 3.75MB 6.60MB 
distinct - 32050 (0.16ms ±0.12ms) - 6.47MB 6.53MB 3.33MB 
distinct_eventloop - 2627 (1.90ms ±2.94ms) - 2.28MB 6.21MB 5.22MB 
filter - 16977 (0.29ms ±0.08ms) - 5.09MB 5.11MB 6.15MB 
filter_eventloop - 1319 (3.79ms ±4.23ms) - 5.10MB 3.87MB 2.23MB 
skipLast - 1948 (2.57ms ±0.46ms) - 2.17MB 2.18MB 2.24MB 
skipLast_eventloop - 125 (40.04ms ±11.27ms) - 2.18MB 3.49MB 5.14MB 
takeLast - 2142 (2.33ms ±0.39ms) - 5.13MB 5.15MB 5.28MB 
takeLast_eventloop - 131 (38.20ms ±12.35ms) - 5.15MB 6.50MB 5.59MB 
zip - 14106 (0.35ms ±0.10ms) - 5.58MB 5.61MB 6.13MB 
zip_eventloop - 1704 (2.93ms ±2.59ms) - 5.61MB 2.07MB 5.21MB 
============================================================
total duration: 60.22s

After:

Benchmarking 12 file/s (min 5s each)
script_name - total_runs (single_run_mean ±standard_deviation) - mem_start [mem_100_iter] mem_end
==============================================================
bufferWithCount - 28074 (0.16ms ±0.30ms) - 1.14MB 1.16MB 2.20MB 
bufferWithCount_eventloop - 7383 (0.83ms ±1.56ms) - 1.25MB 4.16MB 4.18MB 6.30MB 
distinct - 25544 (0.17ms ±0.16ms) - 6.30MB 6.36MB 2.63MB 
distinct_eventloop - 26442 (0.16ms ±0.10ms) - 1.59MB 1.64MB 3.29MB 
filter - 16575 (0.27ms ±0.10ms) - 3.30MB 3.31MB 4.35MB 
filter_eventloop - 4037 (1.71ms ±2.22ms) - 3.31MB 5.55MB 
skipLast - 1814 (2.66ms ±0.34ms) - 5.55MB 5.57MB 5.63MB 
skipLast_eventloop - 419 (18.88ms ±26.40ms) - 5.56MB 2.89MB 
takeLast - 2215 (2.16ms ±0.29ms) - 2.90MB 2.91MB 3.04MB 
takeLast_eventloop - 443 (17.67ms ±24.99ms) - 2.91MB 6.11MB 
zip - 14025 (0.33ms ±0.08ms) - 6.12MB 6.14MB 6.66MB 
zip_eventloop - 3625 (1.36ms ±1.79ms) - 6.14MB 2.64MB 1.96MB 
============================================================
total duration: 60.26s

davidwdan · 2017-03-18T13:37:01Z

benchmark/run.php

-        if (!$testClosure) {
-            throw new Exception("Unable to load file \"$file\"");
+        if (is_array($testDef)) {
+            $sourceFactory = $testDef[0];


nit: You can use use list($sourceFactory, $loop) = $testDef here

coveralls · 2017-03-21T21:53:56Z

Changes Unknown when pulling a9a372b on martinsik:benchmark-runner into ** on ReactiveX:1.x**.

coveralls · 2017-03-22T11:13:03Z

Changes Unknown when pulling 6a72f45 on martinsik:benchmark-runner into ** on ReactiveX:1.x**.

martinsik added 8 commits February 5, 2017 16:03

benchmark runner

d3d3130

distinct operator benchmark

37d4649

fix variant

830c896

benchmark filter operator

40880cf

optimized distinct() operator

9a85ced

add types

baff252

Revert "optimized distinct() operator"

1873f7c

This reverts commit 9a85ced.

added more benchmarks, updated existing from RxJS 5

1b61dba

davidwdan reviewed Feb 9, 2017

View reviewed changes

added tests using EventLoopScheduler

68d7a0b

davidwdan mentioned this pull request Feb 10, 2017

Benchmark 2.x #138

Open

disallow running benchmarks with xdebug enabled

3518c1c

log memory usage

0c792a7

mbonneau closed this Mar 14, 2017

mbonneau changed the base branch from master to 1.5 March 14, 2017 22:13

mbonneau reopened this Mar 14, 2017

mbonneau closed this Mar 14, 2017

mbonneau changed the base branch from 1.5 to 1.x March 14, 2017 22:27

mbonneau reopened this Mar 14, 2017

refactored benchmark runner

21489a7

davidwdan approved these changes Mar 18, 2017

View reviewed changes

martinsik added 2 commits March 21, 2017 22:34

refactor to use list()

8566445

more benchmarks

a9a372b

more benchmarks

6a72f45

Benchmark runner #137

Benchmark runner #137

Uh oh!

Conversation

martinsik commented Feb 8, 2017

Uh oh!

coveralls commented Feb 8, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidwdan commented Feb 9, 2017

Uh oh!

davidwdan Feb 9, 2017

Choose a reason for hiding this comment

Uh oh!

davidwdan Feb 9, 2017

Choose a reason for hiding this comment

Uh oh!

martinsik Feb 9, 2017

Choose a reason for hiding this comment

Uh oh!

davidwdan commented Feb 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Feb 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidwdan commented Feb 9, 2017

Uh oh!

martinsik commented Feb 9, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coveralls commented Feb 18, 2017

Uh oh!

coveralls commented Feb 18, 2017

Uh oh!

coveralls commented Feb 21, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mbonneau commented Mar 14, 2017

Uh oh!

coveralls commented Mar 14, 2017

Uh oh!

coveralls commented Mar 14, 2017

Uh oh!

coveralls commented Mar 14, 2017

Uh oh!

coveralls commented Mar 14, 2017

Uh oh!

coveralls commented Mar 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martinsik commented Mar 18, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidwdan Mar 18, 2017

Choose a reason for hiding this comment

Uh oh!

coveralls commented Mar 21, 2017

Uh oh!

coveralls commented Mar 22, 2017

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

coveralls commented Feb 8, 2017 •

edited

Loading

davidwdan commented Feb 9, 2017 •

edited

Loading

coveralls commented Feb 9, 2017 •

edited

Loading

martinsik commented Feb 9, 2017 •

edited

Loading

coveralls commented Feb 21, 2017 •

edited

Loading

coveralls commented Mar 18, 2017 •

edited

Loading

martinsik commented Mar 18, 2017 •

edited

Loading