@@ -62,8 +62,15 @@ from [nghttp2.org][] or built from source.
6262
6363### Benchmark analysis requirements
6464
65- To analyze the results, ` R ` should be installed. Use one of the available
66- package managers or download it from < https://www.r-project.org/ > .
65+ To analyze the results statistically, you can use either the
66+ [ node-benchmark-compare] [ ] tool or the R script ` benchmark/compare.R ` .
67+
68+ [ node-benchmark-compare] [ ] is a Node.js script that can be installed with
69+ ` npm install -g node-benchmark-compare ` .
70+
71+ To draw comparison plots when analyzing the results, ` R ` must be installed.
72+ Use one of the available package managers or download it from
73+ < https://www.r-project.org/ > .
6774
6875The R packages ` ggplot2 ` and ` plyr ` are also used and can be installed using
6976the R REPL.
@@ -285,10 +292,11 @@ module, you can use the `--filter` option:*
285292 --no-progress don't show benchmark progress indicator
286293```
287294
288- For analysing the benchmark results use the ` compare.R ` tool.
295+ For analysing the benchmark results, use [ node-benchmark-compare] [ ] or the R
296+ script ` benchmark/compare.R ` .
289297
290298``` console
291- $ cat compare-pr-5134.csv | Rscript benchmark/compare.R
299+ $ node-benchmark-compare compare-pr-5134.csv # or cat compare-pr-5134.csv | Rscript benchmark/compare.R
292300
293301 confidence improvement accuracy (*) (**) (***)
294302 string_decoder/string-decoder.js n=2500000 chunkLen=16 inLen=128 encoding='ascii' *** -3.76 % ±1.36% ±1.82% ±2.40%
@@ -315,17 +323,18 @@ consider at least two stars (`**`) as the threshold, in that case the risk
315323is 1%. If three stars (` *** ` ) is considered the risk is 0.1%. However this
316324may require more runs to obtain (can be set with ` --runs ` ).
317325
318- _ For the statistically minded, the R script performs an [ independent/unpaired
326+ _ For the statistically minded, the script performs an [ independent/unpaired
3193272-group t-test] [ t-test ] , with the null hypothesis that the performance is the
320328same for both versions. The confidence field will show a star if the p-value
321329is less than ` 0.05 ` ._
322330
323- The ` compare.R ` tool can also produce a box plot by using the ` --plot filename `
324- option. In this case there are 48 different benchmark combinations, and there
325- may be a need to filter the csv file. This can be done while benchmarking
326- using the ` --set ` parameter (e.g. ` --set encoding=ascii ` ) or by filtering
327- results afterwards using tools such as ` sed ` or ` grep ` . In the ` sed ` case be
328- sure to keep the first line since that contains the header information.
331+ The ` compare.R ` tool can additionally produce a box plot by using the
332+ ` --plot filename ` option. In this case there are 48 different benchmark
333+ combinations, and there may be a need to filter the csv file. This can be done
334+ while benchmarking using the ` --set ` parameter (e.g. ` --set encoding=ascii ` ) or
335+ by filtering results afterwards using tools such as ` sed ` or ` grep ` . In the
336+ ` sed ` case be sure to keep the first line since that contains the header
337+ information.
329338
330339``` console
331340$ cat compare-pr-5134.csv | sed ' 1p;/encoding=' " '" ascii" '" ' /!d' | Rscript benchmark/compare.R --plot compare-plot.png
@@ -560,5 +569,6 @@ Supported options keys are:
560569[ benchmark-ci ] : https://github.com/nodejs/benchmarking/blob/HEAD/docs/core_benchmarks.md
561570[ git-for-windows ] : https://git-scm.com/download/win
562571[ nghttp2.org ] : https://nghttp2.org
572+ [ node-benchmark-compare ] : https://github.com/targos/node-benchmark-compare
563573[ t-test ] : https://en.wikipedia.org/wiki/Student%27s_t-test#Equal_or_unequal_sample_sizes.2C_unequal_variances
564574[ wrk ] : https://github.com/wg/wrk
0 commit comments