Run baseline and new benchmarks; compare results, report speedups/slowdowns and significance #68
Description
Interpreting benchmark results can be hard; in SwiftUI (and I think this was taken from something done for Swift's benchmarks) the benchmark tool would run a baseline state and new state of the code in the same process, interleaving runs, and report how much speedup or slowdown there is. Also the reports would be adorned with a mark that indicated definite speedup, definite slowdown, or statistically insignificant result. These facilities definitely helped us understand the effects we were seeing and also whether we needed to increase the iteration count or manually kill off processes, etc., to get a useful measurement. IIRC there were three ways to measure the mean result and we mostly ignored everything but the geometric mean. I can probably find out a bit more about the specifics on request. It's also very valuable to have this kind of report from CI.
Activity