Change the order of benchmark execution

[leanil]

I'm using repetitions to cancel out "short" interferences (caused by system processes) by taking the minimum run time. In this case it seems clearly beneficial to space out repetitions evenly, rather then run them one after another.

Let's say I have 3 benchmarks and 2 repetitions, each for 1 sec time. I'd like to run them
b1 b2 b3 b1 b2 b3 rather than b1 b1 b2 b2 b3 b3 ,
because the former resists a 3 sec interference, while a 2 sec interference can ruin the latter.

Do you think this makes sense?

[LebedevRI]

This is essentially what llvm lnt/test-suite does - instead of using in-benchmark repetitions,
it just re-runs the entire benchmarks. I personally have not found that very convincing yet.
Though yeah, i have observed nasty results of system-induced noise, it's ugly.

The caveat is that i think this will give false hope - while sure, this will help somewhat
if the interference was a short spike around a single repetition, if it was longer than that
i don't think it will have much of an effect.

And the main problem, i think, if you are doing two benchmarks (2 executables
from different versions), i'd think you'd want to merge those two runs too
{ob1 ob2 ob3+nb1 nb2 nb3 -> ob1 nb1 ob2 nb2 ob3 nb3}, and you can't do that..

[leanil]

instead of using in-benchmark repetitions, it just re-runs the entire benchmarks.

Yes, I was about to do that, when I thought it could be a library function. I will probably implement it anyway (and make a PR, if you'd like to have it as a separate tooling script).

And the main problem, i think, if you are doing two benchmarks (2 executables
from different versions), i'd think you'd want to merge those two runs too
{ob1 ob2 ob3+nb1 nb2 nb3 -> ob1 nb1 ob2 nb2 ob3 nb3}, and you can't do that..

Sure, I'd want to merge those, and it would be nice if it wasn't impossible. But why is that a major problem? On the other hand, if I have lightweight micro benchmarks so that I can compose the different versions into one executable, then we are back at the original question.

[LebedevRI]

I'm not saying it's a bad idea, but there are some obvious questions, e.g.:

• What happens if different benchmarks have different repetition count?
  Should it just randsort all the different repetitions of all benchmarks?
• What happens with output? If the repetitions are not in order,
  the console output will be unreadable? Granted, this does not affect JSON.

[LebedevRI]

Also, i guess this won't affect fixtures as those are already created for each repetition, so that's good.

[dmah42]

I've been playing with something around this internally. stay tuned.

[dmah42]

#1105 is the implementation that solves this.

[dmah42]

fixed by #1163