Add a running index to set-operations benchmarks.

[AndreasPK]

This avoids issues caused by different benchmarks with the same name.
This fixes #621.

[treeowl]

How many benchmarks have the same name? If we can rename just a few instead of adding a whole numbering mechanism, I would prefer that.

[AndreasPK]

All of set-operations-* appear (and are run) twice.

[treeowl]

OK. I'll accept it. But I absolutely can't stand the way this module was and is written. It's absurdly difficult to see what the heck it's doing with all those hacked up list operations. Do you think you understand what's going on well enough to do it more sensibly?

[AndreasPK]

OK. I'll accept it. But I absolutely can't stand the way this module was and is written. It's absurdly difficult to see what the heck it's doing with all those hacked up list operations.

Thats the reason why I made it 1-n over all benchmarks instead of having -run1 -run2 etc.

Do you think you understand what's going on well enough to do it more sensibly?

Most of it, I guess I can take a second look and see if I can make it saner.

[treeowl]

When the benchmark harness is too complicated to even number the benchmarks the way we'd prefer, it's time to fix the harness.

[treeowl]

Much better already. What is a data variant? What is a data_str?

[treeowl]

Do we want to add swap, or (guessing) remove the nn duplicates?

[AndreasPK]

I removed duplicating of (some) benchmarks which the old version did.
I'm not sure why it was there and I only see <1% differences between the duplicates.

Do we want to add swap, or (guessing) remove the nn duplicates?

They are actually not duplicate (sigh).
For example _nn might be given by

!disj_nn = seqPair $ (all_n, fromList [n+1..n+n])
Swapping that around is(fromList [n+1..n+n], all_n).

So _nn is not the same as _nn_swap. n only identifies the size of the data not the actual data.

[AndreasPK]

What is a data_str?

It's the nn/nt/tn/... identifiers. But I should replace that by size_str I guess given the above.

Benchmark list now looks like this:

difference-block_nn
difference-block_nn_swap
difference-block_ns
difference-block_sn_swap
difference-common_nn
difference-common_nn_swap
difference-common_ns
difference-common_nt
difference-common_sn_swap
difference-common_tn_swap
difference-disj_nn
difference-disj_nn_swap
difference-disj_ns
difference-disj_nt
difference-disj_sn_swap
difference-disj_tn_swap
difference-mix_nn
difference-mix_nn_swap
difference-mix_ns
difference-mix_nt
difference-mix_sn_swap
difference-mix_tn_swap
intersection-block_nn
intersection-block_nn_swap
intersection-block_ns
intersection-block_sn_swap
intersection-common_nn
intersection-common_nn_swap
intersection-common_ns
intersection-common_nt
intersection-common_sn_swap
intersection-common_tn_swap
intersection-disj_nn
intersection-disj_nn_swap
intersection-disj_ns
intersection-disj_nt
intersection-disj_sn_swap
intersection-disj_tn_swap
intersection-mix_nn
intersection-mix_nn_swap
intersection-mix_ns
intersection-mix_nt
intersection-mix_sn_swap
intersection-mix_tn_swap
union-block_nn
union-block_nn_swap
union-block_ns
union-block_sn_swap
union-common_nn
union-common_nn_swap
union-common_ns
union-common_nt
union-common_sn_swap
union-common_tn_swap
union-disj_nn
union-disj_nn_swap
union-disj_ns
union-disj_nt
union-disj_sn_swap
union-disj_tn_swap
union-mix_nn
union-mix_nn_swap
union-mix_ns
union-mix_nt
union-mix_sn_swap
union-mix_tn_swap

[treeowl]

Bleh. OK, could you maybe add a bit more documentation so people can understand what the notation means? Then I'll merge.

[AndreasPK]

As constructed I think the benchmarks could also be bogus when it comes to GC overhead.

Consider this ridiculous example of running benchmarks with a tiny nursery:

Run only two benchmarks so little live data:

$ cabal new-run set-operations-set -- union-disj_nn +RTS -s -A50k
Up to date
benchmarking union-disj_nn ... took 10.95 s, total 201926 iterations

benchmarked union-disj_nn
time                 2.080 us   (902.2 ns .. 3.484 us)
                     0.561 R²   (0.066 R² .. 0.912 R²)
mean                 5.098 us   (4.726 us .. 5.792 us)
std dev              830.2 ns   (516.8 ns .. 1.169 us)
variance introduced by outliers: 58% (severely inflated)

All benchmarks so lots of live data:

cabal new-run set-operations-set -- +RTS -s -A50k
Up to date
benchmarking union-disj_nn ... took 10.93 s, total 201926 iterations

benchmarked union-disj_nn
time                 4.576 us   (2.889 us .. 8.394 us)
                     0.521 R²   (0.204 R² .. 0.855 R²)
mean                 5.374 us   (4.933 us .. 5.974 us)
std dev              850.1 ns   (591.1 ns .. 1.193 us)
variance introduced by outliers: 48% (moderately inflated)

The second example is over twice as slow with these settings!

The difference is less pronounced without the nursery settingings but still very much exists.

It's not a huge issue if you always run all the benchmarks. But if not it's pretty bad.

[AndreasPK]

Bleh. OK, could you maybe add a bit more documentation so people can understand what the notation means? Then I'll merge.

I've added a blurp about the data sizes.

The GC issue is a different one (and not something I want to tackle atm).

[treeowl]

Please kill the GHC_OPTIONS that snuck in.

[AndreasPK]

Please kill the GHC_OPTIONS that snuck in.

done

[treeowl]

Thanks!