Stable, meaningful benchmarks

[ClaudiaComito]

This issue is meant to track changes needed to improve benchmarks reproducibility and meaningfulness, now that our benchmarks are running on a dedicated cluster node (thanks @JuanPedroGHM !) and runtime variance has decreased dramatically.

As usual, feel free to add/edit.

• implement interactive dashboard (feat: CB Grafana Dashboard and PR comments #1583)
• distribute GPU processes across all available GPUs
• increase data sizes where meaningful
• add benchmarks for meanwhile added features (TS-SVD Add tall-skinny SVD #1456 , rSVD Distributed randomized SVD #1561 , iSVD Add incremental SVD/PCA #1458 etc.)

[github-actions]

Branch features/1615-Stable_meaningful_benchmarks created!

[mrfh92]

@ClaudiaComito @JuanPedroGHM I think we need some "design decision" here, especially on:

• granularity of the benchmarks, e.g.:
  • do we only benchmark high-level routines (e.g., standard scaler) and leave away the low level routines (here: mean, std, in-place subtraction and division) or do we benchmark both?
  • do we choose a representative case or do we want to try to cover as many cases as possible, e.g.: cover all split combinations or only a representative one (how to determine that?)? include non-split/trivially-parallel routines, too?
• do we aim at (at least for the beginning) more or less equal run times for each benchmark or more or less equal size of the data for each benchmark? or do we pursue a completely different approach for the benchmarks?
• regarding the names: do we want to keep the old names in order to be able to show the old data as well or do we want to start completely from scratch again once we have set up new benchmark sizes/additional benchmarks?