feat: improve the latency and throughput formatting in the result table #233

pallosp · 2025-01-28T16:26:09Z

Make the result table a bit more compact, as per the discussion under issue #231

Reduce the number of latency and throughput digits, because the least significant ones were just noise.
Drop the ± part from the medians. They indicated the distance between the two middle elements when there was even number of them, which isn't particularly useful.
Output the medians as numbers to get rid of the quotes.

Example result table for simple benchmark gc:

┌─────────┬───────────────┬──────────────────────┬─────────────────────┬────────────────────────────┬───────────────────────────┬─────────┐
│ (index) │ Task name     │ Latency average (ns) │ Latency median (ns) │ Throughput average (ops/s) │ Throughput median (ops/s) │ Samples │
├─────────┼───────────────┼──────────────────────┼─────────────────────┼────────────────────────────┼───────────────────────────┼─────────┤
│ 0       │ 'faster task' │ '3578.7 ± 2.83%'     │ 3125                │ '314227 ± 0.18%'           │ 320000                    │ 27944   │
│ 1       │ 'slower task' │ '1164166 ± 3.01%'    │ 1192251             │ '1321 ± 67.29%'            │ 839                       │ 86      │
└─────────┴───────────────┴──────────────────────┴─────────────────────┴────────────────────────────┴───────────────────────────┴─────────┘

jerome-benoit · 2025-01-28T16:37:48Z

src/bench.ts

              // eslint-disable-next-line @typescript-eslint/no-non-null-assertion
-              'Latency median (ns)': `${mToNs(task.result.latency.p50!).toFixed(2)}${Number.parseFloat(mToNs(task.result.latency.mad!).toFixed(2)) > 0 ? ` \xb1 ${mToNs(task.result.latency.mad!).toFixed(2)}` : ''}`,


Please keep the threshold condition here : when the median absolute deviation is upper zero, it's a sign of statistical significance issue during the benchmark and a valuable information.

You can use the same helper to display it.

I'm not sure whether I get correctly what the absolute deviation is supposed to measure.

So far I've only seen it in the table when the number of samples was even. If the median for even number of samples is problematic, shouldn't the task be executed one more time instead of reporting a problem?

It's totally unrelated to the number of samples. It's a robust statistical indicator of the statistical significance of the median value. And the only one for median. Like the standard deviation (or the absolute mean deviation) is for the average.

Adding a threshold to it is only way to reduce the information to display.

Thanks for the explanation. After some debugging I understood what's going on. There is a bug in absoluteDeviation that makes mad exactly zero as opposed to a small positive number in most cases. I filed #234.

jerome-benoit · 2025-01-28T16:40:11Z

src/bench.ts

@@ -254,12 +255,12 @@ export class Bench extends EventTarget {
            }
          : (convert?.(task) ?? {
              'Task name': task.name,
-              'Latency average (ns)': `${mToNs(task.result.latency.mean).toFixed(2)} \xb1 ${task.result.latency.rme.toFixed(2)}%`,
+              'Latency average (ns)': `${formatNumber(mToNs(task.result.latency.mean), 5, 2)} \xb1 ${task.result.latency.rme.toFixed(2)}%`,


I think you can also use the formaNumber helper for the standard deviation.

Displaying number only in the column would be better, but I do not think it's possible because of string only characters usage.

For percentages less than 1000% the helper returns the same string as toFixed(2). I slightly prefer keeping toFixed(2) because of its simpler semantics.

pkg-pr-new · 2025-01-28T16:46:31Z

Open in Stackblitz

npm i https://pkg.pr.new/tinylibs/tinybench@233

commit: 392af32

src/bench.ts

test/index.test.ts

feat: improve the latency and throughput formatting

775d0d1

jerome-benoit reviewed Jan 28, 2025

View reviewed changes

jerome-benoit requested changes Jan 28, 2025

View reviewed changes

feat: improve the latency and throughput formatting

392af32