Out-of-process benchmarks for more stable wall time

[jzombie]

When running the WallTime reporting on dedicated hardware, I'm seeing more fluctuation than I believe that Criterion is actually reporting.

I'm a bit concerned about using the compatibility layer as well, so I started working on a fork that runs Criterion without a compatibility layer and instead ingests the log files directly and sends them to CodSpeed.

If that works, I'm curious why instead of using process wrappers at all (and compatibility layers), why not just ingest all metrics for benchmarks and profiles directly from the disk and upload them to CodSpeed, instead of wrapping them in a process?

[jzombie]

Update: Got baseline working and posting to CodSpeed without using Criterion compatibility adapter.

jzombie#2

Still need to work on test name resolution, but getting it to post was a major hurdle as it was initially difficult to debug errors.

[jzombie]

The number of fluctuations outside of my thresholds went from a dozen to two, so it seems to be more stable than the wrapped process.

What's currently not working is proper source file detection.

I am also curious about seeing if the flamegraph crate can be easily mapped to display profile info in codspeed.