Check perf results is too sensitive - non-regression noise exceeds 2 sigma variation

[Alan-Jowett]

The script uses the default value for max_sigma, which is 2. Consider passing a higher threshold to avoid false positives on regressions.

ebpf-for-windows/.github/workflows/upload-perf-results.yml

Line 105 in 13cadc6

psql -f ./check_perf_results.sql -v platform='${{inputs.platform}}' --csv > ${{github.workspace}}/results/regression_results.csv

https://github.com/microsoft/bpf_performance/blob/492e048521959aa0e679e8779d8cdf743da0a9a1/scripts/check_perf_results.sql#L24

[shankarseal]

We probably should use a more sophisticated statistical tool than 2xsigma.
Possible candidates

• Kolmogorov-Smirnov
• Run multiple times and pick lowest value
• mean over last 3 runs and then check if it breaches 2 sigma
• Use 3 sigma
• Use a threshold (% of mean).