[testbed] Make load tests useful

[dmitryax]

Is your feature request related to a problem? Please describe.

Our load tests with fixed CPU/RAM limits have become ineffective over time. This is because the CPU/RAM utilization fluctuates during different test runs due to the unstable testing environment. We increase the limits whenever the load tests fail due to this instability. Unfortunately, this approach has led to a situation where significant changes in the CPU/RAM utilization are not detected as happened in #19798

Describe the solution you'd like

1. We need to store all the results from the load tests in a central location where they can be easily accessed via API and UI. This will enable us to analyze the performance trends over time, specifically between the commits in the main branch.
2. To improve the load testing process, we should replace the fixed CPU/RAM limits with a rolling average value computed over a specified number of recent commits. The load test should fail if the CPU/RAM utilization exceeds the rolling average value plus a margin of error. The margin of error should be set according to the level of instability in the testing environment. Hopefully, a margin of 30% will be enough for our testing environment.

[codeboten]

See related issue #9121

[dmitryax]

Sorry, missed that issue. I believe we can keep this one for the second step to replace fixed limits in the load tests

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.