roachtest: add post-process step for perf metrics #138617
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
sambhav-jain-16
force-pushed
the
post-process-metrics
branch
7 times, most recently
from
8e65484 to
c16d5a6
Compare
sambhav-jain-16
force-pushed
the
post-process-metrics
branch
2 times, most recently
from
0045d82 to
62cea99
Compare
![blathers[crl]](https://avatars.githubusercontent.com/in/59700?s=40&u=373d9101a12edc3847d2fb6ae78e3adcfb4ca41e&v=4){kind=avatar}
|
Your pull request contains more than 1000 changes. It is strongly encouraged to split big PRs into smaller chunks. 🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf. |
sambhav-jain-16
force-pushed
the
post-process-metrics
branch
5 times, most recently
from
81f6b0e to
7bc5033
Compare
sambhav-jain-16
force-pushed
the
post-process-metrics
branch
11 times, most recently
from
0a204f5 to
5b23f95
Compare
sambhav-jain-16
force-pushed
the
post-process-metrics
branch
2 times, most recently
from
c765b82 to
ba12c23
Compare
pkg/cmd/roachtest/test_runner.go
Outdated
| @@ -1409,6 +1412,13 @@ func (r *testRunner) runTest( | |||
| if err := r.postTestAssertions(ctx, t, c, 10*time.Minute); err != nil { | |||
| l.Printf("error during post test assertions: %v; see test-post-assertions.log for details", err) | |||
| } | |||
|
|
|||
| if t.spec.Benchmark && t.ExportOpenmetrics() { | |||
| if err := r.postProcessPerfMetrics(ctx, t, c); err != nil { | |||
There was a problem hiding this comment.
From my understanding this downloads the metrics, does post processing and then re-uploads the metrics to the node. And then eventually on:
line 971:
if t.spec.Benchmark {
getPerfArtifacts(ctx, c, t)
}
The perf artifacts (metrics) will get downloaded again?
For my own understanding, is there a reason we don't do the post processing on or after getPerfArtifacts(ctx, c, t) which won't require a re-upload step?
There was a problem hiding this comment.
That's a good point.
Let me take a look and refactor my changes into the getPerfArtifacts function you mentioned.
Thanks for pointing out !
sambhav-jain-16
force-pushed
the
post-process-metrics
branch
2 times, most recently
from
6a81967 to
7a6a072
Compare
sambhav-jain-16
force-pushed
the
post-process-metrics
branch
2 times, most recently
from
529b512 to
e824e9b
Compare
There are many tests in which had some custom post processing done in roachperf to emit the metrics. To add this facility in the roachtests itself, this change aims to : 1.Adds 2 new user facing structs `HistogramMetric`, that gives the user access to the histogram metrics generated by the workload and `PerfMetrics` which user can return with their aggregated metric. 2. Introduce a new `PostProcessPerfMetrics(HistogramMetrics) PerfMetrics` step in test spec where the test writer can do custom aggregation and provide the aggregated metric. If the test has not implemented this step, the runner will use the `DefaultPostProcess` function. 3. The step creates a new openmetrics file with the name `aggregated_stats.om` that will have the aggregated metrics. 4. Implement `PostProcessPerfMetrics` in some roachtests that do post processing in roachperf. Epic: https://cockroachlabs.atlassian.net/browse/CRDB-41852 Release note: None
sambhav-jain-16
force-pushed
the
post-process-metrics
branch
from
e824e9b to
954280a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There are many tests in which had some custom post processing done in
roachperfto emit the metrics.To add this facility in the roachtests itself, this change aims to :
HistogramMetric, that gives the useraccess to the histogram metrics generated by the workload and
AggregatedPerfMetricswhich user can return with their aggregated metric.PostProcessPerfMetrics(HistogramMetrics) AggregatedPerfMetricsstep in test spec where the test writer can do customaggregation and provide the aggregated metric. If the test has not
implemented this step, the runner will use the
DefaultPostProcessfunction.
aggregated_stats.omthat will have the aggregated metrics.PostProcessPerfMetricsin some roachtests that dopost processing in roachperf. The implementation is borrowed from
roachperffor each test.There are some
roachtestsfor whichPostProcessPerfMetricsis not implemented. It will be done in subsequent PRsEpic: https://cockroachlabs.atlassian.net/browse/CRDB-41852
Release note: None