Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a new config and metric for reporting ruler query execution wall time. #4317

Merged
merged 16 commits into from
Jul 9, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Use seconds for our duration rather than nanoseconds
Signed-off-by: Tyler Reid <tyler.reid@grafana.com>
  • Loading branch information
Tyler Reid committed Jun 24, 2021
commit 77fa0f01a5bad2e9cd42985bfc533181c9236f3d
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@
* Ensure that a ring store is configured using `-alertmanager.sharding-ring.store`, and set the flags relevant to the chosen store type.
* Enable the feature using `-alertmanager.sharding-enabled`.
* Note the prior addition of a new configuration option `-alertmanager.persist-interval`. This sets the interval between persisting the current alertmanager state (notification log and silences) to object storage. See the [configuration file reference](https://cortexmetrics.io/docs/configuration/configuration-file/#alertmanager_config) for more information.
* [FEATURE] Ruler: Add new `-ruler.enable-query-stats` which when enabled will report the `cortex_ruler_query_seconds_total` metric that tracks the sum of the wall time of executing queries in the ruler. #4317
* [FEATURE] Ruler: Add new `-ruler.enable-query-stats` which when enabled will report the `cortex_ruler_query_seconds_total` metric that tracks the sum of the wall time of executing queries in the ruler in seconds. #4317
treid314 marked this conversation as resolved.
Show resolved Hide resolved
* [ENHANCEMENT] Alertmanager: Cleanup persisted state objects from remote storage when a tenant configuration is deleted. #4167
* [ENHANCEMENT] Storage: Added the ability to disable Open Census within GCS client (e.g `-gcs.enable-opencensus=false`). #4219
* [ENHANCEMENT] Etcd: Added username and password to etcd config. #4205
Expand Down
2 changes: 1 addition & 1 deletion pkg/ruler/compat.go
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ func MetricsQueryFunc(qf rules.QueryFunc, queries, failedQueries prometheus.Coun
// If we've been passed a counter vec we want to record the wall time spent executing this request.
treid314 marked this conversation as resolved.
Show resolved Hide resolved
if queryTime != nil {
treid314 marked this conversation as resolved.
Show resolved Hide resolved
startTime = time.Now()
defer func() { queryTime.WithLabelValues(userID).Add(float64(time.Since(startTime))) }()
defer func() { queryTime.WithLabelValues(userID).Add(time.Since(startTime).Seconds()) }()
}

result, err := qf(ctx, qs, t)
Expand Down
4 changes: 2 additions & 2 deletions pkg/ruler/compat_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -248,7 +248,7 @@ func TestMetricsQueryFuncMetrics(t *testing.T) {
queryTime := prometheus.NewCounterVec(prometheus.CounterOpts{}, []string{"user"})

mockFunc := func(ctx context.Context, q string, t time.Time) (promql.Vector, error) {
time.Sleep(1 * time.Millisecond)
time.Sleep(1 * time.Second)
return promql.Vector{}, nil
}
qf := MetricsQueryFunc(mockFunc, queries, failures, queryTime, "userID")
Expand All @@ -257,5 +257,5 @@ func TestMetricsQueryFuncMetrics(t *testing.T) {

require.Equal(t, 1, int(testutil.ToFloat64(queries)))
require.Equal(t, 0, int(testutil.ToFloat64(failures)))
require.LessOrEqual(t, float64(1*time.Millisecond), testutil.ToFloat64(queryTime.WithLabelValues("userID")))
require.LessOrEqual(t, float64(1), testutil.ToFloat64(queryTime.WithLabelValues("userID")))
treid314 marked this conversation as resolved.
Show resolved Hide resolved
}