[RFC] Metrics Framework

[Gaganjuneja]

Is your feature request related to a problem? Please describe.
Feature Request #1061
Feature Request #6533
PR - #8816

Describe the solution you'd like

Problem Statement

The current OpenSearch stats APIs offer valuable insights into the inner workings of each node and the cluster as a whole. However, they lack certain details such as percentiles and do not provide the semantics of richer metric types like histograms. Consequently, identifying outliers becomes challenging. OpenSearch needs comprehensive metric support to effectively monitor the cluster. Recent issues and RFCs have attempted to address this in a piecemeal fashion, and we are currently engaged in a significant effort to instrument OpenSearch code paths. This presents an opportune moment to introduce comprehensive metrics support.

Tenets

1. Minimal Overhead – Metrics should not impose minimal overhead on system resources such as CPU and memory.
2. No performance impact – Metrics should not adversely affect cluster operations in terms of performance.
3. Extensible – The metrics framework should be easily extendable to accommodate background tasks, plugins, and extensions.
4. Safety - The framework must ensure that instrumentation code does not result in memory leaks.
5. Well defined Abstractions – Metrics frameworks should offer clear abstractions so that changes in implementation framework or API contracts do not disrupt implementing classes.
6. Flexible – There should be a mechanism for dynamically enabling/disabling metrics through configurable settings.
7. Do not reinvent - We should prefer to use out of the box solutions available instead of building something from scratch.

Metrics Framework

It is widely recognized that observability components like tracing and metrics introduce overhead. Therefore, designing the Metrics Framework for OpenSearch requires careful consideration. This framework will provide abstractions, governance, and utilities to enable developers and users to easily utilize it for emitting metrics. Let's delve deeper into these aspects: –

1. Abstractions - While metrics frameworks like OpenTelemetry can be leveraged, we will abstract the solution behind the OpenSearch APIs. This future-proofs the core OpenSearch code where metrics are added, reducing the need for changes. Hooks for metrics have already been included in the Telemetry abstraction, following a similar pattern as the TracingTelemetry implementation.
2. Governance - Similar to tracing, we should define mechanisms for enabling and disabling metrics at multiple levels.
3. Code Pollution - To mitigate code pollution, we should provide utilities such as SpanBuilder to abstract away repetitive boilerplate code.

HLD

1. Metric APIs - The Metrics API will facilitate the creation and updating of metrics. It should handle most of the heavy lifting and abstract the need for boilerplate code.

public interface Meter extends Closeable {

    /**
     * Creates the counter. This counter can increase monotonically.
     * @param name name of the counter.
     * @param description any description about the metric.
     * @param unit unit of the metric.
     * @return counter instrument.
     */
    Counter createCounter(String name, String description, String unit);

    /**
     * Creates the up/down counter. Value for this counter may go up and down so should be used
     * at places where negative, postive and zero values are possible.
     * @param name name of the counter.
     * @param description any description about the metric.
     * @param unit unit of the metric.
     * @return up/down counter instrument.
     */
    Counter createUpDownCounter(String name, String description, String unit);

    /**
     * Creates the histogram instrument which is needed if some values needs to be recorded against
     * some buckets, samples, etc.
     * @param name name of the counter.
     * @param description any description about the metric.
     * @param unit unit of the metric.
     * @return histogram instrument..
     */
    Histogram createHistogram(String name, String description, String unit);

    /**
     * Creates the Gauge which helps in recording the arbitrary/absolute values like cpu time, memory usage etc.
     * @param name name of the counter.
     * @param description any description about the metric.
     * @param unit unit of the metric.
     * @return counter.
     */
    Gauge createGauge(String name, String description, String unit);
}

2. Storage - Metrics data need not be emitted with each event or request. Instead, they should be stored or buffered (like async loggers) and aggregated over a configurable time frame in an in-memory store before periodic emission.
3. Sink - Periodically emitted metrics should be written to a configurable sink. Users should have the flexibility to define their preferred sink.

Implementation

OpenTelemetry offers decent support for metrics, and we can leverage the existing telemetry-otel plugin to provide the implementation for metrics as well.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[Gaganjuneja]

@reta @Bukhtawar @shwetathareja @backslasht @suranjay @rishabh6788 @nknize @dblock Please provide your inputs.

[msfroh]

Does this give us the opportunity to collect a bunch of counters related to a single end-to-end execution of some "operation"? As a start, will we be able to get an implementation that logs a bunch of counters for every HTTP request, so we'll be able to do out-of-process aggregation / computation of percentiles?

The current stats-based approach that uses global (or per-node, per-index, per-shard) counters obscures a lot of useful information. For example, maybe I see stats showing that we're spending a lot of time doing thing A and a lot of time doing thing B. Are we spending a lot of time doing A and B in the same requests? Are some requests spending time on thing A, while other requests are spending time on thing B? Was there only once request that spent a really long time doing thing B?

Request-level collection of counters would be extremely useful.

[reta]

@Gaganjuneja thanks for putting this one up

public interface Meter extends Closeable {

I believe we should rename it to MeterRegistry or MetricRegistry since Meter is usually an individual metric. But that is minor, what worries me a lot is the amount of the abstractions we will have to introduce to facade the different metric providers (Otel, Dropwziard, Micrometer, ....), I believe it is going to be massive.

[Gaganjuneja]

Does this give us the opportunity to collect a bunch of counters related to a single end-to-end execution of some "operation"? As a start, will we be able to get an implementation that logs a bunch of counters for every HTTP request, so we'll be able to do out-of-process aggregation / computation of percentiles?

The current stats-based approach that uses global (or per-node, per-index, per-shard) counters obscures a lot of useful information. For example, maybe I see stats showing that we're spending a lot of time doing thing A and a lot of time doing thing B. Are we spending a lot of time doing A and B in the same requests? Are some requests spending time on thing A, while other requests are spending time on thing B? Was there only once request that spent a really long time doing thing B?

Request-level collection of counters would be extremely useful.

Yes, we can generate the Request level counters and connect the traces with metrics to see both aggregated and request level view.

[reta]

As a start, will we be able to get an implementation that logs a bunch of counters for every HTTP request, so we'll be able to do out-of-process aggregation / computation of percentiles?

@msfroh could you give an example of what kind of counters for every HTTP request you are looking for?

For example, maybe I see stats showing that we're spending a lot of time doing thing A and a lot of time doing thing B

I believe with traces you should naturally get that, since A and B should be denoted by span or span hierarchy.

Yes, we can generate the Request level counters and connect the traces with metrics to see both aggregated and request level view.

@Gaganjuneja To be fair, I really doubt that we could (and even should) do that, however we could add events + attributes to the spans and than query / aggregate over them as need.

[Gaganjuneja]

@Gaganjuneja To be fair, I really doubt that we could (and even should) do that, however we could add events + attributes to the spans and than query / aggregate over them as need.

Yes, we can do that only challenge comes here is with sampling.

[Gaganjuneja]

@Gaganjuneja thanks for putting this one up

public interface Meter extends Closeable {

I believe we should rename it to MeterRegistry or MetricRegistry since Meter is usually an individual metric. But that is minor, what worries me a lot is the amount of the abstractions we will have to introduce to facade the different metric providers (Otel, Dropwziard, Micrometer, ....), I believe it is going to be massive.

I totally agree with you @reta here. We can start small and keep on adding the instruments based on the requirements. Like for now we can just start with counter and keep in facade simple as shown in the interface. I understand the above interface is motivated from the Otel schema but we should be able to implement it for most of the metering solutions.

[msfroh]

@msfroh could you give an example of what kind of counters for every HTTP request you are looking for?

You bet! If I could have my every per-request metrics record wish granted, it would include things like (in priority order):

1. Overall request latency
2. Counters related to the search request (query clause count, 0/1 counter on presence of various features -- different aggregations, collapse, rescore, etc, request source size in bytes).
3. Counters related to the search response (total hits, hits returned, number of aggregation buckets in the response, response size in bytes).
4. A 0/1 failure counter.
5. Latency of each search phase.
6. 0/1 failure counters for each search phase.
7. Shard success/failed counters for each search phase.
8. Total CPU cost of the request.
9. CPU cost broken down by search phase, per shard.

And of course, I would love to be able to correlate all of this back to the original query (not necessarily from a time series system, since I would expect that to store aggregate histograms of counters, but I should be able to fetch the raw logs that were aggregated for ingestion into the time series data store).

[reta]

1. Overall request latency
2. Latency of each search phase.

This is the goal of tracing - you get out of the box

2. Counters related to the search request (query clause count, 0/1 counter on presence of various features -- different aggregations, collapse, rescore, etc, request source size in bytes).
3. Counters related to the search response (total hits, hits returned, number of aggregation buckets in the response, response size in bytes).
4. A 0/1 failure counter.

Those could be attach to each span at the relevant subflow

8. Total CPU cost of the request.
9. CPU cost broken down by search phase, per shard.

This could extremely difficult and expensive to track but we do try to track the CPU cost of search task per shard, that could be attached to task spans as well.

[msfroh]

@reta -- so all my wishes can potentially be granted? Can I have a pony too? 😁

[reta]

haha @msfroh one more

And of course, I would love to be able to correlate all of this back to the original query (not necessarily from a time series system, since I would expect that to store aggregate histograms of counters, but I should be able to fetch the raw logs that were aggregated for ingestion into the time series data store).

Trace ids could be (and should be) in logs so it should be possible to correlate all logs for specific trace and request

[Gaganjuneja]

1. Overall request latency
2. Latency of each search phase.

This is the goal of tracing - you get out of the box

2. Counters related to the search request (query clause count, 0/1 counter on presence of various features -- different aggregations, collapse, rescore, etc, request source size in bytes).
3. Counters related to the search response (total hits, hits returned, number of aggregation buckets in the response, response size in bytes).
4. A 0/1 failure counter.

Those could be attach to each span at the relevant subflow

8. Total CPU cost of the request.
9. CPU cost broken down by search phase, per shard.

This could extremely difficult and expensive to track but we do try to track the CPU cost of search task per shard, that could be attached to task spans as well.

This information is already available as part of TaskResourceTracking framework which can be reused. Not sure if this should be attached as an event or attribute to the span.

Only use case where we need the metric and trace integration is - Let's say we have a one metric graph for HttpStatusCode and it shows peek for statusCode=500 now we may want to see all the requests failed during this peek Now,

1. If a metric has traceIDs associated, it can immediately pull those traces and we should be able to figure out the reason hopefully.
2. We can query the traces for that time range and with httpStatuCode=500 and do the evaluation.

So, I agree we should be able to handle the storage side instead of integrating in the server because the sampling issue still persists there as well.