Metrics SDK: allow metric readers to filter Meters during Collect()

[marcalff]

What are you trying to achieve?

For my application, I need to export metrics to an endpoint at different frequencies.

To do so, several PeriodicMetricReaders are defined, for example:

• Reader A exports data every 15 seconds
• Reader B exports data every 15 minutes

The meters to export should be different, as the main motivation to have different export frequencies is that:

• some meters and metrics change rapidly, and/or are not expensive to evaluate (high frequency is desired)
• other meters and metrics change slowly, and/or are expensive to evaluate (low frequency is desired)

When each reader collects data, every Meter associated with the global MeterProvider is considered.

This is an issue, because the desired behavior is to:

• export one set of meters with reader A
• export another set of meters with reader B

What did you expect to see?

When a reader invokes Collect() on a MeterProvider, I would like to be able to filter out which Meter is considered, instead of exporting every meter defined in the MeterProvider.

The filter should be evaluated as a MeterPredicate(), so the SDK can provide its own logic to select (or not) a Meter.

At the very minimum, the predicate should take the meter name as input, so that selecting meters based on a naming convention is possible.

A very desirable feature is to allow the instrumentation interface to add key/value pairs when creating a Meter, so that metadata associated with a meter is available to the filter predicate.

Work around.

Because this is currently not possible based on the existing spec, and the implementation used (opentelemetry-cpp), the work around currently used in the application is to:

• define a MeterProvider A
• define a MeterProvider B
• have the instrumented code defines Meters on the proper MeterProvider (A or B)
• have the SDK use MeterProvider A in Reader A
• have the SDK use MeterProvider B in Reader B

so that each reader only sees a subset of meters.

This is highly undesirable, because:

• This totally defeats the purpose of the MeterProvider singleton
• The instrumentation is now coupled (by deciding which provider to use to create a meter) with the SDK configuration
• This only works in monolithic applications, where the same vendor controls both the instrumentation and the SDK configuration
• When the configuration changes, the instrumented code needs to change as well (to create a metric on the proper meter provider). This totally defeats the purpose of a yaml configuration.

This work around is the only solution I could come up with given the current behavior.

Additional context.

Add any other context about the problem here. If you followed an existing documentation, please share the link to it.

[marcalff]

PS:

I am aware of this related issue:

• Proposal: Add filter predicate to MetricReader (push-down predicate) #3324

which is not a duplicate.

#3617 is about adding predicates to filter out Meters as a whole,
while #3324 is about adding predicates to filter out Instruments individually.

I see the two as complementary.

The rationale for filtering a Meter is that,
for instrumentations that organize meter and metrics into coherent groups,
the filtering is more efficient.

For example, an instrumented library could:

• expose meter(s) that defines several operating system related metrics
• expose meter(s) that defines several business related metrics
• etc

A reader may want to export all operating system metrics to one endpoint,
while a different reader may want to export all business related metrics to a different endpoint.

Filtering on a meter by meter basis is more efficient in terms of overhead,
because every instrument provided by a given meter does not need to be considered if the SDK decides a meter is to be filtered out.

Expressing the predicate (either by allowing only some meters by name, or by looking at meters metadata)
is expected to be easier compared to expressing predicates on instruments.

[asafm]

So you basically want to add an operation to MetricFilter called TestInstrumentationScope() ?

[svrnm]

hey @marcalff, do you still want to pursue this issue?

[marcalff]

hey @marcalff, do you still want to pursue this issue?

Yes, this is still relevant, and is still blocking in my application.

[marcalff]

Blocks implementation of:

• [Feature] Metrics SDK: allow metric readers to filter Meters during Collect() opentelemetry-cpp#2709

[jack-berg]

Discussed in 7/31/24 TC meeting. We think its a useful thing to pursue and adjusted the status to triage:accepted:needs-sponsor accordingly. One implementation option is to extend views to be configurable at the reader level in addition to the meter provider level. We noted that this issue is related to but not necessarily dependent on #3616. Additionally, this may be a duplicate of #1432, and if so we should consolidate.