monitoring: Update observable otlp metrics to improve reporting frequency

[isaacaflores2]

APM Server version (apm-server version): 9.0, 8.19

Description of the problem including expected versus actual behavior:
In a prior issue #17104 we noticed that asynchronous (observable) metrics were providing data at the same intervals as other synchronous metrics. This is due to the nature of observable metrics described here.

All asynchronous instruments (Int64ObservableCounter, Int64ObservableUpDownCounter, Int64ObservableGauge, Float64ObservableCounter, Float64ObservableUpDownCounter, and Float64ObservableGauge) are used to measure metrics outside of the execution of source code. They are said to make "observations" via a callback function called once every measurement collection cycle.

This issue track two remaining observable metrics that should be updated: apm-server.otlp.grpc.metrics.consumer.unsupported_dropped and apm-server.otlp.http.metrics.consumer.unsupported_dropped. These metrics should be converted two synchronous metrics to avoid any large time gaps in reporting data.

Steps to reproduce:

TBP

[carsonip]

@isaacaflores2 do you mean that the observable otlp metrics are not affected is solely because there are no naming conflicts? I'm asking because #17512 isn't shipped yet.

[isaacaflores2]

After adding recent changes in #17512 I do not think there is an issue with these observable metrics. The behavior we saw with other observable metrics was most likely caused by the naming conflicts.

Kibana shows this being reported every 1min when I spot checked a couple clusters. So I think we can close this issue. @carsonip please let me know what you think.

[carsonip]

Thanks for confirming. Closing as this is a non-issue.