Observable metric instruments

[iRevive]

OpenTelemetry specification

According to the specification, only Counter, Gauge, and UpDownCounter can be built with a callback.

Note: Gauge can be built exclusively with a callback.

---

An observable metric is an interesting OpenTelemetry concept that provides a handy way to collect metrics via a callback.
The callback is invoked when the collector requests the data. This is useful when you cannot instrument the object but can still collect metrics via getter methods.

Here are a few examples:

1. Fork Join Pool
2. Hikari Pool

For example, we can collect CE starvation metrics:

import java.lang.management.ManagementFactory
import javax.management.{MBeanServer, ObjectName}

val mbeanServer: MBeanServer = ManagementFactory.getPlatformMBeanServer
val mbeanName = new ObjectName("cats.effect.metrics:type=CpuStarvation")
  
val counter: F[ObservableCounter[F, Long]] = Meter[F]
  .observableCounter("cats-effect-runtime-cpu-starvation-count")
  .withDescription("CE runtime starvation count")
  .create(Sync[F].delay(mbeanServer.getAttribute(mbeanName, "CpuStarvationCount").asInstanceOf[Long]))

---

We can implement this in the following way:

1) Add new observableCounter, observableUpDown methods to Meter trait

trait Meter[F[_]] {
  def counter(name: String): SyncInstrumentBuilder[F, Counter[F, Long]]
  def histogram(name: String): SyncInstrumentBuilder[F, Histogram[F, Double]]
  def upDownCounter(name: String): SyncInstrumentBuilder[F, UpDownCounter[F, Long]]
+  def observableCounter(name: String): ObservableInstrumentBuilder[F, Long, ObservableCounter]
+  def observableUpDownCounter(name: String): ObservableInstrumentBuilder[F, Long, ObservableUpDownCounter]
}

2) Update ObservableInstrumentBuilder

trait ObservableInstrumentBuilder[F[_], Input, Instrument[_[_], _]] {
  type Self <: ObservableInstrumentBuilder[F, Input, Instrument]

  def withUnit(unit: String): Self
  def withDescription(description: String): Self
  def create(callback: F[Input]): F[Instrument[F, Input]]
}

---

Questions:
1) How do we call these instruments in the project? The specification uses asynchronous while Java's OpenTelemetry implementation uses observable.

[typo edits by @rossabaker]

[iRevive]

@armanbilge once we agree on how to implement this, it's definitely a good-first-issue.

[iRevive]

@rossabaker @zmccoy @armanbilge what's your opinion there?

[rossabaker]

The spec says:

It is highly recommended that implementations use the name ObservableCounter (or any language idiomatic variation, e.g. observable_counter) unless there is a strong reason not to do so.

I don't love that they introduced a second name, but I don't see any reason to diverge.