Observable metrics with different attributes reported only once

[dmitryax]

Description

The original issue was reported at open-telemetry/opentelemetry-collector#10444.

Several calls of meter.Int64ObservableGauge submitted for the same metric with different attribute sets result in only one of them being reported, randomly picked.

I found that the collector issue was introduced after the upgrade of the go library v1.22.0 -> v1.23.0 in open-telemetry/opentelemetry-collector#9486.

I suspect the regression was introduced in #4820, but I haven't verified it.

[dmitryax]

Confirmed that the issue is caused by #4820. I'll work on a fix

[dmitryax]

I didn't find a clean way to cache callbacks. I fixed the original issue in the collector by registering callbacks separately open-telemetry/opentelemetry-collector#10550

cc @dashpole

[dashpole]

I believe this is WAI. I've opened #5606 to update comments I missed in the initial PR.

The previous behavior created a memory leak for people using the API, and I don't believe there is a way to address that without caching.

[dashpole]

This could be fixed if OTel had bound instruments, since the attributes would be pre-defined instead of defined inside the callback. We would then be able to cache one callback for each bound (with attributes) instrument.

[dmitryax]

This could be fixed if OTel had bound instruments, since the attributes would be pre-defined instead of defined inside the callback. We would then be able to cache one callback for each bound (with attributes) instrument.

That is how I envisioned it before looking into the existing API.