How to stop emitting of metric values?

[mat127]

Hello all,

we use the opentelemetry-cpp client to monitor a state of so called "pipelines" in our application. The important thing is that the pipelines can be created and removed dynamically. So we expect the same for the state metric i.e. the metric value should not be present after the "pipeline" removal. Unfortunately we are not able to achieve this.

Our setup is: opentelemetry-cpp -> otel-collector -> mimir

For the pipeline state we use int64_t async gauge.

Pipeline creation:

• add a callback for emitting a state to the gauge
• when callback invoked add following labels to the value
  • app instance id
  • pipeline id

So far everything is ok, after a pipeline creation we can query the "pipeline" state from Mimir.

Pipeline removal:

• remove the callback

But here things get complicated. Even after the callback removal we can still query a "pipeline" state values from Mimir having timestamps AFTER the moment of callback removal. According to the logs the callback is not called any more.

After the termination of the application "pipeline" state values finally disappear i.e. they are not present for timestamps after the termination.

Can someone please explain this behavior? Are we doing something wrong? We would really appreciate any ideas and/or hints.

Petr