Add ability to flush metrics on demand from push controller

[nabam]

Right now push controller is only able to flush metrics on timer. It would be useful to have a handle to force flush metrics. Such handle could be invoked before application termination to not lose measurements.

opentelemetry-go/sdk/metric/controller/push/push.go

Line 136 in 2c2f985

func (c *Controller) tick() {

[MrAlias]

Seems reasonable to me. @jmacd thoughts?

[nabam]

I can make a PR if you give me a green light

[jmacd]

Sounds good.
I'd like to merge #1378 first, and then it will be a simpler change on top of the new combined controller.

[jmacd]

I think this can be closed. The push controller already flushes metrics on Stop(), and after #1378 it will be possible to stop, manually collect, and then start again. I'll let this close with #1378 unless anyone objects.

[nabam]

@jmacd for monitoring of serverless functions do you think it's a good pattern to start and stop controller on every function invocation?

[jmacd]

That's probably not a good pattern. My understanding is that the ideal for serverless is to use an agent that can flush in the background. If you don't have an agent though, Start() and Stop() are correct. I'm probably not the best person to ask serverless questions. :-)

[nabam]

I have instrumented lambdas with otlp exporter using delta aggregations. It works decently, metrics are flushed in the background while functions are running. There are two caveats though: if lambdas are not called often enough aggregated measurements that were not exported before function is suspended are not exported till next invocation and obviously when lambda instance terminates measurements done before termination are lost. Being able to force export at the end on function invocations would resolve both. I don't see how adding lambda extension similar to that one will help, function will still need to export measurements during invocation. Start-Stop on every invocation doesn't looks like a good pattern to me even though it should work.

[jmacd]

Calling Stop() on the controller is the correct way to flush at export. That's always been the case, and #1378 when it merges makes it possible to repeatedly start and stop the controller, although I'm not sure what reasons there are for doing that.

We could add an explicit flush method, but I still think Stop() should flush before stopping. If there was an explicit flush method, it's just going to add complexity when there is a background timer flushing regularly, so I'd rather not.

open-telemetry/opentelemetry-specification#1297 talks about the algorithm a collector will use to correctly combine metric streams in an agent. I'd like to get this implemented in the collector, I think it'll be a win for use from the lambda extension.

[nabam]

I don't have strong opinions here, stop-start to flush metrics should work. Would be great to hear from someone who has a similar use-case.

[tfendt-belong-gg]

I am running in to this issue. @nabam What is the proposed solution exactly? After reading over the ADOT docs and implementing the lambda layer any metrics collected and not flushed before the lambda terminates are lost. I've had to set the interval really low to make sure metrics are flushed to the collector before the lambda stops executing.

  const { MeterProvider } = require('@opentelemetry/metrics');
  const { CollectorMetricExporter } =  require('@opentelemetry/exporter-collector-grpc');
  const exporter = new CollectorMetricExporter();
  // Register the exporter
  const meter = new MeterProvider({
    exporter,
    interval: 10,
  }).getMeter('test-meter');