[receiver/expvar] Add ability to collect internal memstats

[jamesmoessis]

Component(s)

receiver/expvar

Is your feature request related to a problem? Please describe.

Current the expvarreceiver has the ability to query an HTTP endpoint to gather memstats. However, it doesn't have the ability to query the actual process the collector is running on, and emit that (unless of course, the user has started an expvar server in the same process and pointed the expvarreceiver at it, but this requires code changes from the user).

Having memstats about the collector is useful especially in high-memory usage scenarios.

Describe the solution you'd like

Currently the configuration looks like:

receivers:
  expvar:
    endpoint: "http://localhost:8000/custom/path"
    timeout: 1s
    collection_interval: 30s
    metrics:
      process.runtime.memstats.total_alloc:
        enabled: true
      process.runtime.memstats.mallocs:
        enabled: false

We could add another field similar to metrics, called internal_metrics, which has the same type but collects the metrics from internally (using the runtime package). Disabled by default, but optionally enabled.

Describe alternatives you've considered

There's two other ways I thought of doing this:

1. an expvarextension that starts a server and serves the collector's internal memstats. Would work, is a lot more code and IMO overkill.
2. Compiling my own collector, running the expvar server manually from that, before starting the collector.

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/expvar: @jamesmoessis @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[MovieStoreGuy]

Not a bad idea, though, might be best for the as an internal "telemetry" component of the collector itself instead of making directly part of the expvar receiver.

Would be a better question @open-telemetry/collector-maintainers .

[jamesmoessis]

@MovieStoreGuy this is a good point, it could be part of the internal telemetry of the collector itself. I'd be interested to see what the maintainers would have to say about that.

[dmitryax]

Collector service emits some subset of similar internal metrics using Go API https://github.com/open-telemetry/opentelemetry-collector/blob/main/service/documentation.md#internal-telemetry. What other metrics specifically are you looking for? Maybe we can extend it.

[jamesmoessis]

@dmitryax I think that makes sense, I forgot about those internal metrics actually. I will see if we can cope with those for now, might be nice to get some GC metrics. We are about to run a high-memory workload so I'll update here if we want additional memstats. Will close this for now.

[jamesmoessis]

@dmitryax we have come across some scenarios where we want:

• HeapIdle
• HeapReleased
• NextGC
• PauseTotalNs

This is to keep track of GC and how it's performing. I'd be happy to add this feature to the expvarreceiver or to add it to collector's internal metrics. What is the best way forward?