metric adjusters in prometheus receiver should have a better fallback strategy.

[h0cheung]

Component(s)

receiver/prometheus

Is your feature request related to a problem? Please describe.

There are two metric adjusters in the Prometheus receiver:

• initialPointAdjuster, used by default. It will use the timestamp of the first sample as start_timestamp. There is a function gate receiver.prometheusreceiver.UseCreatedMetric, which, if enabled, will use *_created so that the timestamp of the first sample will be used as a backup.
• startTimeMetricAdjuster, used when use_start_time_metric is set to true. It will take a metric representing the start time of the process and use it as the start_timestamp. If the metric is missing, an error is returned and all metrics are discarded.

It seems odd that using *_created will fall back to using the first sample, but using the process start time will not fallback to anything.

I have a number of targets to scrape, some of which have *_created metrics, others don't have *_created but may have process_start_time_seconds. There is no way to optimally handle all these metrics with a unified configuration.

Describe the solution you'd like

Allow to config a strategy like this:

• At first, try *_created.
• If no *_created found, try process_start_time_seconds. (can be customized by setting start_time_metric_regex)
• Finnaly, use the timestamp of the first sample.

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

I'm supportive of that change. If you are interested, feel free to work on a PoC to see how much refactoring, etc. is needed to make that work