[prometheusreceiver] should not validate combined metrics timestamp for type like gauge, counter #12498
Closed as not planned
Closed as not planned
Description
Is your feature request related to a problem? Please describe.
Failed to scrape cadvisor metrics due to timestamp validation of the combined metrics, however container_cpu_load_average_10s
is gauge
2022-07-16T10:15:54.277+0800 debug scrape/scrape.go:1552 Unexpected error {"kind": "receiver", "name": "prometheus", "pipeline": "metrics", "scrape_pool": "kubernetes-cadvisor-metrics", "target": "https://xxx:10250/metrics/cadvisor", "series": "container_cpu_load_average_10s{container=\"\",id=\"/kubepods.slice\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}", "error": "inconsistent timestamps on metric points for metric container_cpu_load_average_10s"}
2022-07-16T10:15:54.277+0800 debug scrape/scrape.go:1328 Append failed {"kind": "receiver", "name": "prometheus", "pipeline": "metrics", "scrape_pool": "kubernetes-cadvisor-metrics", "target": "https://xxx:10250/metrics/cadvisor", "error": "inconsistent timestamps on metric points for metric container_cpu_load_average_10s"}
2022-07-16T10:15:54.277+0800 warn internal/otlp_metricsbuilder.go:164 Failed to scrape Prometheus endpoint {"kind": "receiver", "name": "prometheus", "pipeline": "metrics", "scrape_timestamp": 1657937752544, "target_labels": "map[__name__:up instance:xxx:10250 job:kubernetes-cadvisor-metrics]"}
2022-07-16T10:15:54.278+0800 INFO loggingexporter/logging_exporter.go:57 MetricsExporter {"#metrics": 5}
...
Metric #4
Descriptor:
-> Name: up
-> Description: The scraping was successful
-> Unit:
-> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2022-07-16 02:15:52.544 +0000 UTC
Value: 0.000000
help text for container_cpu_load_average_10s
metric
# HELP container_cpu_load_average_10s Value of container cpu load average over the last 10 seconds.
# TYPE container_cpu_load_average_10s gauge
Describe the solution you'd like
If histogram and summary points are required to have consistent timestamp(if I read #9385 correctly), should the validation be skipped for other metric types?
Describe alternatives you've considered
Additional context
can be reproduced by given such metrics
# HELP container_cpu_load_average_10s Value of container cpu load average over the last 10 seconds.
# TYPE container_cpu_load_average_10s gauge
container_cpu_load_average_10s{container="",id="/",image="",name="",namespace="",pod=""} 0 1658124636389
container_cpu_load_average_10s{container="",id="/kubepods.slice",image="",name="",namespace="",pod=""} 0 1658124637748
container_cpu_load_average_10s{container="",id="/kubepods.slice/kubepods-besteffort.slice",image="",name="",namespace="",pod=""} 0 1658124640330