Skip to content

[prometheusreceiver] should not validate combined metrics timestamp for type like gauge, counter #12498

Closed as not planned
@newly12

Description

Is your feature request related to a problem? Please describe.
Failed to scrape cadvisor metrics due to timestamp validation of the combined metrics, however container_cpu_load_average_10s is gauge

2022-07-16T10:15:54.277+0800    debug   scrape/scrape.go:1552   Unexpected error        {"kind": "receiver", "name": "prometheus", "pipeline": "metrics", "scrape_pool": "kubernetes-cadvisor-metrics", "target": "https://xxx:10250/metrics/cadvisor", "series": "container_cpu_load_average_10s{container=\"\",id=\"/kubepods.slice\",image=\"\",name=\"\",namespace=\"\",pod=\"\"}", "error": "inconsistent timestamps on metric points for metric container_cpu_load_average_10s"}
2022-07-16T10:15:54.277+0800    debug   scrape/scrape.go:1328   Append failed   {"kind": "receiver", "name": "prometheus", "pipeline": "metrics", "scrape_pool": "kubernetes-cadvisor-metrics", "target": "https://xxx:10250/metrics/cadvisor", "error": "inconsistent timestamps on metric points for metric container_cpu_load_average_10s"}
2022-07-16T10:15:54.277+0800    warn    internal/otlp_metricsbuilder.go:164     Failed to scrape Prometheus endpoint    {"kind": "receiver", "name": "prometheus", "pipeline": "metrics", "scrape_timestamp": 1657937752544, "target_labels": "map[__name__:up instance:xxx:10250 job:kubernetes-cadvisor-metrics]"}
2022-07-16T10:15:54.278+0800    INFO    loggingexporter/logging_exporter.go:57  MetricsExporter {"#metrics": 5}
...
Metric #4
Descriptor:
     -> Name: up
     -> Description: The scraping was successful
     -> Unit: 
     -> DataType: Gauge
NumberDataPoints #0
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2022-07-16 02:15:52.544 +0000 UTC
Value: 0.000000

help text for container_cpu_load_average_10s metric

# HELP container_cpu_load_average_10s Value of container cpu load average over the last 10 seconds.
# TYPE container_cpu_load_average_10s gauge

Describe the solution you'd like
If histogram and summary points are required to have consistent timestamp(if I read #9385 correctly), should the validation be skipped for other metric types?

Describe alternatives you've considered

Additional context
can be reproduced by given such metrics

# HELP container_cpu_load_average_10s Value of container cpu load average over the last 10 seconds.
# TYPE container_cpu_load_average_10s gauge
container_cpu_load_average_10s{container="",id="/",image="",name="",namespace="",pod=""} 0 1658124636389
container_cpu_load_average_10s{container="",id="/kubepods.slice",image="",name="",namespace="",pod=""} 0 1658124637748
container_cpu_load_average_10s{container="",id="/kubepods.slice/kubepods-besteffort.slice",image="",name="",namespace="",pod=""} 0 1658124640330

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions