Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Histogram Aggregator break metric expiration inside Prometheus client output #10043

Closed
opengamer29 opened this issue Nov 2, 2021 · 0 comments · Fixed by #10515
Closed

Histogram Aggregator break metric expiration inside Prometheus client output #10043

opengamer29 opened this issue Nov 2, 2021 · 0 comments · Fixed by #10515
Labels
area/prometheus bug unexpected problem or unintended behavior

Comments

@opengamer29
Copy link
Contributor

opengamer29 commented Nov 2, 2021

Relevent telegraf.conf

[agent]
  interval = "30s"
  round_interval = true
  metric_batch_size = 20000
  metric_buffer_limit = 10000
  collection_jitter = "0s"
  flush_jitter = "0s"
  debug = true
  quiet = false

[[inputs.socket_listener]]
  service_address = "udp://:8125"
  data_format = "influx"

[[outputs.prometheus_client]]
  listen = ":9201"
  expiration_interval = "30s"
  metric_version = 2

[[aggregators.histogram]]
  period = "30s"
  drop_original = true
  reset = false
  cumulative = true

  [[aggregators.histogram.config]]
    measurement_name = "test"
    buckets = [0.0, 100.0, 300.0, 500.0, 700.0, 1000.0, 1500.0]

System info

Telegraf 1.20.0

Docker

No response

Steps to reproduce

  1. Send test ms=1 to udp://:8125
  2. Check /metrics endpoint -> buckets for test metric should be shown
  3. Wait for more that expiration_interval (30 seconds)
  4. Check /metrics endpoint

Expected behavior

There are no metrics inside /metrics endpoint due to its expiration.

Actual behavior

Expired metrics are still presented.

Additional info

It's important for cases described in #10517 It allows to significantly decrease service loading.
Following the code, Histogram Aggregator always pushes metrics with the current timestamp (historgram.go:173).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/prometheus bug unexpected problem or unintended behavior
Projects
None yet
1 participant