Description
I use telegraf to relay metrics to kafka. Was previously running 1.11.2. After upgrading to 1.13.1, I noticed the metric buffers filling up on my telegraf instances. It appears to be due to invalid timestamps that are received to telegraf.
Relevant telegraf.conf:
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 100000
metric_buffer_limit = 1000000
collection_jitter = "5s"
flush_interval = "10s"
flush_jitter = "5s"
precision = "1ns"
debug = false
quiet = false
logfile = "/var/log/telegraf/telegraf.log"
hostname = ""
omit_hostname = true
[[inputs.influxdb_listener]]
service_address = ":8094"
read_timeout = "60s"
write_timeout = "60s"
max_body_size = 0
max_line_size = 0
[[outputs.kafka]]
brokers = ["host1:6667","host2:6667","host3:6667","host4:6667"]
topic = "topicname"
version = "0.10.2.1"
routing_key = "random"System info:
Ubuntu 16
Telegraf Version 1.13.1
Steps to reproduce:
Upgrade to 1.13.1
Send a metric similar to the below line.
telegraf_metric,component=metrics row_count=157796669i -7240509273709551616
Expected behavior:
The metric should get dropped or we should have the option to write metrics into kafka excluding the timestamp, such as previous versions, which wrote the message to the topic regardless of the timestamp.
Actual behavior:
The below error will get logged.
[agent] Error writing to outputs.kafka: kafka: Failed to produce message to topic topicname: kafka: error encoding packet: invalid timestamp (1740-07-22 09:32:28.290448384 -0752 LMT)
The write to kafka fails, but the metric itself does not get dropped causing the metric buffer to fill up.
Additional info:
This functionality was implemented in #6746, however, it causes telegraf to handle messages with invalid timestamps much differently. Previously, metrics would get written regardless of the timestamp value, which is no longer the case.
As this is a limitation with kafka, adding an option to exclude the timestamp from being written into kafka would make this function like previous versions.