Telegraf stops konsuming from partition on GetOffset error, does not try again (affects entire consumer group)

[HristoMohamed]

Telegraf version telegraf-1.4.5-1.x86_64 running on 3.10.0-327.36.3.el7.x86_64, Kafka is 1.0.0, Scala for Kafka is 2.12.

Everything runs fine, until:

Dec 07 18:15:53 telecons02 telegraf[2820]: 2017-12-07T17:15:53Z E! Error in plugin [inputs.kafka_consumer]: Consumer Error: kafka: error while consuming telegrafcommon/26: kafka server: The requested offset is outside the range of offsets maintained by the server for the given topic/partition.

After this, the specified partition is not consumed anymore and messages pile up.
This is fixable by restating telegraf.

Interesting fact is that I have a few telegraf instances consuming kafka messages and all of them hit this issue on a few partitions (random partitions, cannot localize it to one partition). When I restart one telegraf instance the entire consumer group goes back to normal and messages are flowing (even on partitions served by the other instances that were stuck).

Pls help ;(

[HristoMohamed]

This is hw my config ooks like
[[inputs.kafka_consumer]]
brokers = ["kafka01:9092", "kafka02:9092", "kafka03:9092", "kafka04:9092", "kafka05:9092", "kafka06:9092", "kafka07:9092"]
consumer_group = "telegraf_metrics_consumers"
data_format = "influx"
max_message_len = 65536
offset = "oldest"
topics = ["telegrafcommon"]

[danielnelson]

@seuf Have you seen this error before?

[seuf]

@danielnelson Sorry, nope.. Maybe we should switch to the confluent kafka client : https://github.com/confluentinc/confluent-kafka-go

[HristoMohamed]

I tried restarting my entire Kafka cluster, to no avail.
At the same time I cannot see this on another topic (same partitioning and replication as the one that my Telegraf instances consume from) that is being consumed by Logstash.
At the moment my only solution reload/restart of the telegraf daemon :(

[HristoMohamed]

Could the high number of partitions somehow play role? I have 140 partitions in the topic.
Funny, I also have over 70 FreeBSD 11.0 Telegraf hosts that play as Consumers from various other topics, but those topics only have 7 partitions each.

[danielnelson]

I think this might be bsm/sarama-cluster#121, which has been fixed upstream. I will update to the latest upstream version v2.1.10. I'm a little nervous to put this into 1.5.0 since I'm hoping to release next week, do you think you could test it if I add it to 1.5.0-rc2?

[HristoMohamed]

Yep, no problemo =)

[danielnelson]

Here are the builds of 1.5.0-rc2.

[HristoMohamed]

I will test now and report back

[HristoMohamed]

Running for an hour everything is Ok so far.
I will report tomorrow as well +)