Telegraf kafka_consumer does not work with Kafka 2.0

[HristoMohamed]

Hi,
Updating Apache Kafka to 2.0.0 causes problems with the current version of the sarama library.
2018-08-23T11:32:57Z E! Error in plugin [inputs.kafka_consumer]: Consumer Error: kafka: error while consuming telegraftopic/22: kafka: error decodin:wqg packet: CRC didn't match expected 0x0 got 0xe38a6876

They have already addressed this:
IBM/sarama#1149

Maybe an update in the sarama library version is in order?
I already compiled telegraf (545b59f) with the latest sarama (647feef69a1a7e0e23cccceeb786176bc4fcf5ea). I can see that telegraf works properly now.
I have only tried it against kafka 2.0.0.

System info:

Telegraf version: Telegraf v1.7.3 (git: release-1.7 4dc29d0)
Kafka version: 2.0.0
Scala Version (for Kafka) : 2.12

Steps to reproduce:

1. Try to consume from a kafka 2.0.0 topic
2. Fail with:
   2018-08-23T11:32:57Z E! Error in plugin [inputs.kafka_consumer]: Consumer Error: kafka: error while consuming telegraftopic/22: kafka: error decodin:wqg packet: CRC didn't match expected 0x0 got 0xe38a6876

[danielnelson]

@HristoMohamed Would you be able to try Telegraf master and set the new version option to 1.1.0? I think this may allow using Kafka 2.0 with the current version, and I would feel more comfortable backporting the version option to Telegraf 1.7.x.

[HristoMohamed]

Hey,
Yes I can test by later this week.

[dvenza]

I can confirm that the latest master, with the version option works with Kafka 2.0

[danielnelson]

Good news, thanks for checking this out @dvenza.

I think this will need to wait for 1.8 because of the changes to the sarama library version.