Offsets handling inconsistent with Java consumers

[ducas]

Use Cases

As a developer
I want to be able to use Java and .NET consumers in the same group
So that I can test them side by side

As a developer
I want lags to be reported with .NET consumers in the same way Java consumers report them
So that I don't have to change my monitoring tools depending on the client

Description

Java consumers commit the offset of the next expected message and start from there when next consuming. Instead, this library commits the offset of the last message received. E.g.

• Java consumer
  • Receives message with P:0 O:1
  • Commits to P:0 with O:2
• kafkanet consumer
  • Receives message with P:0 O:1
  • Commits to P:0 with O:1

This means that all monitoring tools (e.g. Kafka Manager) report a lag of 1 when the consumer is at the end of the topic instead of a lag of 0. It also means that when I swtich from a Java consumer to kafkanet consumer I'll miss a message, because it will try to start at P:0 O:3.

Proposed fix

A configuration flag to change the behaviour of the ZookeeperConsumerConnector and Fetcher to allow it to use offsets the same way that Java consumers do.