Duplicate message processing

Duplicate message processing

The library follows at-least-once semantics, meaning that your controller will receive messages at least once but it is also possible to receive them more than once in some cases.

Reasons

Your controller may receive the same messages more than once for numerous reasons:

• The same messages consumed again:
  • Transient processing errors. There was an error during message processing and your controller threw TransientException. Consumer had to accumulate the same messages again.
  • Service interruption. Your service was stopped during the message processing, so the library didn't have chance to commit offsets. When service starts again the library has to accumulate the same messages again.
  • Failed "dead-lettering". There was an error in your controller and the library attempted to send messages to a "dead-lettering" topic but failed to do so. Consumer had to restart and accumulate the same messages again.
  • Kafka commit errors. The library wasn't able to commit offsets after message processing. This resulted in consumer restart and consuming the same messages again.
• The same messages published again:
  • Publish retries. We shouldn't forget that something publishes messages to Kafka topics. In might be that the publisher retried message publish a few times. In this case, your controller will receive duplicate messages too.

Tolerate duplicate messages (controller idempotence)

As mentioned above, there are many reasons why your controller may receive the same messages again. It is therefore recommended to design idempotent controllers, i.e. tolerant to duplicate messages.

For example, if you save messages to a database, you may need to check if a corresponding record already exists there. If it exists you should either skip the current message or overwrite the existing record depending on your use-case.