Non transient processing errors

Non-transient processing errors

The library can handle non-transient errors that may occur in your controller during message processing.

An error is non-transient if it repeats regardless of how many times you retry the operation. For example, your external database is unreachable due to a firewall blocking. There's no point in retrying unless you update the firewall rules.

Unhandled exceptions thrown by your ProcessAsync method are considered as non-transient.

There are 2 possible ways to handle non-transient errors - either the consumer is stopped or the current intake of messages is sent to a separate "dead-lettering" topic.

Stop consumer

The default behavior is "stop consumer", meaning that the consumer simply disconnects from the Kafka and no longer consumes any messages.

It will very likely lead to all the remaining consumers stop too!

The reason is the following. After consumer disconnection the Kafka re-assigns its partition(s) to another consumer. If there is a "dirty message" (aka "dead letter") which leads to unhandled exception it will result in stopping this consumer too. This process will repeat until all the consumers are eventually stopped.

This may be acceptable in cases when it is important to process every message and it is not critical to delay the message processing until the problem is fixed.

However, when it is critical to continue message processing and give up the "dirty message" you should enable the "dead-lettering".

Dead-lettering

Dead-lettering is essentially putting the "dirty" messages away into a separate topic so that the consumer can continue processing the next messages from the main topic.

Please read a separate Dead-lettering article where it is explained how to enable dead-lettering in your service.