Dead Letter Queue and error handling upgrades

[loicgreffier]

Overview

This issue aims to highlight the upgrades to the Dead Letter Queue (DLQ) and error handling in Kstreamplify, particularly with the changes introduced by:

• KIP-1033 - Add Kafka Streams exception handler for exceptions occurring during processing
• KIP-1034 - Dead letter queue in Kafka Streams

Kstreamplify vs Kafka Streams

Even when using Kstreamplify, it is recommended to adopt the upcoming native Dead Letter Queue routing feature of Kafka Streams.

• It handles all processing exceptions across all operators, not just in map or processor.
• It allows having multiple DLQ topics, rather than being limited to just one.
• It routes the original input record that triggered the exception, making it easier to reprocess.

Backward Compatibilty

Users who rely on Kstreamplify's DLQ may face difficulties when switching to Kafka Streams’ native DLQ because:

• Kstreamplify forwards a KafkaError Avro object to the DLQ topic, whereas Kafka Streams will forward the original input record from the sub-topology that triggered the exception (which could be in Avro or another format). A direct migration could result in a DLQ topic containing multiple message formats (KafkaError, KafkaOrder for example, String, etc.).

It could be valuable for Kstreamplify to provide an “in-between” solution that:

• Leverages the new Kafka Streams Dead Letter Queue mechanism, which is more robust.
• Preserves the KafkaError Avro format, ensuring the existing DLQ topic format remains compatible.

This approach would allow users to benefit from the new Kafka Streams DLQ mechanism with minimal effort when upgrading Kstreamplify.

Proposed Changes

Deprecation

The following classes are now deprecated:

• TopologyErrorHandler
• ProcessingResult

New Processing Exception Handler Implementation

A new processing.exception.handler implementation is proposed. It:

• Converts the incoming message into a KafkaError.
• Routes the message to the DLQ topic defined by public String dlqTopic(), accessible via KafkaStreamsExecutionContext.

Thus, migrating away from the deprecated classes requires minimal effort by using the provided processing.exception.handler implementation.

DlqTopic

The override of the dlqTopic() method should be optional, as users can choose to rely entirely on the new Kafka Streams Dead Letter Queue mechanism. It only needs to be overridden if the user wants to use Kstreamplify’s exception handlers.

Deserialization and Production Exception Handlers

Both the provided deserialization and production exception handlers remain available; they just need to be updated to comply with the new interfaces.

@ThomasCAI-mlv @sebastienviale @mlmomplot