Skip to content

How to support at least once for CamelKafkasourceSourceConnector #1639

Open
@seanshi007

Description

@seanshi007

We've been testing a few Camel Kafka Connectors (more specifically PubSub source connector and Kafka source connector). And it looks like PubSub source connector is able to support at least once by ack the message via calling the oncompletion synchronization in the override commitRecord method.

However, for Kafka source connector, even if we disable auto commit, enable manual commit, create a new connector and task that extends the auto generated CamelKafkasourceSourceConnector and CamelKafkasourceSourceTask, and override the commitRecord in the extended task to get the KafkaManualCommit from the exchange's header and try to call it to commit, because the actual Kafka consumer and connector task are two different threads, the commit is never going to work as Kafka consumer can only be operated on the same thread. So, I am just wondering is there any approach that we can make the Camel Kafka source connector support at least once?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions