Description
Hello,
In our recent use case, we have to move messages from an AWS SQS Queue to a Kafka topic. This is a perfect use case where we can leverage the power of the Camel Connectors. More specifically the aws2-sqs connector.
According to the docs, the connector adds the SQS messageId as a header to the Kafka record. and inserts the record as-is into the specified Kafka topic.
A link the documentation for context: Message headers set by the sqs producer
Now, in our use case, we need to set this header to something specific based on the data before it is inserted into the Kafka topic. In the sqs connector docs, I can see no mention where a transformer can be hooked in and run before the data is sent to the target destination. The only way around this, as suggested by @oscerd in the gitter channel, was to use custom archetypes and then generate the connector source code and add functionality as required.
While this approach definitely works, from my experience, a connector reading and making light edits to the data before inserting it into its target destination is a very common use case. In the example of Kafka specifically, their connect API already provides a framework to add-in your own transformations.
A link to the relevant Kafka documentation for context: Transformation with the Connect API
My request is to add functionality to these connectors so that the users of these connectors can chain/add lightweight transformers to make light edits to the data before the connector finally sends it off to its target destination.
PS: if this request is approved, I would love to help out and contribute in implementing this feature :)