Skip to content

Question: S3 source connector pulls duplicate data when task.max > 1 #959

Closed
@IAmMonmoy

Description

@IAmMonmoy

I am using the aws2-s3-kafka-source-connector connector ([https://github.com/apache/camel-kafka-connector-examples/tree/master/aws2-s3/aws2-s3-move-after-read)]) with following configuration.
{ "name": "s3-connector", "config": { "connector.class": "org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector", "tasks.max": 3, "camel.source.path.bucketNameOrArn" : "bucketname", "camel.source.endpoint.prefix": "test", "camel.source.endpoint.region": "region-name", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "value.converter": "org.apache.kafka.connect.converters.ByteArrayConverter", "camel.source.endpoint.moveAfterRead": "true", "camel.source.endpoint.destinationBucket": "bucketname", "camel.source.endpoint.destinationBucketPrefix": "move", "topics": "topicname" } }
when used wih task.max = 1 it works perfectly but pulls duplicate data in case task max is greater than 1.
Is there any workaround for this ?

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions