Description
try to use CamelAws2s3SourceConnector to connect to another S3 provider (minio)
when I try to upload a simple text file to my s2 bucket -> error
file is deleted from the bucket.
KafkaConnector
apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
name: aws2-s3-source-connector
labels:
strimzi.io/cluster: my-connect-cluster
spec:
class: org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector
config:
camel.component.aws2-s3.configuration.access-key: minio
camel.component.aws2-s3.configuration.secret-key: minio123
camel.component.aws2-s3.configuration.region: EU_WEST_1
camel.source.endpoint.autocloseBody: false
camel.source.endpoint.overrideEndpoint: true
camel.source.endpoint.uriEndpointOverride: http://minio-server
camel.source.maxPollDuration: 10000
camel.source.path.bucketNameOrArn: camel-connector-test
key.converter: org.apache.kafka.connect.storage.StringConverter
topics: s3-topic
value.converter: org.apache.camel.kafkaconnector.aws2s3.converters.S3ObjectConverter
tasksMax: 1
Log
2020-07-28 12:35:44,903 INFO Setting initial properties in Camel context: [{camel.source.endpoint.autocloseBody=false, connector.class=org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceConnector, camel.source.maxPollDuration=5000, camel.source.endpoint.runLoggingLevel=TRACE, camel.component.aws2-s3.configuration.region=EU_WEST_1, topics=s3-topic, tasks.max=1, camel.source.endpoint.overrideEndpoint=true, camel.source.endpoint.uriEndpointOverride=http://minio-server, camel.source.component=aws2-s3, task.class=org.apache.camel.kafkaconnector.aws2s3.CamelAws2s3SourceTask, camel.source.path.bucketNameOrArn=camel-connector-test, camel.component.aws2-s3.configuration.access-key=minio, name=aws2-s3-source-connector, camel.component.aws2-s3.configuration.secret-key=minio123, value.converter=org.apache.camel.kafkaconnector.aws2s3.converters.S3ObjectConverter, key.converter=org.apache.kafka.connect.storage.StringConverter}] (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:44,913 INFO Creating Camel route from(aws2-s3://camel-connector-test?autocloseBody=false&overrideEndpoint=true&runLoggingLevel=TRACE&uriEndpointOverride=http://minio-server).to(direct:end?pollingConsumerQueueSize=1000&pollingConsumerBlockTimeout=0&pollingConsumerBlockWhenFull=true) (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:44,916 INFO [Producer clientId=connector-producer-aws2-s3-source-connector-0] Cluster ID: k_EXjSBzQvmhHkadWFBbcg (org.apache.kafka.clients.Metadata) [kafka-producer-network-thread | connector-producer-aws2-s3-source-connector-0]
2020-07-28 12:35:44,922 INFO Starting CamelContext (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:44,923 INFO Using properties from: classpath:application.properties;optional=true (org.apache.camel.main.BaseMainSupport) [pool-14-thread-1]
2020-07-28 12:35:44,928 INFO Auto-configuration summary: (org.apache.camel.main.BaseMainSupport) [pool-14-thread-1]
2020-07-28 12:35:44,928 INFO camel.component.aws2-s3.configuration.region=EU_WEST_1 (org.apache.camel.main.BaseMainSupport) [pool-14-thread-1]
2020-07-28 12:35:44,928 INFO camel.component.aws2-s3.configuration.accessKey=minio (org.apache.camel.main.BaseMainSupport) [pool-14-thread-1]
2020-07-28 12:35:44,928 INFO camel.component.aws2-s3.configuration.secretKey=minio123 (org.apache.camel.main.BaseMainSupport) [pool-14-thread-1]
2020-07-28 12:35:44,929 INFO No additional Camel XML routes discovered from: classpath:camel/.xml (org.apache.camel.main.DefaultRoutesCollector) [pool-14-thread-1]
2020-07-28 12:35:44,929 INFO No additional Camel XML rests discovered from: classpath:camel-rest/.xml (org.apache.camel.main.DefaultRoutesCollector) [pool-14-thread-1]
2020-07-28 12:35:44,938 INFO Apache Camel 3.3.0 (CamelContext: camel-4) is starting (org.apache.camel.impl.engine.AbstractCamelContext) [pool-14-thread-1]
2020-07-28 12:35:44,938 INFO StreamCaching is not in use. If using streams then its recommended to enable stream caching. See more details at http://camel.apache.org/stream-caching.html (org.apache.camel.impl.engine.AbstractCamelContext) [pool-14-thread-1]
2020-07-28 12:35:44,952 INFO Route: route4 started and consuming from: aws2-s3://camel-connector-test (org.apache.camel.impl.engine.AbstractCamelContext) [pool-14-thread-1]
2020-07-28 12:35:44,952 INFO Total 1 routes, of which 1 are started (org.apache.camel.impl.engine.AbstractCamelContext) [pool-14-thread-1]
2020-07-28 12:35:44,952 INFO Apache Camel 3.3.0 (CamelContext: camel-4) started in 0.014 seconds (org.apache.camel.impl.engine.AbstractCamelContext) [pool-14-thread-1]
2020-07-28 12:35:44,953 INFO CamelContext started (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:44,953 INFO CamelSourceTask connector task started (org.apache.camel.kafkaconnector.CamelSourceTask) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:44,953 INFO WorkerSourceTask{id=aws2-s3-source-connector-0} Source task finished initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,968 INFO WorkerSourceTask{id=aws2-s3-source-connector-0} Committing offsets (org.apache.kafka.connect.runtime.WorkerSourceTask) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 INFO WorkerSourceTask{id=aws2-s3-source-connector-0} flushing 0 outstanding messages for offset commit (org.apache.kafka.connect.runtime.WorkerSourceTask) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 ERROR WorkerSourceTask{id=aws2-s3-source-connector-0} Task threw an uncaught and unrecoverable exception (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-aws2-s3-source-connector-0]
org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error handler
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:178)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:104)
at org.apache.kafka.connect.runtime.WorkerSourceTask.convertTransformedRecord(WorkerSourceTask.java:295)
at org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:321)
at org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:245)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:184)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:234)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to software.amazon.awssdk.core.ResponseInputStream
at org.apache.camel.kafkaconnector.aws2s3.converters.S3ObjectConverter.fromConnectData(S3ObjectConverter.java:37)
at org.apache.kafka.connect.storage.Converter.fromConnectData(Converter.java:63)
at org.apache.kafka.connect.runtime.WorkerSourceTask.lambda$convertTransformedRecord$2(WorkerSourceTask.java:295)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:128)
at org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:162)
... 11 more
2020-07-28 12:35:45,969 ERROR WorkerSourceTask{id=aws2-s3-source-connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 INFO Stopping CamelSourceTask connector task (org.apache.camel.kafkaconnector.CamelSourceTask) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 INFO Stopping CamelContext (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 INFO Apache Camel 3.3.0 (CamelContext: camel-4) is shutting down (org.apache.camel.impl.engine.AbstractCamelContext) [task-thread-aws2-s3-source-connector-0]
2020-07-28 12:35:45,969 INFO Starting to graceful shutdown 1 routes (timeout 45 seconds) (org.apache.camel.impl.engine.DefaultShutdownStrategy) [task-thread-aws2-s3-source-connector-0]
`