Question: How to provide aws role:arn value to authenticate S3

[saranyaeu2987]

Hi,
I am using strimzi kafka connect to push data into S3 using camel-s3-sink-connector.
I wanted to use IAM role authentication instead of access key and secret.

How should I need to provide AWS role:arn in my yaml file when camel.component.aws-s3.useIAMCredentials: true??

Sample deployment file (reference: https://developers.redhat.com/blog/2020/02/14/using-secrets-in-apache-kafka-connect-configuration/)

apiVersion: kafka.strimzi.io/v1alpha1
kind: KafkaConnector
metadata:
  name: s3-sink-connector
  labels:
    strimzi.io/cluster: my-connect-cluster
spec:
  class: org.apache.camel.kafkaconnector.CamelSinkConnector
  tasksMax: 1
  config:
    key.converter: org.apache.kafka.connect.storage.StringConverter
    value.converter: org.apache.kafka.connect.storage.StringConverter
    topics: abc_abc
    camel.sink.url: aws-s3://testse1232?keyName=s3-connect/${date:now:yyyyMMdd-HHmmssSSS}
    camel.component.aws-s3.configuration.autocloseBody: false
    camel.component.aws-s3.useIAMCredentials: true

[valdar]

If it is the bucket s3 arn I think you can place it where testse1232 is in the example config you provided.

[saranyaeu2987]

@valdar
Its is an IAM Role arn which has full access S3 bucket policy
arn:aws:iam::*******:role/stg-abc-s3-ae4c7ad7.

Can I give IAM role arn in front of bucket name ? If so, will the sink url be something like this
camel.sink.url: aws-s3://arn:aws:iam::*******:role/stg-abc-s3-ae4c7ad7/testse1232?keyName=s3-connect/${date:now:yyyyMMdd-HHmmssSSS}

Can you please provide me an example ?

[valdar]

As far as I know, it is not possible to authenticate just using an arn role. There are 2 options:

1. Provide access-key, secret-key and useIAMCredentials: false.
2. useIAMCredentials: true, in this case, the connector should configure itself from the EC2 environment in which it runs.

[saranyaeu2987]

@valdar

1. For option 2 useIAMCredentials: true -> do you mean that EC2 instance should already have access to S3 and yaml file should only specify the boolean useIAMCredentials ?

2. How about camel strimzi kafka-connect running on kubernetes cluster in EC2 instances. I wanted only k8s pod have access to S3 and not the entire EC2 instances. -> Any suggestions on how to do that ?

[oscerd]

1. Exactly

2. This is a question for Strimzi project.

[saranyaeu2987]

@oscerd
Thanks for prompt response!'

2. Does it mean, if I some how make the only pod to access S3 using IAM role (instead of EC2 instances). Do I need to any other configuration for camel s3 sink to work apart from this property camel.component.aws-s3.useIAMCredentials: true

[valdar]

@saranyaeu2987 yep, that is how it should theoretically work.

[valdar]

@saranyaeu2987 I am closing this one for now. Feel free to reopen this one or open another issue if something still needs to be clarified.