Question: Does Camel looks for IAM roles from Env variables

[saranyaeu2987]

This is a followup of #267.

@valdar @oscerd

We configured EKS k8s pod with serviceaccount with IAM role for S3 authentication (https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-technical-overview.html) which will create 2 ENV variables inside pod

AWS_ROLE_ARN=arn:aws:iam::AWS_ACCOUNT_ID:role/IAM_ROLE_NAME
AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

When I try to run Camel Sink connector, I get Access Denied issue for S3

QUESTION: Does camel.component.aws-s3.useIAMCredentials: true looks for IAM role arn from those env variables in Pod?

[oscerd]

The useiamcredentials option just looks for using the ec2 IAM credentials and use them.

[saranyaeu2987]

@oscerd
Where do you look for IAM credentials in a EC2 instance? Are Env variables in credentials provider chain?

[oscerd]

That's related to the underline instance. It's the IAM role with you created the ec2 instance. No the env variables are not taken into account. With the Aws2 components you can inject your own client instance and it will look to env variables if you don't provide a credential provider

[valdar]

@saranyaeu2987 there are more info here: aws/aws-sdk-java-v2#1470 but you need sdk version >= 2.10.11, so you need to use camel-aws2-*-kafka-connector versions...

[saranyaeu2987]

@valdar
Thanks for prompt response. I am currently using camel-aws-s3-kafka-connector-0.2.0.jar
https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/ has only 0.2.0 version.
Where can I find camel-aws2-2.10.11-kafka-connector jar/binaries ?

[geota]

@valdar is correct. This should be as easy as Camel using the correct SDK version that has support for the AWS_WEB_IDENTITY_TOKEN_FILE env var. This should just be crawled in the DefaultCredentialProvider chain that the SDK executes to find credentials.

[oscerd]

For the env variables for secret and access key, the standard client instance should check them out of the box

[geota]

@oscerd we are trying to get EKS role based credentials working. To be clear, this is the official AWS supported way of providing role base credentials to Kubernetes pods. Our company does not allow access key/secret keys to avoid credential management overhead (securing, cycling, invalidating etc) and enforces all AWS access to use roles.

AWS added a new way to source credentials in their SDKs to support K8s pod scoped IAM roles. @valdar linked the correct issue discussing this need in the java-sdk. It was fxed via aws/aws-sdk-java-v2#1501 and released as part of AWS Java SDK 2.10.11

See:
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

[oscerd]

Anyway, it should possible when instancing your own client. Using the correct version

[saranyaeu2987]

@oscerd @valdar
Camel repo has only camel-aws2-0.2.0-kafka-connector.jar , Does it support aws-sdk >= 2.10.11 ?

[saranyaeu2987]

@valdar
I am not very successful in resolving access issue.

I tried both camel-aws2-s3-kafka-connector-0.2.0.jar and camel-aws-s3-kafka-connector-0.2.0.jar both gave me access issue.

Below are list of plugins used.

I am unsure if camel-aws2-s3-kafka-connector-0.2.0.jar and camel-aws-s3-kafka-connector-0.2.0.jar uses aws-sdk >= 2.10.11

[oscerd]

Only the Aws2 connectors are based on sdk v2. The version used on 0.2.0 kafka connector of sdk v2 is 2.11.5

[saranyaeu2987]

@oscerd @valdar
I tried 2 images having camel aws2 jars and both gave the same following error
Image 1 with following plugins obtained from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws-s3-kafka-connector/0.2.0/ and replaced 2 aws jars ( camel-aws-s3-3.2.0.jar, camel-aws-s3-kafka-connector-0.2.0.jar) with aws2 jar (camel-aws2-s3-3.3.0.jar, camel-aws2-s3-kafka-connector-0.2.0.jar)

Image 2 with following plugins obtained from https://repo1.maven.org/maven2/org/apache/camel/kafkaconnector/camel-aws2-s3-kafka-connector/0.2.0/

Caused by: java.lang.IllegalArgumentException: Error configuring property: camel.component.aws-s3.configuration.autocloseBody because cannot find component with name aws-s3. Make sure you have the component on the classpath
	at org.apache.camel.main.BaseMainSupport.lambda$autoConfigurationFromProperties$15(BaseMainSupport.java:989)
	at org.apache.camel.main.BaseMainSupport.computeProperties(BaseMainSupport.java:1249)
	at org.apache.camel.main.BaseMainSupport.autoConfigurationFromProperties(BaseMainSupport.java:985)
	at org.apache.camel.main.BaseMainSupport.autoconfigure(BaseMainSupport.java:562)
	at org.apache.camel.main.BaseMainSupport.postProcessCamelContext(BaseMainSupport.java:607)
	at org.apache.camel.main.BaseMainSupport.initCamelContext(BaseMainSupport.java:440)
	at org.apache.camel.main.Main.doInit(Main.java:109)
	at org.apache.camel.support.service.BaseService.init(BaseService.java:83)
	at org.apache.camel.support.service.BaseService.start(BaseService.java:111)
	at org.apache.camel.main.MainSupport.run(MainSupport.java:83)
	at org.apache.camel.kafkaconnector.utils.CamelMainSupport$CamelContextStarter.run(CamelMainSupport.java:220)
	... 3 more
2020-06-18 18:43:23,384 ERROR WorkerSinkTask{id=heb-s3-s123456-connector-0} Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelSinkTask connector task (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO Stopping CamelContext (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelContext stopped (org.apache.camel.kafkaconnector.utils.CamelMainSupport) [task-thread-heb-s3-s123456-connector-0]
2020-06-18 18:43:23,384 INFO CamelSinkTask connector task stopped (org.apache.camel.kafkaconnector.CamelSinkTask) [task-thread-heb-s3-s123456-connector-0]

I feel camel-aws2-s3-.jar is giving me this error, because if I have camel-aws-s3-.jar doesnt through above error.

Please guide me which jar Am I missing !!

[oscerd]

The scheme for aws2-s3 is aws2-s3, so it is camel.component.aws2-s3...