ECS Task IAM profile credentials ignored in repository-s3 plugin

[JeffDownie]

Elasticsearch version (bin/elasticsearch --version): 5.6.2

Plugins installed: [repository-s3]

JVM version (java -version): Doesn't seem to matter

OS version (uname -a if on a Unix-like system): Doesn't seem to matter

Description of the problem including expected versus actual behavior:

When using the repository-s3 plugin from within an ECS Task that has an ECS Task IAM role
, the repository-s3 instead loads the underlying ec2 host's credentials, which appears to be contrary (or at least unexpected!), based on what is in the usage guide

Steps to reproduce:

1. Install elasticsearch 5.6.2 on a docker container
2. Use docker container in a task definition on AWS ECS
3. Give task definition an IAM role with access to an s3 bucket
4. Give EC2 instance an IAM role that denies access to s3 bucket
5. Install repository-s3 plugin using elasticsearch-plugin install repository-s3
6. Start elasticsearch
7. Attempt to create a repository using as specified at https://www.elastic.co/guide/en/elasticsearch/plugins/5.6/repository-s3-repository.html#repository-s3-repository, which fails.
8. Note that the aws cli is, however able to access the s3 bucket from inside the container correctly (e.g., aws s3 ls s3::/<BUCKET_NAME> works).

Believed cause:

I believe I've tracked the issue down to

elasticsearch/plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/InternalAwsS3Service.java

Line 172 in 16431a6

this.credentials = new InstanceProfileCredentialsProvider();

, which uses an InstanceProfileCredentialsProvider, rather than the DefaultAWSCredentialsProviderChain, which I would have thought would be more appropriate, unless I am missing something?

[tlrx]

I agree with your analyse, it looks like the repository-s3 plugin must be adapted for this use case.

[rjernst]

I believe we need to upgrade to aws sdk 1.11 in order to support this. We do not use the default credentials chain because it uses insecure methods of passing credentials (env vars, sysprops and plaintext files). So after upgrading, we will need to create our own chain that includes iam+instance profile.

[dadoonet]

Wasn't that the goal of #26479?

[dadoonet]

Sorry I meant #25552

[rjernst]

@dadoonet Yes. However, that PR undoes the secure credentials work (adding back the default credentials provider).

[dadoonet]

Very true @rjernst. I did not review it yet cause travels. Wanna add your thoughts on https://github.com/elastic/elasticsearch/pull/25552/files#diff-68ff507ea23f19b8d4b5cc0e8d1826bfR172?

[JeffDownie]

Ah! I must have missed that PR when searching for if the issue already existed!

[CliveJL]

I have just come across this issue too when trying to use an IAM Role associated with the ECS Task for S3 Repository access, and I've only just found this GitHub issue.

I am using the official Docker image for ES 5.5.3 and the "repository-s3" plugin, which seems to include version 1.10.69 of the AWS SDK for Java according to DEBUG. AWS state that the minimum required Java SDK version to support IAM Roles for Tasks is 1.11.16. See here.

To prove the point, when I attach the S3 IAM Policy to the ECS Container Instance's IAM Role, the S3 Repository "verify" command works fine. But the same Policy on the Task's assigned IAM Role doesn't work.

There are additional steps to configure the ECS Agent to allow IAM Role's to be assigned to Tasks, but I have followed these and it still doesn't work, so the SDK version seems to be the root cause...

Looking forward to seeing this one fixed, and I hope this information might help a little.