Upgrade AWS SDK for Go to support AWS IAM Roles for K8s ServiceAccounts

[tomaspinho]

Is your feature request related to a problem? Please describe.
Yes, the current version of the AWS SDK for Go that ships with Vault (v1.19.39) does not support using a K8s ServiceAccount to authenticate against a a AWS IAM Role when Vault is deployed inside a Kubernetes cluster. See: https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts-minimum-sdk.html

Describe the solution you'd like
Bump the version of AWS SDK for Go.

Describe alternatives you've considered
We could hijack the entrypoint of our Vault containers to prepopulate IAM credentials, but that would be ugly.

Explain any additional use-cases
NA

Additional context
https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

[micahhausler]

The way the AWS code in Vault is currently structured, using IAM for Service Accounts on Kubernetes would not fully work with Vault.

There are several places Vault calls out to AWS, (I may be missing others)

• Auth
• DynamoDB and S3 Storage
• Seal
• Secret Engine

All these use the /helper/awsutil package for AWS credential discovery, which will probably work fine for Auth, Storage, and Seal, but the Secret Engine is where this gets problematic. According to the IAM AssumeRoleWithWebIdentity docs:

The temporary security credentials created by AssumeRoleWithWebIdentity can be used to make API calls to any AWS service with the following exception: you cannot call the STS GetFederationToken or GetSessionToken API operations.

The AWS Secret Engine supports 3 credential methods

• iam_user- This would work fine if the service account IAM role had user creation permissions
• assumed_role- This would probably work.
• federation_token - This will not work.

It may be worth allowing for different source credentials for the Auth, Storage, and Seal functionality than the Secret Engine, or just using as-is and documenting the Federation Token limitation when used with IAM for Service Accounts on Kubernetes

[joelthompson]

Hey @micahhausler -- thanks for the heads up here! I don't see this as a major drawback as the same limitations are true when using Vault on an EC2 instance in an IAM instance profile or an ECS cluster with a task role, and this is also called out in the documentation:

Notice: Due to limitations in AWS, in order to use the federation_token credential type, Vault must be configured with IAM user credentials. AWS does not allow temporary credentials (such as those from an IAM instance profile) to be used.

[micahhausler]

@joelthompson Awesome.

The other UX issue here is that Web Identity and Assume Role providers use the role_arn field, and you can use the SDK to do an Assume Role after a Web Identity cred fetch, so you may want to consider if that is allowed and how that can be specified

[joelthompson]

Hey @micahhausler -- I submitted a PR to bump the golang SDK in #7458. However, I don't think that will achieve the goals of @tomaspinho.

The only time that the automated k8s credentials (stscreds.NewWebIdentityCredentials) will be used, as far as I can see in the SDK, is in session.assumeWebIdentity. The only calls to that are from session.resolveCredentials and session.resolveCredsFromProfile. The only calls to these come from session.mergeConfigSrcs and that, in turn, only gets called by session.newSession and that only gets called by session.NewSessionWithOptions. And the only time that gets called by session.New (which Vault uses) is when the AWS_SDK_LOAD_CONFIG environment variable is set. Maybe this is a bug in the SDK that session.New and session.NewSession have such different behaviors?

If the upstream SDK won't be changing, it seems like the options would be to switch to session.NewSession (rather than session.New) or manually create a stscreds.WebIdentityRoleProvider (which involves duplicating and reading the associated environment variables for the token file path, role ARN, and role session name) and insert that provider explicitly into the credential chain so we're not depending on the SDK to do this for us.

Am I missing something? Does this seem right to you?

[tomaspinho]

@joelthompson setting that environment variable is not a problem for us, as long as it works (and is documented for others). Let me know if I can help with anything!

[joelthompson]

Hey @tomaspinho -- I don't have familiarity with running things on k8s. Would you be able to build the code in my PR and test to see if it works, both with and without the environment variable set? Alternatively, if you have a quick script you can share to get Vault up and running on EKS to test it myself, that would also be great.

Agree that this needs documenting, but I want to separate out the SDK version bump from other changes to Vault for ease of review.

[tomaspinho]

@joelthompson Yeah, I can definitely help with that. Is there a way to enable debug logs for Vault, especifically for AWS SDK interactions?

[joelthompson]

@tomaspinho I'm not sure there's a good way to enable that without more code changes. Probably the easiest way is to just enable the AWS KMS seal and only give access to the KMS key to the Vault pod's service account IAM role.

[micahhausler]

The only time that the automated k8s credentials (stscreds.NewWebIdentityCredentials) will be used, as far as I can see in the SDK, is in session.assumeWebIdentity. The only calls to that are from session.resolveCredentials and session.resolveCredsFromProfile. The only calls to these come from session.mergeConfigSrcs and that, in turn, only gets called by session.newSession and that only gets called by session.NewSessionWithOptions. And the only time that gets called by session.New (which Vault uses) is when the AWS_SDK_LOAD_CONFIG environment variable is set. Maybe this is a bug in the SDK that session.New and session.NewSession have such different behaviors?

Yes, session.New() is missing the behavior (unless AWS_SDK_LOAD_CONFIG is set), it would be worth raising this as a Go SDK bug.

If the upstream SDK won't be changing, it seems like the options would be to switch to session.NewSession (rather than session.New) or manually create a stscreds.WebIdentityRoleProvider (which involves duplicating and reading the associated environment variables for the token file path, role ARN, and role session name) and insert that provider explicitly into the credential chain so we're not depending on the SDK to do this for us.

Am I missing something? Does this seem right to you?

I would lean against manually creating a stscreds.WebIdentityRoleProvider and adding that to the chain.

The shortest path to adding support might be just to switch from session.New to session.NewSession, as the input signatures are the same, and you can just bubble up a possible error.

[tomaspinho]

Honestly think the whole AWS_SDK_LOAD_CONFIG in session.New() shenanigan is by design, but having hit that sort of thing in the past multiple times, it would be best if they documented this behaviour a little bit better.