Add AWS pod identity support #499

zach-dunton-sf · 2019-12-05T19:15:39Z

No description provided.

zach-dunton-sf · 2019-12-05T19:32:26Z

docs are here kedacore/keda-docs#56

pkg/apis/keda/v1alpha1/triggerauthentication_types.go

pkg/handler/scale_handler.go

'kiam' -> 'aws-kiam' 'eks' -> 'aws-eks'

otterley · 2020-01-22T01:51:37Z

I'm a bit puzzled by the proposed implementation.

In an EKS-managed cluster, if the KEDA operator Pod is properly associated with a ServiceAccount, and the ServiceAccount has the proper eks.amazonaws.com/role-arn annotation, then the AWS Go SDK will do the right thing automatically if you use the default credential chain. A mutating webhook that EKS provides will automatically inject the necessary AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE environment variables in the pod, and the default credential chain knows how to look for them and what to do with them.

In other words, there are usually no code changes needed if the manifest is correctly set up and you haven't messed with the default credential chain. (Unfortunately, it looks like in the existing code, the credential chain is being overridden for reasons that aren't clear to me yet; we probably need to think about that.)

If I understand the proposal correctly (and I might not be!), this change would cause the KEDA operator to look at the annotations applied to the pod template of the target deployment and use that as a role name to assume. But I do not think this will work as designed. The IAM role is bound to the pod making the API calls -- only that pod can assume the IAM role. The annotation needs to be applied to the operator, not the target.

Let me know if you think I'm confused.

Also, out of curiosity, how did you test this change?

zach-dunton-sf · 2020-01-22T13:33:20Z

You can run a single role and attach it to the operator that allows it to read the stats for all SQS queues.

A more granular approach would be to craft a role for your application that only allows "roleAppA" to read SQS queue information for only the SQS queues of "AppA". The IAM role "roleAppA" would have to have a trust policy that allows the service account of KEDA to assume it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "$PROVIDER_ARN"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${ISSUER_HOSTPATH}:sub": "system:serviceaccount:default:keda-serviceaccount"
        }
      }
    }
  ]
}

I will investigate further into the eks implementation. I have not tested it fully.

tomkerkhove · 2020-03-04T10:43:45Z

@zach-dunton-sf would it be possible to merge with master please?

I presume you are still waiting on a review? Care to pick this up @ahmelsayed ?

ahmelsayed · 2020-03-04T21:39:24Z

Sorry for the delay @zach-dunton-sf, I'm not too familiar with how pod identity works on AWS so was just keeping an eye on the conversation here. The change looks fine from my prospective. I will merge it once the build passes

ncrothe · 2020-03-10T16:11:14Z

Is this solution expected to also work for kube2iam which uses similar techniques to kiam?

zach-dunton-sf · 2020-03-10T16:25:44Z

Yes. kube2iam should work also.

…

On Tue, Mar 10, 2020 at 5:11 PM Christian Rothe ***@***.***> wrote: Is this solution expected to also work for kube2iam which uses similar techniques to kiam? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#499?email_source=notifications&email_token=AEH7ZDOC4JAYOK7DIMMCBDDRGZRDDA5CNFSM4JV6P5O2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOMCFEQ#issuecomment-597172882>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEH7ZDLF2JEGPJ6S3E4UJ3LRGZRDDANCNFSM4JV6P5OQ> .

ncrothe · 2020-03-16T13:48:54Z

Tried it out and it does not seem to work for kube2iam (or I don't understand what I am supposed to do).
I have added a kube2iam role to the metrics-apiserver pod thus allowing the metrics server to get the correct credentials to fetch metrics from Cloudwatch. However, I am getting 403s. The following can be found in the logs of the apiserver:

AccessDenied: User: <MY_KUBE2IAM_PROVIDED_READER_ROLE> is not authorized to perform: sts:AssumeRole on resource: <ROLE_OF_THE_POD_TO_BE_SCALED>

I do not understand that. Why would it try to assume the role of the thing it is supposed to be scaling? That thing might not even have access to the metrics. What is the right way to use aws-kiam? The docs are, unfortunately, rather sparse on this.

zach-dunton-sf · 2020-03-16T14:04:37Z

You need to modify the role <ROLE_OF_THE_POD_TO_BE_SCALED> so it can be assumed by <MY_KUBE2IAM_PROVIDED_READER_ROLE>. You can do that in the IAM console, if you select the <ROLE_OF_THE_POD_TO_BE_SCALED> role, then click on "Trust Relationships" then add the ARN for <MY_KUBE2IAM_PROVIDED_READER_ROLE>

…

On Mon, Mar 16, 2020 at 2:49 PM Christian Rothe ***@***.***> wrote: Tried it out and it does not seem to work for kube2iam (or I don't understand what I am supposed to do). I have added a kube2iam role to the metrics-apiserver pod thus allowing the metrics server to get the correct credentials to fetch metrics from Cloudwatch. However, I am getting 403s. The following can be found in the logs of the apiserver: AccessDenied: User: <MY_KUBE2IAM_PROVIDED_READER_ROLE> is not authorized to perform: sts:AssumeRole on resource: <ROLE_OF_THE_POD_TO_BE_SCALED> I do not understand that. Why would it try to assume the role of the thing it is supposed to be scaling? What is the right way to use aws-kiam? The docs are, unfortunately, rather sparse on this. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#499 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEH7ZDNDW6SBVECC4JHSAO3RHYU5NANCNFSM4JV6P5OQ> .

ncrothe · 2020-03-16T14:09:05Z

I understand how to do the mechanical AWS parts, but I don't understand why I would want to set it up that way? ROLE_OF_THE_POD_TO_BE_SCALED is the role that pod needs to do it's work and that work has nothing to do with metrics. At most it might typically have some PutMetrics permissions, but I don't think it should, as a general rule, have GetMetrics permissions - that is not the task of the scaled pod and thus IMO it should not have the permission.

ncrothe · 2020-03-16T14:09:29Z

(but thank you for clarifying your intention with the different roles)

ncrothe · 2020-03-16T14:18:31Z

Briefly thinking out loud: What would the alternative be, though? I came in expecting to add the permissions to the apiserver, which I'm starting to realise would mean I need to add all relevant permissions for all possibly AWS-scaled things to that pod/role. Which I guess could be seen as excessive. OTOH, I really dislike having to assume the role of the pod-to-be-scaled. Can we maybe add an annotation to the ScaledObject or the TriggerAuthentication object instead and then make sure the role the apiserver runs with can assume that one? From a separation of concerns perspective that would be much cleaner IMO.

zach-dunton-sf · 2020-03-16T14:24:45Z

It is possible to assign a separate more limited role using the TriggerAuthentication, don't use podIdentity. You can specify a Role ARN in the TriggerAuthentication.

…

On Mon, Mar 16, 2020 at 3:18 PM Christian Rothe ***@***.***> wrote: Briefly thinking out loud: What would the alternative be, though? I came in expecting to add the permissions to the apiserver, which I'm starting to realise would mean I need to add all relevant permissions for all possibly AWS-scaled things to that pod/role. Which I guess could be seen as excessive. OTOH, I really dislike having to assume the role of the pod-to-be-scaled. Can we maybe add an annotation to the ScaledObject or the TriggerAuthentication object instead and then make sure the role the apiserver runs with can assume that one? From a separation of concerns perspective that would be much cleaner IMO. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#499 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEH7ZDOLFSTFBXLOGI3BM3DRHYYMXANCNFSM4JV6P5OQ> .

ncrothe · 2020-03-16T14:36:04Z

Hmm, ok. But if I understand correctly: If I want to use TriggerAuthentication (with or without roleARN), I will then need to hard-code AWS_ACCESS_KEY_ID etc. (which is a big no-no in our organization) or jump through some hoops to make dynamic secrets work with this possibly (which is what kube2iam and kiam were designed to avoid).

zach-dunton-sf · 2020-03-16T14:39:17Z

No, you just need the role arn. Christian Rothe <notifications@github.com> schrieb am Mo. 16. März 2020 um 15:36:

…

Hmm, ok. But if I understand correctly: If I want to use TriggerAuthentication (with or without roleARN), I will then need to hard-code AWS_ACCESS_KEY_ID etc. (which is a big no-no in our organization) or jump through some hoops to make dynamic secrets work with this possibly (which is what kube2iam and kiam were designed to avoid). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#499 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AEH7ZDI4C2AUIR3SBQVOGP3RHY2OJANCNFSM4JV6P5OQ> .

ncrothe · 2020-03-16T15:01:59Z

Could you please point in the direction of a working example then?

apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-trigger-auth-aws-credentials
  namespace: default
spec:
  awsRoleArn: <MY_METRICS_READER_ROLE>

Does result in errors like
{"level":"error","ts":1584369099.61145,"logger":"controller_scaledobject","msg":"Failed to create new HPA resource","Request.Namespace":"default","Request.Name":"aws-sqs-queue-scaledobject","HPA.Namespace":"default","HPA.Name":"keda-hpa","error":"error getting scaler for trigger #0: Error parsing Cloudwatch metadata: 'AWS_ACCESS_KEY_ID' doesn't exist in the deployment environment","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.1/zapr.go:128\ngithub.com/kedacore/keda/pkg/controller/scaledobject.(*ReconcileScaledObject).reconcileDeploymentType\n\tkeda/pkg/controller/scaledobject/scaledobject_controller.go:229\ngithub.com/kedacore/keda/pkg/controller/scaledobject.(*ReconcileScaledObject).Reconcile\n\tkeda/pkg/controller/scaledobject/scaledobject_controller.go:155\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:192\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.2/pkg/internal/controller/controller.go:171\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88"}

zach-dunton-sf · 2020-03-16T15:11:57Z

It's a bit weird because you have to go through a Secret. I do not think it is possible to set it directly on the TriggerAuthentication.

https://github.com/kedacore/keda-docs/blob/master/content/scalers/aws-sqs.md

apiVersion: keda.k8s.io/v1alpha1
kind: TriggerAuthentication
metadata:
  name: trigger-auth-aws-role
spec:
  secretTargetRef:
  - key: AWS_ROLE_ARN
    name: keda-aws-secrets
    parameter: awsRoleArn
---
apiVersion: v1
kind: Secret
metadata:
  name: keda-aws-secrets
data:
  AWS_ROLE_ARN: some role arn

ncrothe · 2020-03-16T15:51:45Z

That did it! Thanks for the help, I got it running now. Looks like I completely misunderstood what podIdentity is about and plain kube2iam would've worked before.

To recap in case someone runs into this ticket again:

I have set up a kube2iam role on both operator (for setup, I guess) and metrics-apiserver that is allowed to assume the role configured in the TriggerAuthentication
TriggerAuthentication with awsRoleArn reference in secret as explained by Zach
no podIdentity involved in my use case anywhere

tomkerkhove · 2020-03-16T15:52:11Z

TriggerAuthentication doesn't support hardcoded values indeed, for now.

Was everything clear or do we need to update the docs?

ncrothe · 2020-03-16T16:10:55Z

I think a full example with the awsRoleArn could be useful. It is not obvious that that can e.g. pull default AWS authenticators and that it needs to go into the secret.

tomkerkhove · 2020-03-16T16:33:07Z

Tracking this in kedacore/samples#10 but happy to link to a sample if you have one

bmacauley · 2020-03-29T12:20:01Z

Having problems setting the SQS scaler up on a kubernetes cluster using kube2iam. We are using v 1.3.0.

Followed @ncrothe approach above...

kube2iam role on both operator (for setup, I guess) and metrics-apiserver that is allowed to assume the role configured in the TriggerAuthentication (the kube2iam annotation is being written to the wrong section in the charts)
TriggerAuthentication with awsRoleArn reference in secret as explained by Zach
no podIdentity involved in my use case anywhere
created a keda role(k8s-keda), which contains a trust relationship for the kube2iam role attached to the kubernetes worker node(same as the many other roles we have)

Get the following error in the keda-operator...
{"level":"debug","ts":1585482843.978178,"logger":"scalehandler","msg":"Error getting scale decision","ScaledObject.Namespace":"hydra-sbx1-mobilelogins", "ScaledObject.Name":"hydra-ingest-outbound-etl","ScaledObject.ScaleType":"deployment", "Error":"AccessDenied: User: arn:aws:sts::123456789123:assumed-role/k8s-keda/e7450c49-k8s-keda is not authorized to perform: sts:AssumeRole on resource: arn:aws:iam::123456789123:policy/k8s-keda\n\n\tstatus code: 403,

kube2iam provides the role for the pod(uses the annotation)...so session should be available inside pod?

Any thoughts?

Looking at the code...it seems to use a different technique to consume the kiam/kube2iam role compared to other tools eg externaldns
aws_sqs_queue_scaler.go.
aws_iam_authorization.go

externaldns. ...uses session.NewSessionWithOptions
external-dns/provider/aws.go

ncrothe · 2020-03-29T17:47:36Z

I think: To make the current version of this work, you need one more indirection. As the error implies, your pod is assuming the intended role via kube2iam, but then keda tries to assume the same role as that role. So either:

you add the role as trust to itself or
you split this into two roles: one role A that the pod assumes via kube2iam and a role B that role A can assume (via Keda) that has the actual permissions.

bmacauley · 2020-03-30T08:09:45Z

@ncrothe
Good suggestions... I was thinking similar
I will try these out and give an update

tomkerkhove · 2020-03-30T08:18:27Z

Would you be open to create a new issue for this to see what we need to fix and/or improve the docs?

bmacauley · 2020-03-30T08:20:16Z

@tomkerkhove
Yep...no problem

…dacore#499) Signed-off-by: Ratnadeep Debnath <rtnpro@gmail.com>

Add AWS pod identity support

759c955

zach-dunton-sf requested review from Aarthisk and ahmelsayed as code owners December 5, 2019 19:15

zach-dunton-sf mentioned this pull request Dec 5, 2019

Provide support for Amazon EKS Pod Identity Webhook #457

Closed

tomkerkhove reviewed Dec 6, 2019

View reviewed changes

pkg/apis/keda/v1alpha1/triggerauthentication_types.go Outdated Show resolved Hide resolved

tomkerkhove reviewed Dec 6, 2019

View reviewed changes

pkg/handler/scale_handler.go Show resolved Hide resolved

Rename pod identity providers

14bc06f

'kiam' -> 'aws-kiam' 'eks' -> 'aws-eks'

tomkerkhove mentioned this pull request Mar 4, 2020

podIdentity.provider : aws-kiam #645

Closed

tomkerkhove assigned zach-dunton-sf Mar 4, 2020

This was referenced Mar 4, 2020

Revert "Add documenation for AWS pod identity providers" kedacore/keda-docs#102

Merged

Add documenation for AWS pod identity providers kedacore/keda-docs#103

Merged

ahmelsayed approved these changes Mar 4, 2020

View reviewed changes

Merge branch 'master' into feature/aws_pod_identity

ca9cfa4

ahmelsayed merged commit 4180c15 into kedacore:master Mar 4, 2020

SatishRanjan mentioned this pull request Mar 13, 2020

KEDA release 1.3.0 #676

Merged

bmacauley mentioned this pull request Mar 30, 2020

kube2iam issues ...cannot assume role correctly in AWS SQS scaler in EKS #706

Closed

preflightsiren pushed a commit to preflightsiren/keda that referenced this pull request Nov 7, 2021

Update RabbitMQ scaler docs for adding custom metric name support (ke…

bba6056

…dacore#499) Signed-off-by: Ratnadeep Debnath <rtnpro@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AWS pod identity support #499

Add AWS pod identity support #499

zach-dunton-sf commented Dec 5, 2019

zach-dunton-sf commented Dec 5, 2019

otterley commented Jan 22, 2020 •

edited

Loading

zach-dunton-sf commented Jan 22, 2020

tomkerkhove commented Mar 4, 2020

ahmelsayed commented Mar 4, 2020

ncrothe commented Mar 10, 2020

zach-dunton-sf commented Mar 10, 2020 via email

ncrothe commented Mar 16, 2020 •

edited

Loading

zach-dunton-sf commented Mar 16, 2020 via email

ncrothe commented Mar 16, 2020

ncrothe commented Mar 16, 2020

ncrothe commented Mar 16, 2020

zach-dunton-sf commented Mar 16, 2020 via email

ncrothe commented Mar 16, 2020

zach-dunton-sf commented Mar 16, 2020 via email

ncrothe commented Mar 16, 2020

zach-dunton-sf commented Mar 16, 2020

ncrothe commented Mar 16, 2020

tomkerkhove commented Mar 16, 2020

ncrothe commented Mar 16, 2020

tomkerkhove commented Mar 16, 2020

bmacauley commented Mar 29, 2020 •

edited

Loading

ncrothe commented Mar 29, 2020

bmacauley commented Mar 30, 2020

tomkerkhove commented Mar 30, 2020

bmacauley commented Mar 30, 2020

Add AWS pod identity support #499

Add AWS pod identity support #499

Conversation

zach-dunton-sf commented Dec 5, 2019

zach-dunton-sf commented Dec 5, 2019

otterley commented Jan 22, 2020 • edited Loading

zach-dunton-sf commented Jan 22, 2020

tomkerkhove commented Mar 4, 2020

ahmelsayed commented Mar 4, 2020

ncrothe commented Mar 10, 2020

zach-dunton-sf commented Mar 10, 2020 via email

ncrothe commented Mar 16, 2020 • edited Loading

zach-dunton-sf commented Mar 16, 2020 via email

ncrothe commented Mar 16, 2020

ncrothe commented Mar 16, 2020

ncrothe commented Mar 16, 2020

zach-dunton-sf commented Mar 16, 2020 via email

ncrothe commented Mar 16, 2020

zach-dunton-sf commented Mar 16, 2020 via email

ncrothe commented Mar 16, 2020

zach-dunton-sf commented Mar 16, 2020

ncrothe commented Mar 16, 2020

tomkerkhove commented Mar 16, 2020

ncrothe commented Mar 16, 2020

tomkerkhove commented Mar 16, 2020

bmacauley commented Mar 29, 2020 • edited Loading

ncrothe commented Mar 29, 2020

bmacauley commented Mar 30, 2020

tomkerkhove commented Mar 30, 2020

bmacauley commented Mar 30, 2020

otterley commented Jan 22, 2020 •

edited

Loading

ncrothe commented Mar 16, 2020 •

edited

Loading

bmacauley commented Mar 29, 2020 •

edited

Loading