AWS credentials in EKS suppplied via ISRA/serviceaccount not honored

[scott-grimes]

Describe the bug
A loki-stack pod running on eks with credentials supplied by ISRA still attempts to retrieve credentials via sts:AssumeRoleWithWebIdentity

Expected behavior
Loki should the IAM role provided to it via a serviceaccount, and not require hard-coded values.

Environment:

loki-stack chart version: 2.3.1
eks v1.17.12-eks-7684af

Screenshots, Promtail config, or terminal output
Config

serviceAccount:
    create: true
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNTID:role/ROLENAME
config:
    schema_config:
      configs:
      - from: 2020-10-24
        store: boltdb-shipper
        object_store: aws
        schema: v11
        index:
          prefix: index_
          period: 24h
    storage_config:
     boltdb_shipper:
       active_index_directory: /data/loki/index
       cache_location: /data/loki/boltdb-cache
       resync_interval: 5s
       shared_store: s3
     aws:
       s3: s3://eu-central-1/mybucket
    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 24h
    compactor:
      working_directory: /data/loki/boltdb-shipper-compactor
      shared_store: aws

The role has the correct ODIC trust relationship, and rw permissions on the bucket

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::mybucket/*",
                "arn:aws:s3:::mybucket"
            ]
        }
    ]
}

Error:

level=error ts=2021-01-28T22:42:19.15177334Z caller=compactor.go:83 msg="failed to run compaction" err="WebIdentityErr: failed to retrieve credentials\ncaused by: AccessDenied: Not authorized to perform sts:AssumeRoleWithWebIdentity\n\tstatus code: 403, request id: dcb419e7-4b05-4093-bfc0-d78829c23eb7"

[iamsudip]

I think you need to have a trust relationship in this role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::<account>oidc-provider/oidc.eks.<region>.amazonaws.com/id/<region-provider>"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.<region>.amazonaws.com/id/<region-provider>:sub": "system:serviceaccount:<namespace>:<service-account-name>"
        }
      }
    }
  ]
}

[scott-grimes]

yes the correct oidc role is in place

[larsrnielsen]

yes the correct oidc role is in place

did you manage to figure it out? In my case Loki tries to use the node's instance role which generates an access exception:

AccessDeniedException: User: arn:aws:sts::something:assumed-role/eks-ng-cms_dev-cms_dev/i-something is not authorized to perform: dynamodb:ListTables on resource: arn:aws:dynamodb:eu-central-1:something:table/*\n\tstatus code: 400, request id: something"

[scott-grimes]

nope. ended up having to hardcode the creds in the config.storage_config.aws.s3 connection string per the docs 👎

[larsrnielsen]

nope. ended up having to hardcode the creds in the config.storage_config.aws.s3 connection string per the docs 👎

I managed to reproduce the error you experienced by changing the name found on the IAM Role's trust policy to point to a non-existing SA. Then I changed it back to real name, redeployed and suddenly the pod reported a ready status.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[bearice]

I had same problem and found that IAM policy from loki document was wrong:

When using S3 as object storage, the following permissions are needed:

- `s3:ListBucket`
- `s3:PutObject`
- `s3:GetObject`
- `s3:DeleteObject` (if running the Single Store (boltdb-shipper) compactor)

Resources: `arn:aws:s3:::<bucket_name>`, `arn:aws:s3:::<bucket_name>/*`

But actually s3:ListObjects is needed instead of s3:ListBucket

I'm not sure if other permissions is needed so I had give a very permissive policy like s3:* to the role, and start a cloudtrail log to see what is really needs.

It would be good if developers could clarify this and correct the document.

[amilanoski]

Please reopen this issue. We should be able to provide an eks annotation as this follows security best practises of PoLP.

https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html

[amilanoski]

@bearice How can we reopen this?

[Jasper-Ben]

Please reopen this issue. We should be able to provide an eks annotation as this follows security best practises of PoLP.

https://docs.aws.amazon.com/eks/latest/userguide/specify-service-account-role.html

Works here. I've just successfully written logs to s3 with role-based access and k8s annotations, basically by following the example here: https://grafana.com/docs/loki/latest/configuration/examples/#aws-basic-config-no-credsyaml