AWS Dynamic Providers not working

[jensrotne]

Feedback

I am trying to get the dynamic providers to work following the official documentation, but I can't get my head why it doesn't.

I first followed the documentation to get the jwks and openid-configuration endpoints to work: https://docs.terrakube.io/user-guide/workspaces/dynamic-provider-credentials#generate-public-and-private-key

Secondly, I followed the documentation for how to set it up in AWS: https://docs.terrakube.io/user-guide/workspaces/dynamic-provider-credentials/aws-dynamic-provider-credentials

Also following the Github TF example: https://github.com/AzBuilder/terrakube/tree/main/dynamic-credential-setup/aws

However, whenever I try to run the example pipeline defined in the example, then I just get this error:

Initializing the backend...
���
��� Error: validating provider credentials: retrieving caller identity from STS: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: 9d3eb4f3-2d48-4457-913b-c7611dd63c00, api error InvalidClientTokenId: The security token included in the request is invalid.
��� 
��� 
���

The role is set up with this trust policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::REDACTED:oidc-provider/terrakube-api-test.wag.tools"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "terrakube-api-test.READCTED:aud": "aws.workload.identity",
                    "terrakube-api-test.READCTED:sub": "organization:WhiteawayGroup:workspace:terrakube-test-repo"
                }
            }
        }
    ]
}

The IAM Provider is set up like this:

It generates the JWT just fine, and I'm able to generate a session using STS locally, but it just keeps failing.

Any suggestion on what have gone wrong?

[BenjaminDecreusefond]

Hi @jensrotne !

Could provide us with the following details please :

• Are you running your workspace remotely or locally
• Could you show us the environment variables defined for your workspace
• can you show us your terraform aws provider configuration

Best regards !

[alfespa17]

Check #1425 that can help you doing some debugging, you could also check #839

[alfespa17]

By the way make sure these 2 endpoint are returning information

https://TERRAKUBE.MYSUPERDOMAIN.COM/.well-known/openid-configuration

https://TERRAKUBE.MYSUPERDOMAIN.COM/.well-known/jwks

And make sure the endpoints are public

[jensrotne]

Thanks for the quick answers!

Check #1425 that can help you doing some debugging, you could also check #839

I already followed those, and the token generation seems to be working just fine. I am able to generate the JWT, intercept it from the API pod logs, decode it, and even use it with the aws sts assume-role-with-web-identity CLI command.

By the way make sure these 2 endpoint are returning information

https://TERRAKUBE.MYSUPERDOMAIN.COM/.well-known/openid-configuration

https://TERRAKUBE.MYSUPERDOMAIN.COM/.well-known/jwks

And make sure the endpoints are public

Just checked, they are already public.

[jensrotne]

Hi @jensrotne !

Could provide us with the following details please :

• Are you running your workspace remotely or locally
• Could you show us the environment variables defined for your workspace
• can you show us your terraform aws provider configuration

Best regards !

I tried both the CLI workflow and local in Terrakube. Both results in the same error.

We have set the following environment variables:

And of course. It's here:

resource "aws_iam_openid_connect_provider" "provider" {
  url             = data.tls_certificate.terrakube_certificate.url
  client_id_list  = [var.terrakube_audience]
  thumbprint_list = [data.tls_certificate.terrakube_certificate.certificates[0].sha1_fingerprint]
}

resource "aws_iam_role" "role" {
  name = "terrakube-role-${var.env}"

  assume_role_policy = <<EOF
{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Principal": {
       "Federated": "${aws_iam_openid_connect_provider.provider.arn}"
     },
     "Action": "sts:AssumeRoleWithWebIdentity",
     "Condition": {
        "StringEquals": {
        "${var.terrakube_host}:aud": "${var.terrakube_audience}",
        "${var.terrakube_host}:sub": "organization:${var.terrakube_organization_name}:workspace:terrakube-test-repo"
        }
     }
   }
 ]
}
EOF
}

resource "aws_iam_policy" "policy" {
  name        = "terrakube-policy-${var.env}"
  description = "Policy for Terrakube IAM provider"
  policy      = data.aws_iam_policy_document.policy.json
}

data "aws_iam_policy_document" "policy" {
  statement {
    effect = "Allow"
    actions = [
      "s3:*",
    ]
    resources = ["*"]
  }
}

resource "aws_iam_role_policy_attachment" "policy_attachment" {
  role       = aws_iam_role.role.name
  policy_arn = aws_iam_policy.policy.arn
}

data "tls_certificate" "terrakube_certificate" {
  url = "https://${var.terrakube_host}"
}

I have been targeting the API when fetching the certificate thumbprint. I assume that's correct? :)

[jensrotne]

Maybe also one thing to note, I also found that it does actually persist the JWT token file in the executor pod just fine. Just seems like it might not be used?

[jensrotne]

My current working theory is that since we run it in EKS, then it inherits the role of node that it's running on superseeding the Terrakube STS role.

[alfespa17]

But in the end what terraform check is the environment variable here

terrakube/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java

Line 104 in b462866

workspaceEnvVariables.put("AWS_ROLE_ARN", workspaceEnvVariables.get("WORKLOAD_IDENTITY_ROLE_AWS"));

Maybe you can get the jwt token and decode it, check the token audience and subject match your policy

[alfespa17]

Hi @jensrotne !
Could provide us with the following details please :

• Are you running your workspace remotely or locally
• Could you show us the environment variables defined for your workspace
• can you show us your terraform aws provider configuration

Best regards !

I tried both the CLI workflow and local in Terrakube. Both results in the same error.

We have set the following environment variables:

And of course. It's here:

resource "aws_iam_openid_connect_provider" "provider" {
  url             = data.tls_certificate.terrakube_certificate.url
  client_id_list  = [var.terrakube_audience]
  thumbprint_list = [data.tls_certificate.terrakube_certificate.certificates[0].sha1_fingerprint]
}

resource "aws_iam_role" "role" {
  name = "terrakube-role-${var.env}"

  assume_role_policy = <<EOF
{
 "Version": "2012-10-17",
 "Statement": [
   {
     "Effect": "Allow",
     "Principal": {
       "Federated": "${aws_iam_openid_connect_provider.provider.arn}"
     },
     "Action": "sts:AssumeRoleWithWebIdentity",
     "Condition": {
        "StringEquals": {
        "${var.terrakube_host}:aud": "${var.terrakube_audience}",
        "${var.terrakube_host}:sub": "organization:${var.terrakube_organization_name}:workspace:terrakube-test-repo"
        }
     }
   }
 ]
}
EOF
}

resource "aws_iam_policy" "policy" {
  name        = "terrakube-policy-${var.env}"
  description = "Policy for Terrakube IAM provider"
  policy      = data.aws_iam_policy_document.policy.json
}

data "aws_iam_policy_document" "policy" {
  statement {
    effect = "Allow"
    actions = [
      "s3:*",
    ]
    resources = ["*"]
  }
}

resource "aws_iam_role_policy_attachment" "policy_attachment" {
  role       = aws_iam_role.role.name
  policy_arn = aws_iam_policy.policy.arn
}

data "tls_certificate" "terrakube_certificate" {
  url = "https://${var.terrakube_host}"
}

I have been targeting the API when fetching the certificate thumbprint. I assume that's correct? :)

It is the same from example using the API certificate

https://github.com/AzBuilder/terrakube/tree/main/dynamic-credential-setup/aws

[jensrotne]

But in the end what terraform check is the environment variable here

terrakube/api/src/main/java/org/terrakube/api/plugin/token/dynamic/DynamicCredentialsService.java

Line 104 in b462866

workspaceEnvVariables.put("AWS_ROLE_ARN", workspaceEnvVariables.get("WORKLOAD_IDENTITY_ROLE_AWS"));
Maybe you can get the jwt token and decode it, check the token audience and subject match your policy

Not sure I follow? What do you mean? :)

But what I did to test it was to extract the token from the terrakube_config_dynamic_credentials_aws.txt file from the executor, run aws sts assume-role-with-web-identity locally with my AWS CLI to get a session, and that works as it should.

[jensrotne]

We're getting closer to the root, but still quite far away it seems. So the problem is most likely not with the dynamic providers, but with the S3 configuration for the remote state. For some reason, every time we try and do anything, even without dynamic providers, then the executor pod is not allowed to communicate with our Terrakube S3 state bucket.

Seems like it could be the way that Terrakube works with IRSA in the pod that might cause the problem. We're still investigating.