[Proposal] Enhance CARM with team-id and service-level isolation

[a-hilaly]

Context

Currently, all the ACK controllers are shipped with CARM (Cross Account Resource Management) capabilities to extend the scope of controllers to manage resources across multiple AWS Accounts. This is achieved through a ConfigMap and namespace annotations.

The CARM ConfigMap allows users to specify AWS account IDs and the corresponding IAM Role ARNs that the controllers should assume when managing resources in those accounts. For example:

# CARM ConfigMap
data:
  "123456789012": "arn:aws:iam::123456789012:role/accountRole"

Additionally, users annotate their Kubernetes namespaces with the AWS account ID they want the controllers in that namespace to manage resources for. For example:

# Namespace Annotations
metadata:
  name: production
  annotations:
    services.k8s.aws/owner-account-id: "123456789012"

When a controller (watching production namespace) needs to manage resources, it looks up the CARM ConfigMap (more precisely the CARM Cache) for the accountID specified in the namespace annotation and retires the corresponding IAM Role ARN. The controller then needs to assume this role and pivot the client to start managing resources in that account.

Limitations

However, the current implementation has the following limitations:

• 1:1 namespace to AWS Account mapping, each namespace can only be mapped to a single AWS AccountID and a single IAM Role. And vice versa, an AccountID can only be mapped to one namespace. This prevents multiple teams (namespaces) from using different IAM Roles when managing resources in the same AWS Account. Forcing admins to often provide one RoleARN that works with all the provided namespaces.
  PS: We've heard of users hacking around this by specifying a different ID than an AWS account ID. While this approach works, it is not intuitive or recommended.
• Lack of service-level isolation: CARM also lacks the ability to assign different IAM roles to different service controllers (e.g s3 dynamodb controllers) handling resources in the same namespace. This means that all the service controllers (watching the same namespace) muist share the same IAM role, which contradicts the principle of least privilege and can result in giving extra/unnecessary permissions to some controllers.

These limitations can lead to overly broad permissions being granted, violating the principle of least privilege and making it challenging to manage resource across teams and services effectively.

Proposals

We propose an extension and backward-compatible solution to the current CARM model to address these limitations and allow users to continue fine-grain/isolating their AWS/Kubernetes environments.

Introduce a new team-id annotation

This will allow users to reuse the same account against multiple namespaces, but provide different roles for each team. For example, let's say we have two teams team-a and team-b both working with resources in the same account (11111111111). Admins could annotate their namespaces way

metadata:
  name: testing-a
  annotations:
    services.k8s.aws/team-id: "team-a"
---
metadata:
  name: testing-b
  annotations:
    services.k8s.aws/team-id: "team-b"

Another alternative would be to use services.k8s.aws/owner-team-id to keep consistency with services.k8s.aws/owner-account-id naming.

On other hand, the ConfigMap would then be updated to use team-id as a key and the associated RoleARN as value. For example:

# CARM ConfigMap
data:
  team-a: "arn:aws:iam::111111111111:role/team-a-role"
  team-b: "arn:aws:iam::111111111111:role/team-b-role"

Introduce a new service-prefixed-annotation

In addition to the team annotation, we suggest introducing a service-prefixed annotations and CARM entries. These annotations would allow you to specify different IAM roles for different service controllers whitin the same team and AWS Account.

For example, you might want the s3 controller to assume a different IAM Role than the dynamodb controller, even when managing resources in the same team/aws account.

The namespace annotations would look like this:

# Namespace for team-a
metadata:
  annotations:
    services.k8s.aws/team-id: "team-a-global"
    services.k8s.aws/team-id: "team-a"

Then, in the CARM ConfigMap, you can specify the IAM role ARNs for each service controller and team combination:

data:
  team-a: "arn:aws:iam::111111111111:role/team-a-global-role"
  s3.team-a: "arn:aws:iam::111111111111:role/team-a-s3-role"
  dynamodb.team-a: "arn:aws:iam::111111111111:role/team-a-dynamodb-role"

With this setup, the S3 controller would assume the team-a-s3-role when managing S3 resources for team-a, while the DynamoDB controller would assume the team-a-dynamodb-role when managing DynamoDB resources for team-a. Any other controller will assume team-a role

The order of precedence for the controller to pick up the role ARNs would be:

• Service controller specific team id (e.g., s3.services.k8s.aws/team-id)
• General team id (services.k8s.aws/team-id)

cc @mikestef9 @zicongmei @jlbutler @jose-fully-ported @eadasiak

[a-hilaly]

I'd like to utilize this GitHub issue as a space for brain storming and defining the general UX. Depending on the complexity, we may consider creating a separate design proposal PR for a more detailed discussion and implementation.

[nabuskey]

Overall, I like it. One suggestion I have is to reserve the possibility to expand the CM structure without disrupting users in the future. Having one to one mapping between role arn and team-id may prove too restricting.

data:
  team-a: |
    roleArn: "arn...."
    proxyConfig: ....

[zicongmei]

As we discussed in last week meeting. We can developed a v2 CARM map while still supporting the existing ack-role-account-map.

The new map will have the owner-account-id/ or team-id in the prefix such as

apiVersion: v1
kind: ConfigMap
metadata:
  name: ack-carm-map
  namespace: $ACK_SYSTEM_NAMESPACE
data:
  owner-account-id/111111111111: arn:aws:iam::111111111111:role/s3FullAccess
  team-id/team-a/role-arn: arn:aws:iam::111111111111:role/team-a
  team-id/team-a/endpoint: ...

We can easily expend in the future to the key such as team-id/proxyConfig/team-a if needed.

[a-hilaly]

This is currently available for all controllers under a new feature gate ServiceLevelCARM and TeamLevelCARM https://github.com/aws-controllers-k8s/s3-controller/blob/main/helm/values.yaml#L167-L168

[zarko-a]

@a-hilaly sorry to bring up the old topic. but we were curious if it's possible to use TeamLevelCARM to have ACK resources from a single namespace managed in multiple accounts?

We have an application living in a single namespace for which we'd like to provision IAM resources in multiple accounts. It's unclear based on the docs if this is even supported with the new feature gates.