Kafka Consumer check: Add support for AWS MSK IAM (SASL/IAM) authentication to enable Kafka UI / cluster monitoring on MSK IAM-only clusters

[fedeostrit]

Hi 👋

We’re running Apache Kafka on Amazon MSK with IAM authentication enabled (MSK IAM / AWS SigV4) and we’d like to use the kafka_consumer integration (as a Cluster Check via Kubernetes annotations) to power Kafka UI (preview) / cluster monitoring in Datadog.

Today, the kafka_consumer check configuration supports common SASL mechanisms like PLAIN, GSSAPI, SCRAM-*, and OAUTHBEARER with an OAuth2 token endpoint (url, client_id, client_secret). However, MSK IAM authentication is not a standard OAuth2 flow and typically requires generating AWS SigV4-based tokens (e.g., for SASL_OAUTHBEARER token provider or AWS_MSK_IAM mechanism depending on client language). Because of this, we can’t configure the Agent’s Kafka admin client to authenticate to MSK using our IAM roles.

What we need

Add first-class support in the kafka_consumer check for connecting to Amazon MSK using IAM roles (IRSA / instance role / assumed role), so Datadog can retrieve cluster metadata and consumer group information from MSK clusters secured with IAM.

Concretely, we’d like one of these options:

Support MSK IAM token generation for SASL/OAUTHBEARER

Allow sasl_mechanism: "OAUTHBEARER" with a non-OAuth2 token provider that generates MSK IAM tokens (AWS SigV4).

The Agent should use the AWS credential chain (IRSA, env, instance profile, etc.) to sign tokens.

Support the MSK IAM SASL mechanism explicitly

Allow something like sasl_mechanism: "AWS_MSK_IAM" (or a dedicated config block) and implement the auth flow internally.

Suggested configuration shape (example)

Something along these lines (flexible on exact schema):

{
  "kafka_consumer": {
    "instances": [{
      "kafka_connect_str": "<MSK_BOOTSTRAP_BROKERS>",
      "security_protocol": "SASL_SSL",
      "sasl_mechanism": "AWS_MSK_IAM",
      "aws": {
        "region": "us-east-1",
        "auth": "default", 
        "role_arn": "arn:aws:iam::<account>:role/<optional-assume-role>"
      },
      "enable_cluster_monitoring": true
    }]
  }
}

Notes:

auth: default should use the standard AWS credential provider chain (including IRSA on EKS).

role_arn would be optional, for environments that require assume-role into a monitoring role.

This would let us run the check as a Kubernetes Cluster Check where the Agent/Cluster Agent pod has the IAM permissions via IRSA.

Why this matters

Many MSK deployments (especially MSK Serverless) are IAM-only for authentication.

Datadog’s Kafka UI / cluster monitoring workflows rely on the Agent being able to connect as an admin client.

Without IAM support in the check, users are forced to either:

enable alternative auth methods (not always possible), or

run a separate Kafka UI outside Datadog, losing the integrated experience.

Environment details

Platform: Kubernetes (EKS)

Auth: Amazon MSK IAM (SigV4), credentials via IRSA / IAM roles

Goal: kafka_consumer check can connect to MSK and collect cluster/consumer group metadata for Kafka UI / monitoring

Ask

Is MSK IAM support on the roadmap for kafka_consumer?
If not, would you accept a contribution / design proposal? I’m happy to provide more details, test results, or help validate a PR once there’s guidance on the preferred approach.

Thanks!