Support additional, custom AWS resource tags for CloudWatch metrics

[maxmoehl]

Feature Request

We would like to enrich metrics from the AWS CloudWatch input about EC2 resources with their respective tags. However, the current plugin aws_ec2 does not support this. From what I understand it is using the AWS Instance Metadata to detect which instance it's running on and selects the tags based on that. In our case Telegraf runs on a dedicated VM and collects the metrics from CloudWatch, therefore, the existing plugin cannot be used to collect tags.

Proposal:

Have a way to enrich metrics that are coming from CloudWatch with tags from the resource the metric is from. Either as part of the existing CloudWatch plugin or as an additional processor (which would heavily rely on the CloudWatch input plugin).

Current behavior:

Metrics from AWS CloudWatch only contain the resource name, instance id and region:
cloudwatch_aws_nat_gateway,nat_gateway_id=nat-0a99753733657e0f5,region=eu-central-1 active_connection_count_sum=198 1644743280000000000

Desired behavior:

Metrics from AWS CloudWatch contain arbitrary, configurable labels from the origin resource (as well as resource name, instance id and region) e.g. foo=bar:
cloudwatch_aws_nat_gateway,foo=bar,nat_gateway_id=nat-0a99753733657e0f5,region=eu-central-1 active_connection_count_sum=198 1644743280000000000

Use case:

Many instances of the same resource run in a single account and region. Being able to divide the metrics from them into groups by custom tags would greatly improve the observability of the individual deployments a resource belongs to.

Steps taken:

As a first step I copied the existing aws_ec2 plugin and stripped all the instance identity stuff from it. It now collects the instance id from the given metric and adds any configured tags. See POC in my fork

Edit: I implemented a tag cache that does most of the required work, but still has some open questions.

I'm happy to work on this and provide a solution (this time with a signed CLA :D). I'm not sure whether a new plugin or extending the existing CloudWatch plugin would be more desirable, I'm open for suggestions.

[vipinvkmenon]

+1 for this feature!

[sri-shetty]

+1 for this feature. When could we expect to see this feature?