[AWS] [EC2] enrich events with EC2 tags with add_cloud_metadata processor

[Kavindu-Dodan]

Proposed commit message

This PR adds support to enrich events with EC2 tags iff,

• IMDS endpoint is enabled ¹
• Tag access is enabled through IMDS endpoint. ²

Tags are added to event payload with key aws.tags.<KEY>,

{
  "aws" : {
   "tags" : {
      "org" : "acme",
      "owner" : "userA"
    }
   }
}

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

The best way to validate this is through a beats deployment (ex:- metricbeat) in an EC2 instance.

• Create an EC2 instance
  • Enable IMDS endpoint
  • Enable tags access through IMDS endpoint
  • Assign instance with a role with adequate permissions
  • Add few tags to the instance
• Build metricbeat from this branch
• Deploy custom build into EC2
• Enable add_cloud_metadata processor
• Run and observe metrics containing tags

Related issues

Closes #31899

Screenshots

Footnotes

1. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-options.html ↩

2. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/work-with-tags-in-IMDS.html#allow-access-to-tags-in-IMDS ↩

[elasticmachine]

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

[kaiyan-sheng]

Thanks for working on this @Kavindu-Dodan !!
@axw Do you have opinion on the naming for ec2 tags here from the processor?

Tags are added to event payload with key ec2.tag.<KEY>,
{
  "ec2" : {
   "tag" : {
      "org" : "acme",
      "owner" : "userA"
    }
   }
}

Just a reference: In AWS CloudWatch metricset, we are using aws.tags.<key>:<value>.

[axw]

@kaiyan-sheng my first reaction was that I think the tags field should be independent of the AWS service, because you can tag all/most AWS resources and not just EC2 instances. That could be useful generally for filtering metrics & logs by organisation or something.

OTOH, opentelemetry-collector-contrib's resourcedetection processor uses ec2.tag.*: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/40fa8b8a925cadf569e785cbc85d6dfca152bde2/processor/resourcedetectionprocessor/internal/aws/ec2/ec2.go#L29 (maybe that's what you modelled after @Kavindu-Dodan?)

OTOOH: it seems that resourcedetectionprocessor only supports adding tags for EC2, e.g. there's no support for tags for ECS: open-telemetry/opentelemetry-collector-contrib#14960.

So... I don't know, but I'd be inclined to go with aws.tags.<key>:<value>

[Kavindu-Dodan]

@kaiyan-sheng my first reaction was that I think the tags field should be independent of the AWS service, because you can tag all/most AWS resources and not just EC2 instances. That could be useful generally for filtering metrics & logs by organisation or something.

OTOH, opentelemetry-collector-contrib's resourcedetection processor uses ec2.tag.*: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/40fa8b8a925cadf569e785cbc85d6dfca152bde2/processor/resourcedetectionprocessor/internal/aws/ec2/ec2.go#L29 (maybe that's what you modelled after @Kavindu-Dodan?)

OTOOH: it seems that resourcedetectionprocessor only supports adding tags for EC2, e.g. there's no support for tags for ECS: open-telemetry/opentelemetry-collector-contrib#14960.

Yes @axw , I came across OTel resource detection processor EC2 tag [1] and tried to match our implementation with that. Currently processor also support Azure tags [2]

So... I don't know, but I'd be inclined to go with aws.tags.<key>:<value>

Since we already use aws.tags. with AWS Cloudwatch metrics and to have consistency, I will go ahead and use the same format for the processor implementation.

Update

I am testing the tag prefix change and will update the observations. Initially I am seeing some data ingestion issue but not sure this is related to tag key prefix.

Update - 2

Okay, it was a permission issue when starting metricsbeat inside my AWS ec2 testbed. Change works fine (see Kibana screenshot)

@kaiyan-sheng @axw appreciate another review

[1] - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.112.0/processor/resourcedetectionprocessor/internal/aws/ec2/ec2.go#L29
[2] - https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.112.0/processor/resourcedetectionprocessor/internal/azure/azure.go#L23