[resourcedetectionprocessor] Gather EC2 tags

[mx-psi]

Is your feature request related to a problem? Please describe.

When using AWS resources like EC2 instances it is common to tag them with metadata that can help identify a specific instance by e.g. owner or environment. Having this information together with telemetry can help in discoverability and enrich their context.

Describe the solution you'd like

It would be useful to add this metadata to the resource detection processor (e.g. in the EC2 detector) so that this information can be added to telemetry data like metrics, traces or logs to quickly identify the source of them.

Describe alternatives you've considered

A new component could be created to do this, but I think it makes more sense to add it to this existing one.

Additional context

These are key-value Unicode string pairs of limited size that can be added to many AWS resources. To be accessed a IAM role giving access to DescribeTags needs to be set up by the user. This would mean that it could be an opt-in since it needs manual setup.

[rakyll]

@mx-psi do you need a solution where you add the tag keys you want to the config file and the resourcedetectionprocessor fetches and adds them as resources? Or are you looking for something else?

[mx-psi]

@rakyll Thanks for having a look at this!

do you need a solution where you add the tag keys you want to the config file and the resourcedetectionprocessor fetches and adds them as resources?

That could definitely work, but I think it would be easier to set up for end users if all available tags are added (if that is technically possible and you see no concerns)

[rakyll]

We probably don't want to attach all the available tags because there might be way too many of them and user can't always can alter/optimize tags for OpenTelemetry collector.

[mx-psi]

I see, having a list of tag keys to be added works then

[amanbrar1999]

Is the idea here that the collector is running in an EC2 instance and we want to get the EC2 instance's tags? or the data ingested by the collector came from an EC2 instance?

[mx-psi]

Is the idea here that the collector is running in an EC2 instance and we want to get the EC2 instance's tags? or the data ingested by the collector came from an EC2 instance?

@amanbrar1999 the former: the Collector is running in an EC2 instance and we want to get the tags associated to that instance.

[mx-psi]

We probably don't want to attach all the available tags because there might be way too many of them and user can't always can alter/optimize tags for OpenTelemetry collector.

@rakyll It looks like at Datadog we optionally pull all tags and we have users that use this successfully, so it might not be a bad idea to at least have the option to add all tags.

The relevant code of the Datadog Agent is here; it could be used as a start point to implement this (in whichever version), although I think the token handling should be done using as-sdk-go if possible (I am not sure how one does that).

[rakyll]

@mx-psi We can make it an opt-in. If user provide a setting explicitly, we can pull them all. @amanbrar1999 what do you think?

[amanbrar1999]

For sure that can be done. My proposal is to add optional configs for each detector, so within the EC2 detectors config we can provide both options to pull all tags or provide a list of tags. I have the idea in mind but haven’t quite gotten around to coding it, hopefully will have some more time next week.

[mx-psi]

We can make it an opt-in. If user provide a setting explicitly, we can pull them all.

👍, that sounds good to me