Feature request: add option to locally inject tags in dogstatsd before forwarding

[ewdurbin]

Describe what happened:

I set DD_TAGS to a set of tags joined by spaces and kicked off dogstatsd

Describe what you expected:

I expected the tags to be added before metrics were submitted to DataDog's endpoints

Steps to reproduce the issue:

export DD_API_KEY=deadbeefdeadbeefdeadbeef
export DD_TAGS="simple-tag-0 key-tag-0:value-0"
./dogstatsd

Then submit some metrics, note that nothing from DD_TAGS is added before submitting

[xvello]

Hello @ewdurbin ,

Host tags set via DD_TAGS are not added to every timeseries, but are sent in the host metadata payload, to be injected by our backend. You can have a look at the Sent host metadata payload, content debug log line to confirm they are.

Could you please confirm your tags are not visible in the metric summary page?

[ewdurbin]

@xvello in this instance I'm not sending host metadata and just want to run dogstatsd as a sidecar to specific services to isolate them to a separate account/key, attaching tags managed by the infrastructure along the way.

this is a departure from the previous agent as far as I'm aware and would be a nice feature, even if behind a separate flag.

[xvello]

Please note that disabling DD_ENABLE_METADATA_COLLECTION is only supported if another agent/dsd is running on the host with host metadata collection turned on. The host tags of the main agent would then be used for all metrics sent on that hostname.

As the DD_TAGS envvar binding works as intended, would you be OK to rename the issue "Feature request: add option to locally inject tags before forwarding" ?

[ewdurbin]

@xvello in this case, the datadog/dogstatsd is running as a sidecar inside a Kubernetes Pod for whom tags from the host should not be exposed, instead I want to expose "obfuscated" tags for them. The daemonset running the full agent is running on each host, but again I don't want these pods to submit metrics tagged with the underlying infra.

Renaming the issue is fine, but I'd like to restate that previous versions of the agent attached configuration from tags in the conf to emitted metrics.

[dhawal55]

+1

[2rs2ts]

Given that host aliases aren't unique e.g. in EC2 not having these tags sent with the timeseries causes metrics from hosts in other VPCs to get confused with each other and it's extremely inconvenient when this pops up which is fairly regularly. I think the rename proposed is fine, but please implement this 🙏

[olivielpeau]

Hi @2rs2ts, thanks for chiming in!
I'm not sure I understand your use case: the hostname reported by the Agent from every host should be unique (across VPCs), and as long as you send dogstatsd metrics to the Agent without an explicit hostname, the Agent's hostname will be attached to these dogstatsd metrics. So this Agent's host tags should then be associated with these metrics in Datadog. You mention host aliases, but I don't think they're relevant for your problem unless you're explicitly sending your dogstatsd metrics with one of these host aliases as the hostname. Is there anything I'm missing though?

@ewdurbin Sorry, only reading through your feature request now. AFAIK previous versions of the Agent did not attach tags directly to the emitted metrics, but relied on the same mechanism as Agent/Dogstatsd v6 (i.e. host tags associated in Datadog's backend). Do you have any details on how you set up your previous Agents and how it differs from your current setup?

[ewdurbin]

@olivielpeau Your explanation of previous versions makes sense, as I previously configured the tags value in the configuration file but simply disabled as much as I could except for dogstatsd.

The main thing I think that lead to this feature request is that the core agent in dd-agent v6 accepts tags via the DD_TAGS environment variable, but the dogstatsd entrypoint doesn't read them.

For use cases that don't necessitate the full agent, the dogstatsd docker image is rather useful, but doesn't accept the same DD_TAGS env var as the full agent docker image.

This issue is requesting that the dogstatsd only image accept DD_TAGS in the same way that the full agent image does.

[olivielpeau]

Thanks @ewdurbin!

I read the whole convo again and noticed that you disabled host metadata collection (with DD_ENABLE_METADATA_COLLECTION) on your dogstatsd sidecar, which explains why the tags in DD_TAGS/tags aren't sent to Datadog.

So this feature request is indeed something we the Agent doesn't support at the moment. The feature could work like this:

• a new option would be exposed by the dogstatsd image, which would accept a list of tags
• this list of tags would be injected to all incoming dogstatsd metrics before they're forwarded to Datadog, regardless of the host: tag attached to these incoming metrics (if any). Note that this would be a different behavior compared to the current Agent tags option: the tags only get associated to the incoming metrics that either have no host: tag, or a host: tag set to the Agent's hostname or one of its host aliases.

Does this look like a good description of the feature you'd like?

[ewdurbin]

@olivielpeau that sounds spot on, thanks.

[2rs2ts]

@olivielpeau

You mention host aliases, but I don't think they're relevant for your problem unless you're explicitly sending your dogstatsd metrics with one of these host aliases as the hostname. Is there anything I'm missing though?

I have an open ticket about this (152366), your backend treats host aliases as the same thing as the primary hostname value for hosts, ergo we get alerts for hosts that don't post metrics we're alerting on, even when we group by a tag that should only exist on the hosts that do post the metrics. Yes we are absolutely certain we use EC2 instance ID for the primary hostname, but it does not matter, host aliases screw it up. This happens from metrics and events sent by agent checks, autodiscovery, and metrics sent via statsd, and it affects both 5.x and 6.x. Quite a pickle, and it wasn't a problem until maybe 4 months ago, so I assume it is backend changes... but you could make your backend problems easier to solve if you attached these tags on the metrics you send to your backend to disambiguate them.

[olivielpeau]

@2rs2ts Thanks for your response. From what I've checked there hasn't been any backend change that could explain what you're experiencing: host aliases are only used so that metrics coming in with a host alias can be attached to the host that has that host alias (and once the metric is attached to the host, the host tags of that host are applied to the metric).

I don't think the feature request here is going to solve the issue you're experiencing with host aliases on 5.x and 6.x agents for metrics coming from both dogstatsd and agent checks. Could you share more information about what you're experiencing in the open ticket you have with our support team? (in particular, instances of metrics from 5.x agent checks being attached to the wrong host, or of dogstatsd metrics coming attached to wrong host tags). Thanks!

[2rs2ts]

Yeah we already sent the flares and everything... if this feature request won't help with my situation then ok, that's a bummer, sorry for the distraction 😅

[nevir]

So this feature request is indeed something we the Agent doesn't support at the moment. The feature could work like this:

• a new option would be exposed by the dogstatsd image, which would accept a list of tags
  this list of tags would be injected to all incoming dogstatsd metrics before they're forwarded to Datadog, regardless of the host: tag attached to these incoming metrics (if any). Note that this would be a different behavior compared to the current Agent tags option: the tags only get associated to the > * incoming metrics that either have no host: tag, or a host: tag set to the Agent's hostname or one of its host aliases.

We'd like this as well: mainly, we generally don't find the host tag useful for custom metrics (nearly all our metrics come from containers), and would like to reduce cardinality

[millerick]

My organization would also like a way to eliminate the host tag from custom metrics submitted by our applications running inside of containers. They can be scheduled on any host of our application cluster, and the host that they are running on is not particularly meaningful to us, and only serves to increase our tag cardinality (and therefore bill).

We have tried manually adding a fake value for the host tag for custom metrics submitted from within our applications, but we have observed that results in many missing metrics in the datadog UI, even though they all should be received by the agent.