Service tag wrong randomly after restarts

[bsteve99]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Seemingly randomly my service tag comes out as _RESERVED_service: blah after restarts. But restart again and it might be correct as service: blah ( when sending logging requests to datadog ). Maybe a race condition..???

The condition will not fix itself until I continue to restart vector to get it to work correctly.

I've tried versions 0.39.0 to 0.44.0 with the same issue occurring.

I've done "tap's" and do not see any difference in the json log output (prior to sending to DD); when it works correctly vs when it does not work correctly.

Configuration

Version

0.44.0

Debug Output

Example Data

No response

Additional Context

This is running on Linux.

References

No response

[pront]

Hi @bsteve99, can you share your config (redact sensitive data first) and log samples so I can better understand the differences you attempted to describe above?

If I understand correctly, you are using a datadog_logs sink and it sounds related to this #22157 (comment).

[bsteve99]

I actually may have stumbled on a fix, with many restarts and no issues.

I was using datadog_log sink and re-wrote it in http sink; and its looking good.

old sink here:
_srv_dd_log_out:
    type: datadog_logs
    inputs:
      - _strip_verbose_log_lines
    compression: gzip
    default_api_key: "${DD_API_KEY:?DD_API_KEY must be specified}"
    tls:
      enabled: true
    batch:
      max_events: ${DATADOG_LOG_BATCHSIZE:-200}
      timeout_secs: 3
    buffer:
      type: memory
      max_events: 20
      when_full: block
    request:
      concurrency: adaptive

new sink here.
  _srv_dd_log_out:
    type: http
    inputs:
      - _strip_verbose_log_lines
    compression: gzip
    uri: https://http-intake.logs.datadoghq.com/api/v2/logs
    method: post
    tls:
      verify_certificate: false
      verify_hostname: true
    batch:
      max_events: ${DATADOG_LOG_BATCHSIZE:-200}
      timeout_secs: 3
    encoding:
      codec: json
    buffer:
      type: memory
      max_events: 20
      when_full: block
    request:
      headers:
        DD-API-KEY: "${DD_API_KEY:?DD_API_KEY must be specified}"
        Content-Type: application/json
        Accept: application/json
      concurrency: adaptive

[pront]

Sounds great. But also, I am curious why datadog_logs didn't work out of the box.

[bsteve99]

Yes, agree. But random _RESERVED_service should be enough info to see why this might happen randomly, as I said seems like a race condition that might be worth looking into.

The other detail I have is once this _RESERVED_service tag starts coming out, it never stops until we can restart vector. So some state is getting set; and never re-evaluating. Restart vector is the only solution; and once its fixed, it never messes it up ( until a restart may or may not cause it to be good or bad )

[pront]

I've done "tap's" and do not see any difference in the json log output (prior to sending to DD); when it works correctly vs when it does not work correctly. I also like adding a console sink with the same inputs as the sink I want to debug.

It would help debug this further if we had some sample events. With vector tap you can easily to get some, guide here.

I suspect that your source is populating the service field and the datadog_logs sink wants to use that field. In an effort to avoid data loss we move the existing value under the _RESERVED_service event path and populate service with a new value.

If you are interested, the code is here:

vector/src/sinks/datadog/logs/sink.rs

Lines 153 to 165 in 68ee4f2

	// if an existing attribute exists here already, move it so to not overwrite it.
	// yes, technically the rename path could exist, but technically that could always be the case.
	if log.contains(desired_path) {
	let rename_attr = format!("_RESERVED_{}", meaning);
	let rename_path = event_path!(rename_attr.as_str());
	warn!(
	message = "Semantic meaning is defined, but the event path already exists. Renaming to not overwrite.",
	meaning = meaning,
	renamed = &rename_attr,
	internal_log_rate_limit = true,
	);
	log.rename_key(desired_path, rename_path);
	}

More debugging tools:

• vector top to see component metrics - docs here
  • For example, you can inspect how many events are produced by sources vs how many events reach the sinks. In your case, you can compare http vs datadog_logs sink metrics.
  • You can also add an internal_metrics source and inspect the metrics

[gautamg795]

I don't have events I can share at the moment, but I'm seeing this as well after upgrading from 0.36.0 to 0.44.0.
Some vector instances just get into this "bad" state where service gets put into _RESERVED_service and service becomes empty.

Also notable is that I set a host_key: instance_id setting in the log schema settings, but for these vector instances in the "bad " state, there is no host field in event on Datadog -- just the original instance_id field.

[gautamg795]

I can't explain the nondeterministic behavior yet, but this seems to have been introduced by 38f4868
I haven't tested enough to understand if the bug is triggered by a different commit entirely, but this is the only commit that seems to care about the .service field. This commit was released in v0.38.0, which tracks with the OP seeing this behavior from v0.39.0-v0.44.0

I'm not too familiar with log namespacing and semantic meaning as the features were added long after we started using Vector, but if I'm specifically setting .service in a remap transform (because I want it to show up on DataDog as such), that should be fine? Given that .service is empty when this bug occurs, I'm not sure what value Vector is moving my intended .service "out of the way" for.

[pront]

I don't have events I can share at the moment, but I'm seeing this as well after upgrading from 0.36.0 to 0.44.0.
Some vector instances just get into this "bad" state where service gets put into _RESERVED_service and service becomes empty.

Do you have any sample logs that you can share?
The sink will create a _RESERVED_service field, if an existing attribute exists on service already but the meaning path points to another field. We want to use the value pointed to by the meaning path.

I'm not too familiar with log namespacing and semantic meaning as the features were added long after we started using Vector, but if I'm specifically setting .service in a remap transform (because I want it to show up on DataDog as such), that should be fine? Given that .service is empty when this bug occurs, I'm not sure what value Vector is moving my intended .service "out of the way" for.

This is correct. When you set .service = "foo", you set a reserved attribute.

https://github.com/vectordotdev/vector/blob/master/src/common/datadog.rs#L21-L31

The following info on how to manually set reserved attributes might be helpful. Assuming you didn't turn ON the log namespacing feature.

1. Set the special paths:

   .ddsource = "my-custom-source-value"
   .host = "my-custom-host-value"

2. Using semantic meanings (mappings):

set_semantic_meaning(%source, "source")
%source = "my-custom-source-value"

set_semantic_meaning(%host, "host")
%host = "my-custom-host-value"

Note that some sources set semantic meanings by default, for example the file source assigns the message meaning to the . event path.

[gautamg795]

Yeah, so here's a sample log from vector tap -- after the remap, but before the datadog sink.

{
  "container_id": "07e3a759375003291f35aa16185d8da728a02d0fdcc0c3cfff18353df2fdef68",
  "container_name": "plugin-0341f3f7-bd97-ee13-4186-5a2b85c4c7a0",
  "ddsource": "plugin",
  "facility": "daemon",
  "instance_id": "i-03fe1261deda234ab",
  "message": "I0225 22:26:05.901647       1 node.go:565] \"NodeGetCapabilities: called\" args={}",
  "nomad_alloc_id": "0341f3f7-bd97-ee13-4186-5a2b85c4c7a0",
  "nomad_job_name": "plugin-aws-ebs-nodes",
  "nomad_namespace": "internal",
  "nomad_node_name": "i-03fe1261deda234ab",
  "nomad_task_group_name": "nodes",
  "nomad_task_name": "plugin",
  "source_ip": "127.0.0.1",
  "source_type": "syslog",
  "service": "plugin",
  "timestamp": "2025-02-25T22:26:05Z"
}

My config has

log_schema:
  host_key: instance_id

So instance_id should be grabbed as the host (i.e. when it works, vector seems to rename this field to hostname when sending to Datadog)

I've manually set service in the remap transform (e.g. .service = .some_parsed_value.service_name).
So sometimes this event ends in Datadog as I'd expect, and sometimes Vector gets into a state where the event that makes it to Datadog contains instance_id (and no host/hostname field) and my service value gets renamed to _RESERVED_service but there's no service field in the event.

I've worked around the latter by setting .svc instead in my remap transform, and calling set_semantic_meaning(.svc, "service").
I could have done the same for the former, though I didn't notice this bug until later so for now, I've just edited the log pipeline on the Datadog side to accept instance_id as a host field.

[gautamg795]

hey @pront, any thoughts on this? from your explanation and my symptoms it sounds like this is a bug rather than user error, but I'm happy to try any other suggestions as well.
as mentioned we've sidestepped the issue for now by using .svc instead of .service, but because this remap transform is used for many other sinks, we've had to deal with the consequences of renaming that field elsewhere too. we'd love to be able to stick to .service (and not lose .hostname) as we could on 0.36.0

[pront]

Hi @gautamg795, I took another look and I think the sink is working as designed.

we'd love to be able to stick to .service (and not lose .hostname) as we could on 0.36.0

You can keep using .service AFAIK. The move & rename to _RESERVED_service only happens if the service path is taken and the semantic meaning for service maps to another path. This might be confusing so here is an example:

1. The service path is set to vector:

{
  "host": "localhost",
  "message": "{ \"bar\": \"bar_value\"}, \"foo\": \"foo_value\"}",
  "service": "vector",
  "source_type": "demo_logs",
  "timestamp": "2025-03-10T16:32:41.949076Z"
}

2. The remap source code maps the service meaning:

transforms:
  t0:
    type: remap
    inputs:
      - s0
    source: |
      # This is important because the original message comes in bytes
      .message = parse_json!(.message)

      .message.foo = "another service"
      set_semantic_meaning(.message.foo, "service")

3. Then when the datadog logs sink processes an event:

2025-03-10T16:32:43.115041Z  WARN vector::sinks::datadog::logs::sink: Semantic meaning is defined, but the event path already exists. Renaming to not overwrite. meaning="service" renamed="_RESERVED_service" internal_log_rate_limit=true

So we end up with:

_RESERVED_service: "vector",
service: "another service"

The assumption here is that events have the same formats, if some events have different format e.g. some set service and you see _RESERVED_service and some aren't so there's no move. We were solely focused on the sink behavior but the source is equally important here. You can add some defensive checks in your VRL to detect if that happens.

[gautamg795]

The move & rename to _RESERVED_service only happens if the service path is taken and the semantic meaning for service maps to another path.

I agree with this part -- but in the events where I'm seeing the warning log line + move to _RESERVED_service, there is no .service key in the resulting event.
i.e. my value is getting "moved out of the way" (as expected if there were another semantic "service value") but the resulting event sent to Datadog has no service at all

And as @bsteve99 experienced, this isn't deterministic -- sometimes Vector would work just fine on one machine, then at some point (without any clear change in the logs on this machine) would start moving .service to .RESERVED_service and never revert back to the original behavior until a restart