Clarification on Log Payload Size Limit and Truncation Behavior for Datadog Logs Sink

[EricaJ6]

Question

Hi Vector team,

I'm looking for some clarification regarding the payload size behavior in the datadog_logs sink.

According to the Vector documentation, the batch.max_bytes value for the datadog_logs sink is set to 4.25MB by default.
From the source code (ref: logs/config.rs#L30-L31), it appears that the maximum payload size is hardcoded to 5MB, and the default batch size is set to 4.25MB:

// The Datadog logs API has a hard payload limit of 5MB.
pub const MAX_PAYLOAD_BYTES: usize = 5_000_000;
pub const BATCH_GOAL_BYTES: usize = 4_250_000;

I have a few questions related to this behavior:

1. Is the 5MB log payload limit hardcoded and non-configurable, or is there a way to override it via configuration?
2. When a single log event exceeds the 5MB limit, will Vector truncate the log and append "[TRUNCATED]" at the end of the payload?
3. If truncation occurs, what is the recommended approach to avoid this? Should we proactively split large log messages upstream, or is there a configuration or transform that can help mitigate this?

Appreciate any insights or best practices you can share for handling large log events in this scenario.

Vector Config

No response

Vector Logs

No response

[thomasqueirozb]

Hi @EricaJ6

#1: There is no way to override this. Looks like it's a Datadog limit, increasing this value probably will only lead to logs being rejected by the DD Api.

Regarding your other questions: do you think you will have a single log event that exceeds 4MB? That seems like an astronomically large number for a single line, and at that point I'd even question how valuable storing that log line actually is.

To answer question #2 some more investigation is needed. I'm not sure the what output would be but I think you can artificially replicate it using something like a 6MB line in a file and using the file source as an input. I'd wager you get an error log.

To answer question #3: I'd recommend splitting events this big if you really want to keep a line that's over 4MB long. You can probably do that from within vector by using VRL. Example:

data_dir: /tmp/vector_data_dir/

api:
  enabled: true

sources:
  in:
    type: "file"
    include:
      - "/tmp/in"

transforms:
  t0:
    type: "remap"
    inputs: ["in"]
    source: |
      . = split(string!(.message), " ")

sinks:
  out:
    type: "console"
    inputs: ["t0"]
    encoding:
      codec: "json"
      json:
        pretty: true

By writing first second third into /tmp/in you'll get 3 events into the console

[EricaJ6]

Thank you @thomasqueirozb for looking into this further.
In my case, I observed the "[TRUNCATED]" message in the Datadog logs even before explicitly using the truncate
VRL function. When I intentionally sent a large payload (over 6 MB) to Vector as part of an experiment, I was unable to reproduce that behavior. Currently, similar large messages are simply being dropped without any "[TRUNCATED]" indication in the logs.

Do you have any insight into what might be causing this change in behavior?

[pront]

Note: Sharing full Vector config is very helpful for us. We highly encourage users to include their config (sensitive data redacted first) when they initiate a discussion, thanks!

[thomasqueirozb]

Agree with what @pront said, very difficult to root cause without a config. Although I suspect the [TRUNCATED] might be added by Datadog and not by vector. You can add a separate sink (file, s3, kafka, console, etc.) with the same transform to debug this and compare outputs with what you see in DD, as there is no clear indication that this is being added by Vector.

[EricaJ6]

@thomasqueirozb As the logs are being dropped by the Vector sink, they are not reaching Datadog. This raises the question: how is Datadog able to append the [TRUNCATED] tag?

I tested writing the output to a file and observed no issues—logs were written in full without truncation.

I'll share the Vector configuration currently deployed in our EKS cluster. I've removed some of the transform stages that enrich logs with additional tags for clarity.

    api:
      address: 0.0.0.0:8686
      enabled: true
      playground: false
    data_dir: /custom-data-dir
    healthchecks:
      enabled: true
      require_healthy: false
    sinks:
      datadog_sink_1:
        buffer:
          max_size: 500000000
          type: disk
          when_full: block
        compression: gzip
        default_api_key: ${DATADOG_API_KEY_1}
        inputs:
          - redact_dd_agent_status_logs
        site: datadoghq.com
        type: datadog_logs
      datadog_sink_2:
        buffer:
          max_size: 268435500
          type: disk
          when_full: drop_newest
        compression: gzip
        default_api_key: ${DATADOG_API_KEY_2}
        inputs:
          - logs_route.myroute
        site: datadoghq.com
        type: datadog_logs
    sources:
      datadog_log_agent:
        address: 0.0.0.0:80
        disable_logs: false
        disable_metrics: true
        disable_traces: true
        multiple_outputs: false
        store_api_key: false
        type: datadog_agent
    transforms:
      redact_dd_agent_status_logs:
        inputs:
          - datadog_log_agent
        source: |
          del(.status)
        type: remap
      logs_route:
        inputs:
          - redact_dd_agent_status_logs
        route:
          myroute: exists(.namespace)
        type: route

[thomasqueirozb]

As I mentioned before, Vector doesn't seem to append [TRUNCATED] anywhere (unless the truncate function is explicitly called of course). If you take a look here we emit a AboutToSendHttpRequest event before sending the request, which will debug log the request. I'd suggest setting VECTOR_LOG=debug to view the request and if you want to be absolutely sure what is being sent from Vector to Datadog.