Datadog next gen keeps the lambda alive waiting for DogStatsD events

[tommy-ng0]

Hi team, I’m observing some unexpected behavior with the Datadog Lambda extension (82-next) and would appreciate any insights.

We’re using the end flush strategy, so we expect the extension to send all metrics at the end of the invocation. The extension flushes successfully, and then continues to receive DogStatsD traffic for another ~25 seconds, which keeps the Lambda function alive longer than necessary. Weirdly enough, this behaviour only seems to occur once per day at around the same time.

Here is out setup:

DD_SERVERLESS_FLUSH_STRATEGY: end
DD_TRACE_OTEL_ENABLED: false 
DD_PROFILING_ENABLED: false 
DD_SERVERLESS_APPSEC_ENABLED: false
DD_TRACE_ENABLED: false 
DD_HTTP_PROTOCOL: http1

Here are the relevant logs:

...
2025-07-03T08:11:08.705Z+12:00: DD_EXTENSION | DEBUG | Telemetry event received: TelemetryEvent { time: 2025-07-02T20:11:08.678Z, record: PlatformRuntimeDone { request_id: "133260e5-3e10-42b9-9d2a-beafec7345a5", status: Success, error_type: None, metrics: Some(RuntimeDoneMetrics { duration_ms: 220.626, produced_bytes: Some(17454) }) } }
...
2025-07-03T08:11:08.707+12:00: DD_EXTENSION | DEBUG | Flushing 1 series and 1 distributions
2025-07-03T08:11:08.708+12:00: DD_EXTENSION | DEBUG | Sending body: Series { series: ... }
2025-07-03T08:11:08.709+12:00: DD_EXTENSION | DEBUG | Sending distributions: SketchPayload { ... }
2025-07-03T08:11:08.710+12:00: DD_EXTENSION | DEBUG | Request to http://proxy/api/v2/series took 2ms
2025-07-03T08:11:08.712+12:00: DD_EXTENSION | DEBUG | Request to http://proxy/api/beta/sketches took 3ms
2025-07-03T08:11:08.712+12:00: DD_EXTENSION | DEBUG | Successfully flushed 1 series and 1 distributions
2025-07-03T08:11:09.064+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.metrics:18|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.064+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.events:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_sent:25|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:gauge
2025-07-03T08:11:09.065+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:count
2025-07-03T08:11:09.065+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:histogram
2025-07-03T08:11:09.065+12:00:	datadog.dogstatsd.client.metrics_by_type:18|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:distribution
2025-07-03T08:11:09.065+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:set
2025-07-03T08:11:09.065+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:timing
2025-07-03T08:11:09.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.service_checks:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.aggregated_context:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:gauge
2025-07-03T08:11:09.164+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:set
2025-07-03T08:11:09.164+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:count
2025-07-03T08:11:09.164+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:histogram
2025-07-03T08:11:09.164+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:distribution
2025-07-03T08:11:09.164+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:timing
2025-07-03T08:11:09.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.metric_dropped_on_receive:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_dropped:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.bytes_dropped:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.bytes_dropped_queue:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_dropped_queue:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.bytes_sent:4806|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.bytes_dropped_writer:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:09.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_dropped_writer:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:09.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.metrics:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	datadog.dogstatsd.client.metric_dropped_on_receive:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_dropped:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.bytes_dropped:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.events:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_sent:15|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	datadog.dogstatsd.client.bytes_dropped_queue:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_dropped_queue:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:gauge
2025-07-03T08:11:19.065+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:count
2025-07-03T08:11:19.065+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:histogram
2025-07-03T08:11:19.065+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:distribution
2025-07-03T08:11:19.065+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:set
2025-07-03T08:11:19.065+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:timing
2025-07-03T08:11:19.065+12:00:	datadog.dogstatsd.client.bytes_sent:2777|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.bytes_dropped_writer:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_dropped_writer:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.service_checks:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.aggregated_context:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:19.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:19.164+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:gauge
2025-07-03T08:11:19.164+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:set
2025-07-03T08:11:19.164+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:count
2025-07-03T08:11:19.164+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:histogram
2025-07-03T08:11:19.164+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:distribution
2025-07-03T08:11:19.164+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:timing
2025-07-03T08:11:19.164+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.aggregated_context:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.065+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:gauge
2025-07-03T08:11:29.065+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:set
2025-07-03T08:11:29.065+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:count
2025-07-03T08:11:29.065+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:histogram
2025-07-03T08:11:29.065+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:distribution
2025-07-03T08:11:29.065+12:00:	datadog.dogstatsd.client.aggregated_context_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:timing
2025-07-03T08:11:29.065+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.066+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.metrics:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	datadog.dogstatsd.client.metric_dropped_on_receive:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.066+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_dropped:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.066+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.bytes_dropped:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.066+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.events:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.066+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_sent:12|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	datadog.dogstatsd.client.bytes_dropped_queue:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.066+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_dropped_queue:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.066+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:gauge
2025-07-03T08:11:29.066+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:count
2025-07-03T08:11:29.066+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:histogram
2025-07-03T08:11:29.066+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:distribution
2025-07-03T08:11:29.066+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:set
2025-07-03T08:11:29.066+12:00:	datadog.dogstatsd.client.metrics_by_type:0|c|#client:go,client_version:5.4.0,client_transport:udp,metrics_type:timing
2025-07-03T08:11:29.066+12:00:	datadog.dogstatsd.client.bytes_sent:2775|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.066+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.bytes_dropped_writer:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.066+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.packets_dropped_writer:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:29.066+12:00:	DD_EXTENSION | DEBUG | Received message: datadog.dogstatsd.client.service_checks:0|c|#client:go,client_version:5.4.0,client_transport:udp
2025-07-03T08:11:29.066+12:00:	from 127.0.0.1:54984
2025-07-03T08:11:33.283+12:00:	END RequestId: 133260e5-3e10-42b9-9d2a-beafec7345a5

Has anyone seen something like this before? Could there be some background process or buffering behavior that delays the final DogStatsD traffic?

[astuyve]

Oh Tommy I just realized I led you astray, can you open a support ticket via support@datadoghq.com? Also maybe one thing to try is setting DD_INSTRUMENTATION_TELEMETRY_ENABLED: false

Thanks!
AJ

[astuyve]

hey @tommy-ng0 I haven't heard back and haven't been able to repro this, so I'm gonna close it for now. Give me a shout if you're still fighting this!

Thanks!