Skip to content

aws.dd_forwarder.logs_failed and metrics_failed are never emitted due to passing list instead of len() to send_event_metric #1148

Description

@patbos

Bug

In aws/logs_monitoring/forwarder.py, send_event_metric is called with the failed items list as the metric value instead of its length:

# Line 141 - passes list, should be len()
send_event_metric("logs_failed", failed_logs)

# Line 172 - passes list, should be len()
send_event_metric("metrics_failed", failed_metrics)

Compare with the correct pattern used on the adjacent lines:

send_event_metric("logs_forwarded", len(logs_to_forward) - len(failed_logs))
send_event_metric("metrics_forwarded", len(metrics) - len(failed_metrics))

Root cause

send_event_metric calls lambda_metric, which validates the value with float(value). Passing a list raises TypeError, which is caught and logged as a warning, then the metric is silently dropped:

[WARNING] Ignoring metric submission for metric 'aws.dd_forwarder.logs_failed'
because the value is not numeric: [...]

See datadog_lambda/metric.py:

try:
    float(value)
except (ValueError, TypeError):
    logger.warning(
        "Ignoring metric submission for metric '%s' because the value is not numeric: %r",
        metric_name,
        value,
    )
    return

Impact

aws.dd_forwarder.logs_failed and aws.dd_forwarder.metrics_failed are never emitted, even when failures are actively occurring. Any monitors or dashboards based on these metrics will never fire.

Fix

send_event_metric("logs_failed", len(failed_logs))
send_event_metric("metrics_failed", len(failed_metrics))

Version

Reproduced on v5.4.3 (and present in prior versions based on git history).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions