Bug
In aws/logs_monitoring/forwarder.py, send_event_metric is called with the failed items list as the metric value instead of its length:
# Line 141 - passes list, should be len()
send_event_metric("logs_failed", failed_logs)
# Line 172 - passes list, should be len()
send_event_metric("metrics_failed", failed_metrics)
Compare with the correct pattern used on the adjacent lines:
send_event_metric("logs_forwarded", len(logs_to_forward) - len(failed_logs))
send_event_metric("metrics_forwarded", len(metrics) - len(failed_metrics))
Root cause
send_event_metric calls lambda_metric, which validates the value with float(value). Passing a list raises TypeError, which is caught and logged as a warning, then the metric is silently dropped:
[WARNING] Ignoring metric submission for metric 'aws.dd_forwarder.logs_failed'
because the value is not numeric: [...]
See datadog_lambda/metric.py:
try:
float(value)
except (ValueError, TypeError):
logger.warning(
"Ignoring metric submission for metric '%s' because the value is not numeric: %r",
metric_name,
value,
)
return
Impact
aws.dd_forwarder.logs_failed and aws.dd_forwarder.metrics_failed are never emitted, even when failures are actively occurring. Any monitors or dashboards based on these metrics will never fire.
Fix
send_event_metric("logs_failed", len(failed_logs))
send_event_metric("metrics_failed", len(failed_metrics))
Version
Reproduced on v5.4.3 (and present in prior versions based on git history).
Bug
In
aws/logs_monitoring/forwarder.py,send_event_metricis called with the failed items list as the metric value instead of its length:Compare with the correct pattern used on the adjacent lines:
Root cause
send_event_metriccallslambda_metric, which validates the value withfloat(value). Passing a list raisesTypeError, which is caught and logged as a warning, then the metric is silently dropped:See
datadog_lambda/metric.py:Impact
aws.dd_forwarder.logs_failedandaws.dd_forwarder.metrics_failedare never emitted, even when failures are actively occurring. Any monitors or dashboards based on these metrics will never fire.Fix
Version
Reproduced on v5.4.3 (and present in prior versions based on git history).