Tracer: "failed to send data to X-Ray daemon"

[heitorlessa]

What were you trying to accomplish? Traces not appearing when processing multiple records

This is documented here: aws/aws-xray-sdk-python#201

Expected Behavior

Use sqs batch utility with 10 records and Tracer at the same time without losing traces.

Current Behavior

@pcolazurdo discovered an issue when using Batch processing utility when you have multiple exceptions or multiple records, where X-Ray fails silently and doesn't send the trace data via UDP as expected.

Possible Solution

Send segments as they're complete instead of batching all of them: https://github.com/srprash/aws-xray-sdk-python/blob/9b2704dd534419b3dc2ae6e9b1793e1fc8a405c9/tests/test_recorder.py#L65

Need to experiment whether that has any negative perf impact when enabled

Steps to Reproduce (for bugs)

1. Initialize a new project sam init --location https://github.com/aws-samples/cookiecutter-aws-sam-python
2. Convert to SQS event source
3. Bring SQS Batch processing utility
4. Either add long responses (100 dict keys) or raise an exception for all 10 records

Environment

• Powertools version used: 1.10.1
• Packaging format (Layers, PyPi): PyPi
• AWS Lambda function runtime: Python 3.8
• Debugging logs

The error only appears when debugging is enabled.

[ERROR] 2021-02-03T10:39:10.193Z 2851698b-0b0e-5ccd-9cc7-311f747567f2 failed to send data to X-Ray daemon.
Traceback (most recent call last):
File "/var/task/aws_xray_sdk/core/emitters/udp_emitter.py", line 62, in _send_data
self._socket.sendto(data.encode('utf-8'), (self._ip,
OSError: [Errno 90] Message too long

# paste logs here

[nmoutschen]

Could confirm the issue with a minimal reproduction where the record handler would send back ~100K of data. Changing the streaming_threshold to 0 for the xray_recorder solved the issue.

There is a small performance penalty (96ms vs 93ms @ p90, 62ms vs 58ms @ p50 in my tests) to implement this change. In itself, this is not a big impact, but we need to balance this out with the probability that someone will be impacted by this issue.

[heitorlessa]

This is now available in 1.10.3 - Big thanks for reporting it @pcolazurdo and @nmoutschen for fixing it ;)