Distributed traces across AWS services are not connected

[gkaskonas]

Describe the bug
We are using SQS and EventBridge to call services on AWS. However, when a service on lambda sends a message to another service on Lambda using SQS the spans are not connected. Trace on lambda becomes the parent.

Steps to reproduce
Run service 1 on lambda and send SQS message that is consumed by service 2.

What did you expect to see?
Service 1 should show up in the trace.

What did you see instead?
Only service 2 shows up

What version of collector/language SDK version did you use?
Version: latest collector and otel layers on lambda.

What language layer did you use?
Nodejs

Additional context
OTEL_PROPAGATORS are set to tracecontext,baggage,xray. Backend is Tempo.
Is there something I am missing?
I can see traceheader in the message attributes

"attributes": {
                    "ApproximateReceiveCount": "1",
                    "AWSTraceHeader": "Root=1-6808be41-c7d04735710d987017e5ec88;Parent=371ec507158784ee;Sampled=0;Lineage=240:e7ff9e16:238",
                    "SentTimestamp": "1745403457578",
                    "SenderId": "id",
                    "ApproximateFirstReceiveTimestamp": "1745403457579"
                },

Is it because I need sqsExtractContextPropagationFromPayload enabled? How could I do that without creating my own custom instrumentation layer?

[wpessers]

Hi @gkaskonas that does indeed not sound right. I will try to reproduce your issue this weekend when I have some more time. Do you receive any other spans for service 1 and only the sqs ones are missing?

[nic-littlepay]

+1
Fairly certain we're seeing the same issue in a similar setup we have, trying to auto-instrument Java Lambdas using otel java agent layer. I'd reached out to the community in this discussion post - #1792

[wpessers]

I've started looking into this. One thing I can say immediately is that sqsExtractContextPropagationFromPayload should not make a difference. This just influences how context propagation is done, and using sqs message attributes should be just fine.

[wpessers]

I have been able to reproduce this I think. I have this small example with 2 lambda functions. 1 is invoked through an http request (through apigateway), then sends an sqs message. Then using event source mapping the other lambda is invoked whenever a message becomes available on the sqs queue (this one also make a dynamodb call but that is not relevant).

Does this match your problem?
Here's an example from tempo where I have the 2 separate traces:
1st lambda:

2nd lambda after sqs message received:

I haven't looked deeply into the instrumentation code, but am currently thinking that maybe the propagation of the context doesn't work because we are using event source mapping. Meaning in the "receiving" lambda we don't explicitly use the sqs client to consume messages, which would otherwise be instrumented and allow propagating the context... Will try to investigate this further.

[gkaskonas]

Yes thats what I see

[wpessers]

So reading this in the aws-lambda instrumentation readme: https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/plugins/node/opentelemetry-instrumentation-aws-lambda#context-propagation

It seems to me that it is actually impossible to link the spans together if we're not using X-Ray 🤔

[gkaskonas]

Kind of defeats the purpose of distributed tracing in my opinion. Even with X-Ray set to PASSTHROUGH?

[wpessers]

Agreed, going to do some more testing on this as the aws sdk instrumentation does seem to state the opposite: https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/plugins/node/opentelemetry-instrumentation-aws-sdk/doc/sqs.md#sendmessage--sendmessagebatch

OpenTelemetry trace context is injected as SQS MessageAttributes, so the service receiving the message can link cascading spans to the trace which created the message.

[nic-littlepay]

Thanks for your work on this @wpessers

OpenTelemetry trace context is injected as SQS MessageAttributes, so the service receiving the message can link cascading spans to the trace which created the message.

A couple of things on this part of the doc:

1. to me, this could be interpreted as, "the service receiving the message can add their own code to manually extract the trace/span values and update the context in that service". If this was the meaning, then I think we had been expecting the otel auto-instrumentation to do this trace passing automatically.

2. In the Java agent lambda layer at least, trace values are not transported across SQS MessageAttributes unless the experimental-use-propagator-for-messaging option is set (see this doc on that). When we tested setting this option we did see trace id passed in MessageAttributes but not in the way we expected and it still was not automatically propagated to the context in the receiving service (more details on this in the 'What We've Tried' section of my discussion post on this repo)

[nic-littlepay]

I haven't looked deeply into the instrumentation code, but am currently thinking that maybe the propagation of the context doesn't work because we are using event source mapping. Meaning in the "receiving" lambda we don't explicitly use the sqs client to consume messages, which would otherwise be instrumented and allow propagating the context... Will try to investigate this further.

this crossed our mind too, perhaps the auto-instrumentation isn't covering the event driven SQS lambda handler

[gkaskonas]

Any progress?
We are about to roll out otel across multiple teams which is tied together using Eventbridge and SQS mostly so it would be great to ge to the bottom of this

[wpessers]

Hi @gkaskonas I'm sorry for the late reply, have been quite busy but finally getting some more free time on my hands to look into this. Thanks @nic-littlepay for your findings as well.

I have already discussed this with @serkan-ozal (who is a maintainer on the project) as well and he proposed trying to link the spans together.

The big issue here is that with how sqs batch processing works, we can never really be sure to which "parent" a lambda invocation span belongs. We can probably try to do something with span-linking, so that your receive span is not actually added to that same trace. But it will be linked, so depending on the capabilities of your observability backend you will be able to still correlate the spans.

[serkan-ozal]

Hi @gkaskonas @nic-littlepay,

As @wpessers mentioned that the real challenge here is more than propagating trace context over SQS messages. Let me share the example I have used while discussing with Warre:

Assume that we have the following flow: "Lambda-A => SQS queue => Lambda-B"

And we have two different invocations and traces from "Lambda-A" :

- trace-1:  Lambda-A-invocation-span-1 => SQS-client-span-1 (sent message-1 to the SQS queue)
- trace-2:  Lambda-A-invocation-span-2 => SQS-client-span-2 (sent message-2 to the SQS queue)

Then, "Lambda-B" is invoked/triggered with the batch of SQS messages 
including both "message-1" and "message-2" in the same invocation. 
And assume that we are able to extract trace context from both 
"message-1" and "message-2" as "trace-1" and "trace-2" accordingly. 
We also have an "Lambda-B" invocation span as root span here.

Then the question is that which one of the "trace-1" and "trace-2" will be used 
as the parent trace content of the Lambda invocation as there can only be one parent trace context.

If the Lambda could be triggered with just one SQS message, 
the problem will be easier to solve as we can just extract the trace context 
from the SQS message attributes and use it as parent trace context of the Lambda invocation span. 
But, in a single invocation, Lambda function might be triggered by multiple SQS messages from different traces.

So, I think, for the cases where there are multiple upstream traces, it makes sense to

• extract all the parent trace contexts (e.g. from all SQS messages in batch)
• start a new trace with the Lambda invocation span
• add all the extracted parent trace contexts to the current trace as span link

When there are multiple traces connected to each other over span links, showing all span in a single trace map will depend on the capabilities of the tracing backend you use.

In addition to this behavior, as an optimization (not sure though), we might also have the following logic to cover some cases:
If all the the parent trace contexts belong to the same trace (e.g. Lambda is triggered with a single SQS message or all the SQS messages belong to the same trace), the new Lambda invocation span will inherit the trace id.

In summary, we have to items here:

• How to propagate (inject and extract) trace context over AWS SQS?
• What to do with batch of SQS messages coming from the same or different traces?

[gkaskonas]

I understand that batch can be difficult but I am not using batch. It would be nice to support non batch at least
Also what about EventBridge? It seems that traces get lost after that call