How to group spans under specific traceId in openTelemetry node sdk?

[danielo515]

I have a distributed system where certain operations happen asynchronously over the course of several messages being processed by different actors. Each message has a correlation id of the message that originated the whole flow, so they can be easily traced toghether. Given the idea behind of open telemetry is to provide a standard way to trace computations across different systems, this should be simple to achieve, but the reality is that I'm really struggling with it.

The first surprising problem I faced was how hard is to set a specific traceId in a span.
There are no methods to set the traceId of a span, there are no options to create a span with specific traceId. You need to somehow set that in context, but it also require to provide a spanId, and new spans created within that context will inherit that traceId, but also that spanId.
That lead me to the creation of spans whose parent span does not exist (because, just like the traceId, I was generating the spanId myself).

Another annoying problem was that not any ID is a valid traceId or span Id, so, in order to generate the same traceId based on a specific correlation id I had to hash the string and adjust the length to 32 in the case of trace id and 16 in the case of spanId. Nothing warns you about this, if you provide either an invalid traceId or span Id they will be silently ignored and new ones will be created.

Below is the code that I came up with to try to group spans based on a specific correlation Id:

import { api } from "@opentelemetry/sdk-node";

function deriveHexString(input: string, desiredLength: number) {
  const hash = crypto.createHash("sha256").update(input).digest("hex");
  return hash.slice(0, desiredLength);
}

/**
* Creates a new context where the trace ID is set based on the provided
* correlationId.
* If there is an active span, the details of it will be used
**/
function createContext(correlationId: string) {
  const activeSpan = api.trace.getActiveSpan();
  const traceId = api.isValidTraceId(correlationId)
    ? correlationId
    : deriveHexString(correlationId, 32);
  const newSpan = activeSpan || api.trace.getTracer("doh").startSpan("newSpan");
  const context = api.trace.setSpanContext(
    api.trace.setSpan(api.context.active(), newSpan),
    {
      ...newSpan.spanContext(),
      traceId: traceId,
    },
  );
  newSpan.end();
  return context;
}

export function withTraceContext<A, U extends unknown[]>(
  correlationId: string,
  fn: (...args: U) => A,
) {
  return (...args: U) => {
    const context = createContext(correlationId);
    return api.context.with(context, () => fn(...args));
  };
}

Please note that I already tried several permutations of the createContext function. For example, not creating any new span for the context and just using the traceId for that:

function createContext(correlationId: string) {
  const activeSpan = api.trace.getActiveSpan();
  const traceId = api.isValidTraceId(correlationId)
    ? correlationId
    : deriveHexString(correlationId, 32);
  const spanId = activeSpan?.spanContext().spanId || traceId.slice(0, 16);
  const context = api.trace.setSpanContext(api.context.active(), {
    spanId,
    traceId: traceId,
    traceFlags: 1,
  });
  return context;
}

Both of this lead to several spans not being properly nested (for some reason I can not understand), but the worst, is that the spans that do belong to a trace where many of the spans reference a parent span that does not exist, which is a serious problem.
What is the correct way of achieve this? Maybe the problem is that the new span that I'm generating is not being sent?

[danielo515]

Are traces meant to group this kid of events? IMO they are, but if they are not the right thing, what else? How do I connect all my events under a single unit that I can visualize together? I'm quite dissapointed that this is not doable with OTEL, as the main motto of it was to "observe distributed systems" but it forgets to specify "if they fit withit our traditional vision of it"

[BenGale]

Did you ever work this out?

[danielo515]

No, and I tried all possible channels: stack overflow, discord and even the discord of other communities.
It seems this is not the intended usage of spans, but I was never able to understand why

[danielo515]

For your reference, here is the question on StackOverflow: https://stackoverflow.com/questions/78680955/how-to-group-spans-under-specific-traceid-in-opentelemetry-node-sdk/78752425?noredirect=1#comment138899183_78752425

[pcesara]

Have you considered propagating the Span Context through the message's metadata?

Then every service that consumes the message could have its own trace, while linking itself to the trace in the message's context (see Span Links).
If you would like to search your traces by the correlation ID, you could always add the correlation ID as a Span Attribute.

Please note that, if you have sampling at any point when processing your tracing data, you could still run into the situation where the original span that you are linking to is gone.

[Patafix]

I'm looking for the same system, we have a long processing accross service with a correlation id.