Accessing the root span to mark it as "error"

[timmhirsens]

In our Java Applications we do trace caught exceptions the following way:

    public static void trace(Throwable throwable) {
        try {
            final Span span = GlobalTracer.get().activeSpan();
            if (span instanceof MutableSpan) {
                MutableSpan localRootSpan = ((MutableSpan) span).getLocalRootSpan();
                localRootSpan.setError(true);
            }
            if (span != null) {
                span.setTag(Tags.ERROR, true);
                span.log(Collections.singletonMap(Fields.ERROR_OBJECT, throwable));
            }
        } catch (Exception e) {
            LOG.error("failed to trace exception using datadog", e);
        }
    }

This leads to both the erroneous span and the root span in trace to show up as having an error. This leads to the caught exceptions showing up in the "Error Tracking" view in the Datadog UI.

What would be the equivalent in the ruby trace? Currently we only do

        Datadog::Tracing.active_span&.set_error(exception)

Which marks the erroneous span as error but not the root span, which means the errors dont show up in the error tracking view and, probably (?), won't trigger any monitors.

What is the correct way to trace caught errors in ruby using datadog?

[delner]

Hi @timmhirsens, thanks for bringing this up.

Ruby does not expose the root span through the public API, by design (mostly to prevent spaghetti code and other non-deterministic behavior.) To be clear, we consider "root spans" to be an internal detail, and we do not want users to access or modify the root span directly (by some means other than active_span); such behavior may be subject to unexpected breaking changes.

Instead, we have a TraceOperation which can be accessed through Datadog::Tracing.active_trace. It acts an abstraction for an "in flight trace", and can be tagged with metadata with TraceOperation#set_tag. Typically these tags are applied onto the root span upon trace completion. Do note, however, that this "applied to root span" behavior is also considered an internal detail, subject to change... it may not apply tags to a root span in the future.

Our intent is not to preserve "root span aware behavior" but to allow users to specify "trace-level behavior". We prefer users use this TraceOperation API instead because we believe its the appropriate level of abstraction, and will allow us to make changes to the trace data (root span behavior) without breaking user code in the future.

---

While today we have TraceOperation#set_tag, we do not have TraceOperation#set_error. I think it may be appropriate to add this to the TraceOperation API (e.g. Datadog::Tracing.active_trace&.set_error(exception)), to allow easy tagging of the trace with an error. However, it is worth highlighting that because errors are implemented as tags, and tags can only have one value, it will not be possible to tag a trace with multiple errors. The last error to be set via TraceOperation#set_error would take precedence, including overriding any error that may already be present on the root span itself.

If these limitations are reasonable and still satisfy the use case, then we could consider implementing this.

[delner]

@timmhirsens Any thoughts on the suggestion shared above?

[bquorning]

I was stumbling on this issue today. After upgrading to ddtrace v1 we start seeing exceptions where our old code is trying to call TraceOperation#set_error. So yes, I think it would be nice to have that method implemented.

[delner]

@bquorning That shouldn't have been an issue, as TraceOperation#set_error never existed prior to 1.0 and the trace method signature did not undergo any breaking changes... the signature merely appended trace to available args. E.g.

# Old
Datadog.tracer.trace('my.op') { |span| }

# New
Datadog::Tracing.trace('my.op') { |span, trace| }

[bquorning]

Yeah sorry, I left out the important details 😄

Based on https://github.com/DataDog/dd-trace-rb/blob/master/docs/UpgradeGuide.md#trace-api we switched from using Datadog.tracer.active_root_span (v0.x) to using Datadog::Tracing.active_trace (v1.x). The former returns a Datadog::Span instance, on which we can call #set_error, while the latter returns a Datadog::Tracing::TraceOperation instance.

[timmhirsens]

@delner sorry for the late reply, i was not working on the related ruby project for a while.

The solution you proposed sounds reasonable. What APIs would Datadog::Tracing.active_trace&.set_error(exception) end up calling? Which tags is it going to set exactly? I could implement this manually for now to check if the change would yield the desired results for us.

[bquorning]

The #set_error method is defined in this module and sets the tags 'error.message', 'error.stack', and 'error.type'.

I also implemented it manually for now.

[aenand]

Hi @timmhirsens @bquorning I'm new to Datadog and seeing a similar issue where I'm tagging my exceptions but they're not making their way into Error Tracking. I'm only calling Datadog::Tracing.active_span&.set_error(exception), how did you get errors to surface into Error Tracking?

[dostaglou]

I am currently working on the same kind of problem @aenand. I can set_error on the active_span and this error is indeed set and viewable in the logs. However in the Error Tracking section, it is not highlighted. So without digging it is unknown.

I suspected it was because the overarching root_span is not marked with the error. It is unclear how one can get around this unless I am missing something.

[ivoanjo]

Hey @dostaglou I spoke with a few colleagues and they suggested trying to set the track_error tag to true.

Let us know if that works for you :)

[aenand]

Hey @ivoanjo sorry if I missed this but where would I set the track_error tag?

[ivoanjo]

It should go on the same span as you're calling set_error.

[dgmora]

Answering the original question, I have been able to mark the trace with an error by doing:

root_span = Datadog::Tracing.active_trace&.send(:root_span)
root_span&.set_error(exception)
root_span&.set_tag('track_error', true) # Shows the error also in the errors section

Doing that the whole trace appears as errored. It's not great to mess with the internal API, but using the current_span does not make the error visible in most service dashboards (requests, resources, transactions)