How to remove spans from the UI without confusing users

[jpkrohling]

We often get requests for features involving removing spans from the view. Collapsing spans are typically recommended, but there are some very good use cases for removing spans.

The problem with just removing spans is that it might confuse or mislead users, as the absence of a span is an information in itself. This ticket is to track ideas for solutions to this problem. Possible paths to research:

1. a warning at the top of the trace view, telling users that spans might be missing from the view, with an option to load them
2. a lightweight notation (a line?) where the missing span would be

[lookfwd]

A warning isn't that useful from my viewpoint, cause this is a "user profile". It's not something exceptional. What I see as viable is something along the lines of a drop-down:

You're now in "developer mode" or "infrastructure mode" or "mobile view" or "UI developer" or "Full view" with the ability to switch between the above.

The missing spans are not a "bug". They are a feature. When you have 10k Spans view out of a 800k-Span trace, you want only the relevant Spans to appear in your view. As such, only the back-end can trim the tree appropriately. The front-end will likely not even have all the Spans available.

Now in terms of what the back-end does, there are options:

• It might eliminate Spans (if they're leaf Spans)
• It might turn non-leaf to leaf Spans and prompt (somehow) to expand

it might prioritize based on arbitrary rules. But that's getting too involved of a discussion. If we focus on what the front-end would do, I feel that the answer would be, very little. At a high level it takes the profiles from the back-end and shows them in a drop down, tracks the currently selected profile on a cookie and sends the value to the back-end when it does a request.

As a minimum viable implementation, the back-end could exclude Spans with predefined tags for each profile.

[jpkrohling]

When you have 10k Spans view out of a 800k-Span trace, you want only the relevant Spans to appear in your view.

I'm impressed by this case. Do you actually have a trace with 800k spans? Does it really represent a single business transaction? Are you able to find your way even with a trimmed trace, containing "only" 10k spans?

As such, only the back-end can trim the tree appropriately. The front-end will likely not even have all the Spans available.

If we determine that we need a notation for the missing spans, the backend can add this information where the spans were removed, can't it? Like: "removed 2000 spans", and the UI will display it accordingly.

At a high level it takes the profiles from the back-end and shows them in a drop down

I like this idea, and although it's related, I think it's part of a separate discussion (#648).

[lookfwd]

Do you actually have a trace with 800k spans

:) Yes... I don't remember if it's multi-parent one. It's 680 Mb (!!) We also have a "full-trace" with 20k distinct traces (distinct traceIds connected with follows-from).

Does it really represent a single business transaction?

Funny enough, it does. Let me say at a very high level, that if you ask me for the price of a complex financial index, I might have to do thousands of queries (with massive concurrency) to calculate that. Then of course I will cache things, but at least one time it won't be in the cache.

Are you able to find your way even with a trimmed trace, containing "only" 10k spans?

Those are extreme traces. But even with "normal" traces of let's say a couple of thousand Spans, as an engineer, it's very annoying when you see 3 infrastructure Spans for 1 span you care about. And nobody is wrong there - everyone has right to their views (pun intended) :D

"removed 2000 spans", and the UI will display it accordingly.

Sure. If we get to that point, it will be trivial to add a flag to not show them at all. Yes, the back-end can say "removed 2000 spans", but in terms of infra... Ok, let me put it this way. Imagine if suddenly on your AWS X-Ray you start seeing route 53-internal Spans, Application router Spans, DynamoDB internals. You would get mad, right? :) So, for the engineer, it's not like we're removing Spans when we hide infra. We hide things that are supposed to be transparent.

I think it's part of a separate discussion

ok!

[jpkrohling]

Let me say at a very high level, that if you ask me for the price of a complex financial index, I might have to do thousands of queries (with massive concurrency) to calculate that.

Would you be able to list, possibly with numbers and to whom it is relevant, how a trace looks like in this case? Like:

• x spans that are recording HTTP server/client communication, relevant to both devs and ops
• x spans that are created with manual instrumentation in the applications, relevant to devs only
• x spans that are related to database connections, relevant to devs only
• ...

Those are extreme traces.

It's good to keep this in mind.

But even with "normal" traces of let's say a couple of thousand Spans, as an engineer, it's very annoying when you see 3 infrastructure Spans for 1 span you care about.

Agree.

for the engineer, it's not like we're removing Spans when we hide infra. We hide things that are supposed to be transparent.

That's a great way to put it.

[lookfwd]

Would you be able to list, possibly with numbers and to whom it is relevant, how a trace looks like in this case?

Some ballpark numbers out of an hour of data. I've excluded traces with less than 10 Spans from the calculations because they are less interesting.

Size

>5k spans: 0.02%
>10k spans: 0.01%
>50k spans: 0.002%
>200k spans: 0.0003%

Structure

12%: Databases (5% interesting to developers)
8%: Gateways (for ops)
0.5%: Other Infrastructure (for ops)
?% Feature toggles (so large we don't do it)
85%: Applications Spans of interest to developers

Beyond the story those numbers say, I feel that the traces we typically look at are not average traces.

[jpkrohling]

Thanks, this helps!

I feel that the traces we typically look at are not average traces.

That's a great observation. In other words, you are typically looking at outliers.

[lookfwd]

That's a great observation. In other words, you are typically looking at outliers.

Yes. I can think of 4 different components of that observation:

1. Outliers are interesting so I look for them
2. I might see many non-outliers/small traces but I don't realize/pay attention
3. Large traces have higher probability to be found. If a Span is a lottery ticket, betting on a service name to be found, then larger Traces have more tickets
4. Finally, because I use advanced aggregation tools, as a "Trace developer" maybe I (and only I) see way more of the super-extreme traces