New component: Span Event Filter Processor

[JamieDanielson]

Is your feature request related to a problem? Please describe.

While span events can be useful in some circumstances, other times they can be noisy or undesired. Some things like service meshes (Envoy, Istio) and auto-instrumentation (gRPC) attach a lot of span events, and there is currently no way to filter them out of traces.

Describe the solution you'd like

A Span Event Filter Processor would provide the ability to filter out span events based on attributes and names.

Describe alternatives you've considered

Attempting to drop span events from all possible instrumentations is not feasible. Since it's also not possible (yet?) to drop individual spans, dropping the trace entirely or sampling more heavily are other options... but that results in losing more valuable data to filter out unwanted data.

Additional context
​
There is already a (now closed) PR that I would recommend reviving for this purpose.

[jpkrohling]

Is this related to #6341 as well?

[jpkrohling]

In time: we had some discussions about this in the PR I linked earlier but it wouldn't solve the service mesh or auto-instrumentation scenarios you linked. The problem is that we'd need to remap the parent span ID when removing a span, so that the child spans won't hold references to non-existing spans, which could result in broken traces.

[JamieDanielson]

Is this related to #6341 as well?

That PR does have behavior desired by many (ability to drop an individual span regardless of potential "missing" spans within traces); I did allude to it here as a potential alternative if it existed... though it does not fulfill the true use case, which is to drop span events and keep the span.

[jpkrohling]

Thanks, I did miss the part that this is about events in the span, not the whole span. @anuraaga, @bogdandrutu, is this something the transform processor should support?

[anuraaga]

@jpkrohling Thanks for pointing out, indeed we would probably add a function for this to the transform processor once the initial framework gets settled in

[gregoryfranklin]

I was taking a bit of a look at what would been needed to add this to the transform processor (since I also need the ability to drop span events from spans).

It looks like a new function would be required, which could potentially look something like (similar to the delete_matching_keys function)

delete_span_events(target, pattern)  // delete span events when target matches pattern

eg

delete_span_events(name, ".*")
delete_span_events(attributes["exception.message"], ".*")

The problem I'm running into is that functions cannot take a path as an argument. I can either use a Getter or a string, so using "name" is easy but parsing "attributes["exception.message"] is more difficult.

The other option I could think of was creating a SpanEventTransformContext. This makes is possible to use the Getter.
Using this approach, the function looks like

delete_span_events(events.name, ".*")

Something like this:
gregoryfranklin@cc4adfb

PS. if this is going to be addressed using the transform processor then maybe change the title of the issue?

[Blackbaud-LeoWanderer]

I just looked at the linked PR and got the same issue with Istio/Envoy span events. I found out other span events were created by developers and I'd like to keep those, so having the ability to filter span events by name would be great. The ones that are a problem for me at the moment are cr and ss since they account for a big chunk of events and I'm getting charged for those 😢

Has there been any progress on a new processor or a change to the transform processor out of this thread? Have people sorted out the issue in another way? (maybe disabling those logs from the Istio/Envoy side).

[TylerHelmuth]

I need to add a SpanEvents context (#14578) so that we'll have the proper access to the event names and attributes, but once that is done we should be able to add a drop function to the transform processor that can drop span events based on the ottl condition.