Filter all the data sent by an application

[MaGaudin]

Component(s)

No response

Is your feature request related to a problem? Please describe.

We are always in trouble when we need to filter or drop data coming from an application. We have implemented an OTEL Collector in gateway mode, specifically with 10 apps sending data. We want to completely exclude one application, but this only works well for traces and logs. Using the filter processor for metrics is not feasible as it also drops metrics that are common with other applications.

Describe the solution you'd like

A way to exclude all the data coming from a particular application. Maybe an option on the Otel Receiver could be useful.

Describe alternatives you've considered

Tell to the dev team to stop sending data to the collector. You can understand that it's not feasible if we need to drop data only for test purposes.

Additional context

No response

[crobert-1]

We want to completely exclude one application, but this only works well for traces and logs. Using the filter processor for metrics is not feasible as it also drops metrics that are common with other applications.

Can you share what you want to filter by, in terms of resource attribute or attribute, or some other part of the metric? This would be helpful in better understanding potential solutions here.

[MaGaudin]

Hi @crobert-1,

Thanks for your reply. I just wanna filter everything coming from an application.

Let's say we have App A and App B sending data to the Otel Collector.

If, for testing purposes, I do not want to ingest data from App B, just to understand what's going on with App A, I need to ask developers to stop sending data to the collector.

So, here is my question. Is there a way to filter (or refuse) everything coming from an application?

Maybe, there is a way to do that but we were not able to find and use it.

[crobert-1]

Can you share the configuration you're using that isn't working as expected? It would also be helpful to share some example metric data that isn't behaving in the way you'd expect.

[MaGaudin]

Hey @crobert-1,
We tried something like this:

filter/exclude:
  metrics:
    exclude:
      match_type: expr/strict
      expressions:
        - Label("job") == "AAAA"

I think that, for the way the filter processor was designed, this worked well. I mean... this config dropped all the metrics with that label. But, since we are filtering metrics and not active series, the config dropped the metrics also for other applications, being some of them in common.

From here, we would like to have something more general as explained above.

[crobert-1]

But, since we are filtering metrics and not active series, the config dropped the metrics also for other applications, being some of them in common.

What you want to is to add a resource attribute check to your existing filter to only filter out by a specific value for a given resource attribute. When metrics are being received from different resources, they should have different resource attribute values.

[MaGaudin]

But filtering on that resource attribute will drop the metric also for the others. It seems we are reproducing what we already tried but filtering on a resource attribute. I mean ... the metric is only one but with different attributes. So, filtering it for one of them will drop the metric also for the other resource attributes. did I get it right?

[crobert-1]

If you do a filter statement that includes both label and resource attribute, only metrics/datapoints matching both will be dropped.

Something like:

label == "value" and resource.attributes["name"] == "foo"

If this still isn't clear, can you share examples of the metrics that you're looking at? If you copy and paste output from the debug exporter of one metric that you want to drop, and one that you want to keep, we can have a specific example of how to use the filter processor here.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.