I encountered a problem when I wanted to filter out metrics about certain labels when applying spanmetrics.

[Cairry]

Component(s)

connector/spanmetrics, processor/filter

Describe the issue you're reporting

background:
PromQL: sum(duration_count{span_name="GET"}) by (span_name)
Sample: {span_name="GET"} 27972

Need:
I want to filter out the metric whose span_name is GET through filter, but the following configuration does not work.

    processors:
      batch:
      filter/ottl:
        error_mode: ignore
        metrics:
          metric:
            - 'resource.attributes["span_name"] == "GET"'

The above is to simply test the filtering ability. In fact, I want to filter out span_name that does not start with GET or POST. Please help me complete it. Thank you

[github-actions]

Pinging code owners:

• connector/spanmetrics: @portertech @Frapschen
• processor/filter: @TylerHelmuth @boostchicken

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[iblancasa]

Is the filtering not working just with those metrics that are created with the spanmetrics connector or are you able to reproduce a similar behavior with other metrics?

Can you try this expression?

            - IsMatch(resource.attributes["span_name"], "GET")

[TylerHelmuth]

@Cairry most of the time when a filter isn't working it is because the attribute name or location is wrong. Please enable service.telemetry.logs.level=debug or add a debugexporter with verbosity: detailed to view the data as the collector sees it.

[Cairry]

过滤是否不仅适用于使用这些创建的指标，spanmetrics connector或者您是否能够使用其他指标重现类似的行为？

你能尝试一下这个表达吗？

            - IsMatch(resource.attributes["span_name"], "GET")

I got the following error after using the method you provided, it seems that the IsMatch method is not allowed

    processors:
      batch:
      filter/duration_count_get:
        error_mode: ignore
        metrics:
          metric:
            - IsMatch(resource.attributes["span_name"], "GET")
 ···
    service:
      pipelines:
        traces:
          receivers: [otlp, opencensus]
          exporters: [jaeger]
        traces/spanmetrics:
          receivers: [otlp, opencensus]
          exporters: [spanmetrics]
        metrics:
          receivers: [otlp, opencensus]
          processors: [filter/metrics]
          exporters: [prometheus]
        metrics/spanmetrics:
          receivers: [spanmetrics]
          processors: [filter/metrics]
          exporters: [prometheusremotewrite]

error

Error: invalid configuration: processors::filter/duration_count_get: unable to parse OTTL statement: 1:62: unexpected token "<EOF>" (expected <opcomparison> Value)
2024/08/09 01:27:18 collector server run finished with error: invalid configuration: processors::filter/duration_count_get: unable to parse OTTL statement: 1:62: unexpected token "<EOF>" (expected <opcomparison> Value)

All methods in processors do not meet the requirement of filtering out span_name=GET, which is a headache.

    processors:
      batch:
      filter/exclude_duration_count:
        metrics:
          exclude:
            match_type: strict
            metric_names:
              - duration_count

      filter/duration_count_get:
        error_mode: ignore
        metrics:
          metric:
            - IsMatch(resource.attributes["span_name"], "GET")
      filter/metrics:
        metrics:
          exclude:
            match_type: strict
            metric_names:
              - duration_count
            resource_attributes:
              - key: span_name
                value: GET
      filter/ottl:
        error_mode: ignore
        traces:
          span:
            - 'attributes["http.url"] == "http://opentelemetry-collector.istio-system:4318/v1/traces"'
            - 'attributes["http.url"] == "http://opentelemetry-collector.istio-system:4318/v1/metrics"'
            - 'attributes["grpc.authority"] == "opentelemetry-collector.istio-system:4317"'
        metrics:
          metric:
            - 'name == "duration_count"'
          datapoint:
            - metric.name == "duration_count"

    service:
      pipelines:
        traces:
          receivers: [otlp, opencensus]
          exporters: [jaeger]
        traces/spanmetrics:
          receivers: [otlp, opencensus]
          exporters: [spanmetrics]
        metrics:
          receivers: [otlp, opencensus]
          processors: [filter/metrics]
          exporters: [prometheus]
        metrics/spanmetrics:
          receivers: [spanmetrics]
          processors: [filter/metrics]
          exporters: [prometheusremotewrite]

[Frapschen]

@Cairry please try

      filter/duration_count_get:
        error_mode: ignore
        metrics:
          datapoint:
            - 'attributes["span.name"] == "GET"'

[Cairry]

@Cairry请尝试

      filter/duration_count_get:
        error_mode: ignore
        metrics:
          datapoint:
            - 'attributes["span.name"] == "GET"'

wow Thank you very much, your configuration is useful. Does it support regular expression? I hope to filter out span_name that is not GET and POST

[Frapschen]

@Cairry please refer this docs https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/pkg/ottl/ottlfuncs#ismatch

[iblancasa]

过滤是否不仅适用于使用这些创建的指标，spanmetrics connector或者您是否能够使用其他指标重现类似的行为？
你能尝试一下这个表达吗？

            - IsMatch(resource.attributes["span_name"], "GET")

I got the following error after using the method you provided, it seems that the IsMatch method is not allowed

Maybe the syntax was not correct... https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/filterprocessor#examples

[TylerHelmuth]

When using the datapoint context, assuming the value is a datapoint attribute, you'd do

      filter/duration_count_get:
        error_mode: ignore
        metrics:
          datapoint:
            - not IsMatch(attributes["span_name"], "GET|POST")

or

      filter/duration_count_get:
        error_mode: ignore
        metrics:
          datapoint:
            - attributes["span.name"] != "GET" and attributes["span.name"] != "POST"

[Cairry]

Sorry experts, I have a new problem, which is that I can't filter out the two indicators of duration_count and duration_bucket. Below are my various attempts to filter. Where do I need to correct it?

    processors:
      batch:
      filter/duration_span_kind:
        error_mode: ignore
        metrics:
          metric:
            - 'name == "duration.count"'
            - 'name == "duration.bucket"'
          datapoint:
            - 'attributes["span.kind"] != "SPAN_KIND_SERVER"'
            - metric.name == "duration.count"
            - metric.name == "duration.bucket"

      filter/duration_bucket_count:
        error_mode: ignore
        metrics:
          exclude:
            match_type: strict
            metric_names:
              - duration.count
              - duration.bucket
...
    connectors:
      spanmetrics:
        dimensions:
          - name: http.method
          - name: http.status_code 
          - name: http.route

    service:
      pipelines:
        traces:
          receivers: [otlp, opencensus]
          exporters: [jaeger]
        traces/spanmetrics:
          receivers: [otlp, opencensus]
          processors: [filter/duration_span_kind, filter/duration_bucket_count]
          exporters: [spanmetrics]
        metrics:
          receivers: [otlp, opencensus]
          exporters: [prometheus]
        metrics/spanmetrics:
          receivers: [spanmetrics]
          processors: [filter/duration_span_kind, filter/duration_bucket_count]
          exporters: [prometheusremotewrite]

[Cairry]

@Frapschen Hello, do you have time to help me look at this problem? It cannot filter out the two indicators of duration_bucket and duration_count. I have tried many ways but failed. I don't know where the problem is. It's so strange. Please help. Thank you~

[Frapschen]

@Cairry The original latency metrics name of spanmetrics is duration. You only need to:

    processors:
      filter/duration_span_kind:
        error_mode: ignore
        metrics:
          metric:
            - 'name == "duration"'

[Cairry]

@Cairry The original latency metrics name of spanmetrics is duration. You only need to:

    processors:
      filter/duration_span_kind:
        error_mode: ignore
        metrics:
          metric:
            - 'name == "duration"'

Ok thanks, will it filter out the duration_sum? Because I only need the duration_sum.

[Frapschen]

In my experience, I will add a debug exporter:

exporters:
  debug:
    verbosity: detailed

it will log any detailed data in the console.

[Frapschen]

duration_sum

I think the filter processor can't do that.

[Cairry]

duration_sum

I think the filter processor can't do that.

I understand that these are the basic key indicators that cannot be filtered? Is there any other way? I thought of, for example, adding different labels to different indicators, and then filtering out the metric by label? Because I really don't want duration_bucket and duration_count to exist, because it consumes a lot of resources.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• processor/filter: @TylerHelmuth @boostchicken
• connector/spanmetrics: @portertech @Frapschen

See Adding Labels via Comments if you do not have permissions to add labels yourself.