spanmetricsconnector metric has been increasing continuously

[shicli]

Component(s)

connector/spanmetrics

What happened?

Description

I am using spanmetrics connector to generate metrics from span.There are some doubts about the duration indicator. The generated duration indicator will always exist, even if the value is 0, it will still be collected in Prometheus, so that the data sent to Prometheus will continue to increase each time. Even after a long time, the old duration indicator still exists. Is this a normal phenomenon ？
If this is normal, then after a long period of time, there will be a lot of RED indicators, which will exert great pressure on both otel and prometheus. So what should be done? Is the RED indicator meant to be permanently stored in memory?

Steps to Reproduce

Using spanmetrics connector config and passing it as exporter in trace pipeline and receiving it in metrics pipeline.

connectors:
spanmetrics:
histogram:
explicit:
buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]
dimensions:
- name: http.method
default: GET
- name: http.status_code
exemplars:
enabled: true
exclude_dimensions: ['status.code']
dimensions_cache_size: 1000
aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"
metrics_flush_interval: 15s

Expected Result

calls and duration_count value always increase

Collector version

0.81

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
  nop:

exporters:
  prometheusremotewrite:
    endpoint: http://localhost:9090/api/v1/write
    target_info:
      enabled: true

connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [100us, 1ms, 2ms, 6ms, 10ms, 100ms, 250ms]
    dimensions:
      - name: http.method
        default: GET
      - name: http.status_code
    exemplars:
      enabled: true
    exclude_dimensions: ['status.code']
    dimensions_cache_size: 1000
    aggregation_temporality: "AGGREGATION_TEMPORALITY_CUMULATIVE"    
    metrics_flush_interval: 60s 

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [spanmetrics]
    metrics:
      receivers: [spanmetrics]
      exporters: [prometheusremotewrite]

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• connector/spanmetrics: @albertteoh

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[sakulali]

Hello @shicli, it seems similar to connector/spanmetrics?

[crobert-1]

Thanks for the reference @sakulali, it looks like a duplicate of #27654.

Please let us know if you believe this is a different issue, @shicli!

[shicli]

@crobert-1 @matej-g This is indeed the problem I encountered. Thx