Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Metrics converted from span data using spanmetrics connector are inaccurate #20746

Closed
xwgao opened this issue Apr 7, 2023 · 12 comments
Closed

Metrics converted from span data using spanmetrics connector are inaccurate #20746

xwgao opened this issue Apr 7, 2023 · 12 comments
Labels
bug Something isn't working closed as inactive connector/spanmetrics processor/metricstransform Metrics Transform processor Stale

Comments

@xwgao
Copy link

xwgao commented Apr 7, 2023

Component(s)

connector/spanmetrics

What happened?

Description

I enabled auto-instrumentation for my microservices in Node.js. And I used spanmetrics connector in my otel collector to generate metrics (duration and error) for microservices in my application. I created a Grafana dashboard to show these metrics, sometimes when I send an HTTP request to the microservice, the metrics data shown on the Grafana dashboard are not updated with this new HTTP call.

Steps to Reproduce

My otel collector config is as below:

spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      otlp/spanmetrics:
        protocols:
          grpc:
            endpoint: 0.0.0.0:12346

    processors:
      batch:
        timeout: 10s
        send_batch_size: 10000
      spanmetrics:
        metrics_exporter: prometheus
        latency_histogram_buckets:
          [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 5s, 7s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.target
          - name: http.url
      metricstransform:
        transforms:
          - include: mas-assist.duration
            match_type: regexp
            action: update
            operations:
              - action: add_label
                new_label: masinstanceid
                new_value: assistdevtest
              - action: add_label
                new_label: masappid
                new_value: assist
              - action: add_label
                new_label: masworkspaceid
                new_value: masdev
              - action: update_label
                label: http.url
                new_label: url
              - action: update_label
                label: http.method
                new_label: method
              - action: update_label
                label: http.status_code
                new_label: code
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "service.name": "assistdevtest-ema-api",
              "http.target": "/api/v1/session-management/sessionsCount",
              "method": "GET"
            }
            new_name: mas-assist.collaborate_getsessioncount_duration_seconds
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "service.name": "assistdevtest-ema-api",
              "http.target": "/api/v2/document-query/query$",
              "method": "GET"
            }
            new_name: mas-assist.documentquery_searchdocuments_duration_seconds

    exporters:
      otlp:
        endpoint: "172.30.132.14:4317"
        tls:
          insecure: true
      logging:
        verbosity: detailed
      jaeger:
        endpoint: my-jaeger-collector-headless.jaeger-demo.svc:14250
        tls:
          ca_file: "/etc/pki/ca-trust/source/service-ca/service-ca.crt"
      prometheus:
        endpoint: "0.0.0.0:8889"
        send_timestamps: true
        metric_expiration: 1440m

    connectors:
      spanmetrics:
        namespace: mas-assist
        histogram:
          unit: s
          explicit:
            buckets: [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 5s, 7s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.target
          - name: http.url
          - name: http.route

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [spanmetrics, logging, jaeger, otlp]
        metrics:
          receivers: [otlp/spanmetrics, spanmetrics]
          processors: [batch, metricstransform]
          exporters: [prometheus, logging, otlp]
  mode: statefulset
  volumeMounts:
    - mountPath: /etc/pki/ca-trust/source/service-ca
      name: cabundle-volume
  volumes:
    - configMap:
        name: my-otelcol-cabundle
      name: cabundle-volume
  image: >-
    ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.75.0

Using this otel collector, I can get the metrics mas_assist_xxx and show them on the Grafana dashboard. But I find that the sometimes the metrics data is inaccurate. For example, I send an HTTP request to my backed service, but on the Grafana dashboard, the bucket heat map (the corresponding PromQL is sum(round(increase(mas_assist_documentquery_searchdocuments_duration_seconds_bucket{masinstanceid="assistdevtest",masworkspaceid="masdev"}[$__range]))) by (le)) are not updated with this new HTTP call.

Expected Result

The metrics data generated with spanmetrics connector are accurate.

Actual Result

The metrics data generated with spanmetrics connector are inaccurate.

Collector version

v0.75.0

Environment information

Environment

OS: OpenShift 4.10

OpenTelemetry Collector configuration

spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      otlp/spanmetrics:
        protocols:
          grpc:
            endpoint: 0.0.0.0:12346

    processors:
      batch:
        timeout: 10s
        send_batch_size: 10000
      spanmetrics:
        metrics_exporter: prometheus
        latency_histogram_buckets:
          [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 5s, 7s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.target
          - name: http.url
      metricstransform:
        transforms:
          - include: mas-assist.duration
            match_type: regexp
            action: update
            operations:
              - action: add_label
                new_label: masinstanceid
                new_value: assistdevtest
              - action: add_label
                new_label: masappid
                new_value: assist
              - action: add_label
                new_label: masworkspaceid
                new_value: masdev
              - action: update_label
                label: http.url
                new_label: url
              - action: update_label
                label: http.method
                new_label: method
              - action: update_label
                label: http.status_code
                new_label: code
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "service.name": "assistdevtest-ema-api",
              "http.target": "/api/v1/session-management/sessionsCount",
              "method": "GET"
            }
            new_name: mas-assist.collaborate_getsessioncount_duration_seconds
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "service.name": "assistdevtest-ema-api",
              "http.target": "/api/v2/document-query/query$",
              "method": "GET"
            }
            new_name: mas-assist.documentquery_searchdocuments_duration_seconds

    exporters:
      otlp:
        endpoint: "172.30.132.14:4317"
        tls:
          insecure: true
      logging:
        verbosity: detailed
      jaeger:
        endpoint: my-jaeger-collector-headless.jaeger-demo.svc:14250
        tls:
          ca_file: "/etc/pki/ca-trust/source/service-ca/service-ca.crt"
      prometheus:
        endpoint: "0.0.0.0:8889"
        send_timestamps: true
        metric_expiration: 1440m

    connectors:
      spanmetrics:
        namespace: mas-assist
        histogram:
          unit: s
          explicit:
            buckets: [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 5s, 7s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.target
          - name: http.url
          - name: http.route

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [spanmetrics, logging, jaeger, otlp]
        metrics:
          receivers: [otlp/spanmetrics, spanmetrics]
          processors: [batch, metricstransform]
          exporters: [prometheus, logging, otlp]
  mode: statefulset
  volumeMounts:
    - mountPath: /etc/pki/ca-trust/source/service-ca
      name: cabundle-volume
  volumes:
    - configMap:
        name: my-otelcol-cabundle
      name: cabundle-volume
  image: >-
    ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.75.0

Log output

No response

Additional context

No response
@xwgao xwgao added bug Something isn't working needs triage New item requiring triage labels Apr 7, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Apr 7, 2023

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@xwgao
Copy link
Author

xwgao commented Apr 11, 2023

/label processor/metricstransform

@github-actions github-actions bot added the processor/metricstransform Metrics Transform processor label Apr 11, 2023
@kovrus
Copy link
Member

kovrus commented Apr 11, 2023

Can you clarify how exactly it is inaccurate, for example, the recent metrics are not updated, etc. It would be great to have more information. Can you also simplify the collector config and verify whether the issue is still there after removing the metricstransform? Can you also rely on the logging exporter for now when reasoning about the data correctness, it would also narrow the debugging scope. Thanks.

@xwgao
Copy link
Author

xwgao commented Apr 12, 2023

@kovrus Yes, the recent metrics are not updated. For example, firstly I sent two HTTP requests to endpoint /api/v2/document-query/query of service assistdevtest-ema-api, the bucket heat map was updated on Grafana dashboard as below.
Screen Shot 2023-04-12 at 11 48 26

Then I sent 3 more HTTP requests to endpoint /api/v2/document-query/query of service assistdevtest-ema-api, but the bucket heat map was NOT updated on Grafana dashboard. The count in le=2 bucket is still 2.

I didn't verify after removing the metricstransform processor since we need it to generate new metrics naming with mas_assist_xxx_xxx_duration. I can remove it and retry if needed.

Yes, I've added logging exporter in my otel collector config as below. @kovrus Could you please let me know how to narrow the debugging scope according to the otel collector pod log? Thanks a lot.

exporters:
      ...
      logging:
        verbosity: detailed
...
service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [spanmetrics, logging, jaeger, otlp]
        metrics:
          receivers: [otlp/spanmetrics, spanmetrics]
          processors: [batch, metricstransform]
          exporters: [prometheus, logging, otlp]

@xwgao
Copy link
Author

xwgao commented Apr 14, 2023

@kovrus Is there any update for this issue? Thanks a lot.

@kovrus
Copy link
Member

kovrus commented Apr 14, 2023 •
edited
Loading

It is hard to say something having this information. Can you minimalize the setup to reproduce the same issue? you can drop the metric transform processor, and use logging exporter output to reproduce the issue? Have you taken a look at the collector logs? Also, please note that the spanmetrics connector flushes data periodically, so you won't see data immediately.

@xwgao
Copy link
Author

xwgao commented Apr 17, 2023 •
edited
Loading

@kovrus I've looked at the collector logs, there are lot of messages as below. Is there anything else I need to check from the otel collector log? What's the period time for the spanmetrics connector to flush data?

Metric #3
Descriptor:
     -> Name: mas-assist.query_querydocs_duration_seconds
     -> Description: 
     -> Unit: s
     -> DataType: Histogram
     -> AggregationTemporality: Cumulative
HistogramDataPoints #0
Data point attributes:
     -> service.name: Str(assistdevtest-ema-api)
     -> span.name: Str(HTTPS GET)
     -> span.kind: Str(SPAN_KIND_SERVER)
     -> status.code: Str(STATUS_CODE_UNSET)
     -> method: Str(GET)
     -> code: Int(200)
     -> http.target: Str(/api/v2/document-query/query)
     -> url: Str(https://ema-api-service.mas-assistdevtest-assist.svc/api/v2/document-query/query?projectId=9ba5332c-670b-4a21-a85e-c0af831d284a&collectionIds[]=527328d2-e9f7-eaa9-0000-0184f7461abf&naturalLanguageQuery=&passages=false&count=20&returnFields[]=metadata&returnFields[]=text&returnFields[]=result_metadata&returnFields[]=extracted_metadata&returnFields[]=highlight&returnFields[]=description&returnFields[]=description_longdescription&returnFields[]=wonum&returnFields[]=asset&returnFields[]=failure&returnFields[]=location&returnFields[]=assignment&returnFields[]=failurecode&returnFields[]=worklog&returnFields[]=woactivity&returnFields[]=href&offset=0&highlight=true)
     -> http.route: Str()
     -> masinstanceid: Str(assistdevtest)
     -> masappid: Str(assist)
     -> masworkspaceid: Str(masdev)
StartTimestamp: 2023-04-17 06:37:30.130142734 +0000 UTC
Timestamp: 2023-04-17 08:28:45.130818869 +0000 UTC
Count: 1
Sum: 0.182753
ExplicitBounds #0: 0.010000
ExplicitBounds #1: 0.100000
ExplicitBounds #2: 0.200000
ExplicitBounds #3: 0.400000
ExplicitBounds #4: 0.800000
ExplicitBounds #5: 1.000000
ExplicitBounds #6: 1.200000
ExplicitBounds #7: 1.400000
ExplicitBounds #8: 1.600000
ExplicitBounds #9: 1.800000
ExplicitBounds #10: 2.000000
ExplicitBounds #11: 4.000000
ExplicitBounds #12: 6.000000
ExplicitBounds #13: 8.000000
ExplicitBounds #14: 10.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 1
Buckets #3, Count: 0
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
HistogramDataPoints #1
Data point attributes:
     -> service.name: Str(assistdevtest-ema-api)
     -> span.name: Str(HTTPS GET)
     -> span.kind: Str(SPAN_KIND_SERVER)
     -> status.code: Str(STATUS_CODE_UNSET)
     -> method: Str(GET)
     -> code: Int(200)
     -> http.target: Str(/api/v2/document-query/query)
     -> url: Str(https://ema-api-service.mas-assistdevtest-assist.svc/api/v2/document-query/query?projectId=d6f31006-521d-4975-b969-95001d8befc6&collectionIds[]=0d4201fb-ac04-8046-0000-018417459188&naturalLanguageQuery=&passages=false&count=20&returnFields[]=metadata&returnFields[]=text&returnFields[]=result_metadata&returnFields[]=extracted_metadata&returnFields[]=highlight&returnFields[]=description&returnFields[]=description_longdescription&returnFields[]=wonum&returnFields[]=asset&returnFields[]=failure&returnFields[]=location&returnFields[]=assignment&returnFields[]=failurecode&returnFields[]=worklog&returnFields[]=woactivity&returnFields[]=href&offset=0&highlight=true)
     -> http.route: Str()
     -> masinstanceid: Str(assistdevtest)
     -> masappid: Str(assist)
     -> masworkspaceid: Str(masdev)
StartTimestamp: 2023-04-17 06:37:30.130142734 +0000 UTC
Timestamp: 2023-04-17 08:28:45.130818869 +0000 UTC
Count: 4
Sum: 1.316152
ExplicitBounds #0: 0.010000
ExplicitBounds #1: 0.100000
ExplicitBounds #2: 0.200000
ExplicitBounds #3: 0.400000
ExplicitBounds #4: 0.800000
ExplicitBounds #5: 1.000000
ExplicitBounds #6: 1.200000
ExplicitBounds #7: 1.400000
ExplicitBounds #8: 1.600000
ExplicitBounds #9: 1.800000
ExplicitBounds #10: 2.000000
ExplicitBounds #11: 4.000000
ExplicitBounds #12: 6.000000
ExplicitBounds #13: 8.000000
ExplicitBounds #14: 10.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 0
Buckets #3, Count: 4
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0
HistogramDataPoints #2
Data point attributes:
     -> service.name: Str(assistdevtest-ema-api)
     -> span.name: Str(HTTPS GET)
     -> span.kind: Str(SPAN_KIND_SERVER)
     -> status.code: Str(STATUS_CODE_UNSET)
     -> method: Str(GET)
     -> code: Int(200)
     -> http.target: Str(/api/v2/document-query/query)
     -> url: Str(https://ema-api-service.mas-assistdevtest-assist.svc/api/v2/document-query/query?projectId=b8b3d386-e234-410c-9468-c3f857a0d77c&collectionIds[]=0d4201fb-ac04-8046-0000-018408037aeb&naturalLanguageQuery=&passages=false&count=20&returnFields[]=metadata&returnFields[]=text&returnFields[]=result_metadata&returnFields[]=extracted_metadata&returnFields[]=highlight&returnFields[]=description&returnFields[]=description_longdescription&returnFields[]=wonum&returnFields[]=asset&returnFields[]=failure&returnFields[]=location&returnFields[]=assignment&returnFields[]=failurecode&returnFields[]=worklog&returnFields[]=woactivity&returnFields[]=href&offset=0&highlight=true)
     -> http.route: Str()
     -> masinstanceid: Str(assistdevtest)
     -> masappid: Str(assist)
     -> masworkspaceid: Str(masdev)
StartTimestamp: 2023-04-17 06:37:30.130142734 +0000 UTC
Timestamp: 2023-04-17 08:28:45.130818869 +0000 UTC
Count: 3
Sum: 0.771380
ExplicitBounds #0: 0.010000
ExplicitBounds #1: 0.100000
ExplicitBounds #2: 0.200000
ExplicitBounds #3: 0.400000
ExplicitBounds #4: 0.800000
ExplicitBounds #5: 1.000000
ExplicitBounds #6: 1.200000
ExplicitBounds #7: 1.400000
ExplicitBounds #8: 1.600000
ExplicitBounds #9: 1.800000
ExplicitBounds #10: 2.000000
ExplicitBounds #11: 4.000000
ExplicitBounds #12: 6.000000
ExplicitBounds #13: 8.000000
ExplicitBounds #14: 10.000000
Buckets #0, Count: 0
Buckets #1, Count: 0
Buckets #2, Count: 1
Buckets #3, Count: 2
Buckets #4, Count: 0
Buckets #5, Count: 0
Buckets #6, Count: 0
Buckets #7, Count: 0
Buckets #8, Count: 0
Buckets #9, Count: 0
Buckets #10, Count: 0
Buckets #11, Count: 0
Buckets #12, Count: 0
Buckets #13, Count: 0
Buckets #14, Count: 0
Buckets #15, Count: 0

@xwgao
Copy link
Author

xwgao commented Apr 17, 2023 •
edited
Loading

@kovrus The detailed setup steps are as below:

  1. I installed Community OpenTelemetry Operator 0.75.0 in my OpenShift 4.10 cluster.
  2. I created an OpenTelemetry collector with spec as below
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      otlp/spanmetrics:
        protocols:
          grpc:
            endpoint: 0.0.0.0:12346

    processors:
      batch:
        timeout: 10s
        send_batch_size: 10000
      spanmetrics:
        metrics_exporter: prometheus
        latency_histogram_buckets:
          [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 5s, 7s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.target
          - name: http.url
      metricstransform:
        transforms:
          - include: mas-assist.duration
            match_type: regexp
            action: update
            operations:
              - action: add_label
                new_label: masinstanceid
                new_value: assistdevtest
              - action: add_label
                new_label: masappid
                new_value: assist
              - action: add_label
                new_label: masworkspaceid
                new_value: masdev
              - action: update_label
                label: http.url
                new_label: url
              - action: update_label
                label: http.method
                new_label: method
              - action: update_label
                label: http.status_code
                new_label: code
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "service.name": "assistdevtest-ema-api",
              "http.target": "/api/v1/session-management/sessionsCount",
              "method": "GET"
            }
            new_name: mas-assist.collaborate_getsessioncount_duration_seconds
          - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "service.name": "assistdevtest-ema-api",
              "http.target": "/api/v2/document-query/query$",
              "method": "GET"
            }
            new_name: mas-assist.documentquery_searchdocuments_duration_seconds

    exporters:
      otlp:
        endpoint: "172.30.132.14:4317"
        tls:
          insecure: true
      logging:
        verbosity: detailed
      jaeger:
        endpoint: my-jaeger-collector-headless.jaeger-demo.svc:14250
        tls:
          ca_file: "/etc/pki/ca-trust/source/service-ca/service-ca.crt"
      prometheus:
        endpoint: "0.0.0.0:8889"
        send_timestamps: true
        metric_expiration: 1440m

    connectors:
      spanmetrics:
        namespace: mas-assist
        histogram:
          unit: s
          explicit:
            buckets: [10ms, 100ms, 200ms, 400ms, 800ms, 1s, 1200ms, 1400ms, 1600ms, 1800ms, 2s, 5s, 7s]
        dimensions:
          - name: http.method
          - name: http.status_code
          - name: http.target
          - name: http.url
          - name: http.route

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [spanmetrics, logging, jaeger, otlp]
        metrics:
          receivers: [otlp/spanmetrics, spanmetrics]
          processors: [batch, metricstransform]
          exporters: [prometheus, logging, otlp]
  mode: statefulset
  volumeMounts:
    - mountPath: /etc/pki/ca-trust/source/service-ca
      name: cabundle-volume
  volumes:
    - configMap:
        name: my-otelcol-cabundle
      name: cabundle-volume
  image: >-
    ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-contrib:0.75.0

As you can see above. In my otel collector, I added the following transformation to generate new metrics mas_assist_documentquery_searchdocuments_duration_seconds

         - include: mas-assist.duration
            match_type: regexp
            action: insert
            experimental_match_labels: {
              "service.name": "assistdevtest-ema-api",
              "http.target": "/api/v2/document-query/query$",
              "method": "GET"
            }
            new_name: mas-assist.documentquery_searchdocuments_duration_seconds
  1. I created an OpenTelemetry instrumentation with spec as below.
spec:
  exporter:
    endpoint: 'http://otel-collector-headless:4317'
  java:
    image: >-
      ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:1.24.0
  sampler:
    argument: '0.25'
    type: parentbased_traceidratio
  nodejs:
    image: >-
      ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:0.37.0
  resource: {}
  apacheHttpd:
    configPath: /usr/local/apache2/conf
    version: '2.4'
  propagators:
    - tracecontext
    - baggage
    - b3
    - jaeger
  dotnet:
    image: >-
      ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-dotnet:0.6.0
  python:
    env:
      - name: OTEL_EXPORTER_OTLP_ENDPOINT
        value: 'http://otel-collector-headless:4318'
    image: >-
      ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:0.38b0
  1. And I added the below annotation in my deployment to enable auto-instrumentation for my Node.js services.
spec:
  template:
    metadata:
      annotations:
        instrumentation.opentelemetry.io/inject-nodejs: "true"

  1. After then, I can see the metrics mas_assist_documentquery_searchdocuments_duration_seconds on Prometheus UI. And I added the below panel (Bucket Heat Map (s)) on Grafana dashboard to show the duration bucket heat map for the metrics mas_assist_documentquery_searchdocuments_duration_seconds.
    image

  2. Then I sent 3 more HTTP requests to endpoint /api/v2/document-query/query of service assistdevtest-ema-api, but the bucket heat map was NOT updated on Grafana dashboard. The count in le=0.2 bucket is still 2. But on the Jaeger UI, I can see totally 5 traces when searching traces with tags "http.target=/api/v2/document-query/query method=GET" for service assistdevtest-ema-api.

@jeromeinsf
Copy link

Am I interpreting the bug report as trying to update prometheus histogram values for the same timestamp in different transaction?

@xwgao
Copy link
Author

xwgao commented Apr 18, 2023

@jeromeinsf What does "update prometheus histogram values for the same timestamp in different transaction" mean? Thanks.

@atoulme atoulme removed the needs triage New item requiring triage label Apr 18, 2023
@github-actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Jun 19, 2023
@github-actions
Copy link
Contributor

This issue has been closed as inactive because it has been stale for 120 days with no activity.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Aug 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working closed as inactive connector/spanmetrics processor/metricstransform Metrics Transform processor Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@atoulme @jeromeinsf @kovrus @xwgao