Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[processors/filter] include drop metrics with colon in their name #31911

Closed
rlanore opened this issue Mar 22, 2024 · 4 comments
Closed

[processors/filter] include drop metrics with colon in their name #31911

rlanore opened this issue Mar 22, 2024 · 4 comments
Labels
discussion needed Community discussion needed processor/filter Filter processor

Comments

@rlanore
Copy link

rlanore commented Mar 22, 2024

Component(s)

processor/filter

What happened?

Description

Opentelemetry drop my metric if it contain colon in it's name when use processors/filter include feature

Steps to Reproduce

Deploy last opentelemetry k8s operator and add opentelemetry colector manifest

Expected Result

I must see all metric on sdout

Actual Result

I see only 'up' metric

Collector version

0.96.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

receivers:
      otlp:
        protocols:
          grpc:
            endpoint: "0.0.0.0:4317"
          http:
            endpoint: "0.0.0.0:4318"
      prometheus:
  ... ...
    exporters:
      logging:
        verbosity: detailed
    extensions:
      health_check:
      pprof:
        endpoint: :1888
      zpages:
        endpoint: :55679
    processors:
      filter/limits:
        metrics:
          include:
            match_type: strict
            metric_names:
            - up
            - namespace_cpu:kube_pod_container_resource_limits:sum
            - namespace_cpu:kube_pod_container_resource_requests:sum
            - namespace_memory:kube_pod_container_resource_limits:sum
            - namespace_memory:kube_pod_container_resource_requests:sum
      batch/metrics:
        timeout: 30s
        send_batch_size: 500
    service:
      extensions: [pprof, zpages, health_check]
      pipelines:
        metrics:
          receivers: [prometheus, otlp]
          processors: [filter/limits,batch/metrics, attributes/metrics]
          exporters: [logging,otlp]
      telemetry:
        logs:
          level: debug
        metrics:
          address: 0.0.0.0:8888
          level: basic

Log output

Nothing usefull

Additional context

No response

@rlanore rlanore added bug Something isn't working needs triage New item requiring triage labels Mar 22, 2024
@github-actions github-actions bot added the processor/filter Filter processor label Mar 22, 2024
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@TylerHelmuth
Copy link
Member

@rlanore the filterprocessor already handles metric names with colons as it is doing a string lookup. Make sure the metrics you want to keep have the extact name namespace_cpu:kube_pod_container_resource_limits:sum. You can use a debugexporter with verbosity: detailed to view the data as the filterprocessor will see it.

@TylerHelmuth TylerHelmuth added waiting for author discussion needed Community discussion needed and removed needs triage New item requiring triage bug Something isn't working waiting for author labels Mar 22, 2024
@rlanore
Copy link
Author

rlanore commented Mar 22, 2024

Hi @TylerHelmuth ,
when remove filter processors i can get value from curl prometheus server

@rlanore
Copy link
Author

rlanore commented Mar 22, 2024

hummm ok I did not understood use of colon into promql. It's not the name of metrics but a recording rule computed at prometheus level. So with only 'kube_pod_container_resource_limits' in the include statement and the recording rule this works.
Sorry

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion needed Community discussion needed processor/filter Filter processor
Projects
None yet
Development

No branches or pull requests

2 participants