[exporter/Loki] multiple attributes for loki not working

[saritatewari4]

Component(s)

exporter/loki

Describe the issue you're reporting

Hello,

I am trying to set multiple labels in loki dashboard, but it's are not working. In the below configuration one label at a time is working but both of them not working together. I can always see app-name in loki dashboard, and if I am adding only traceId id as a label then I can see traceId label after removing app_name.This app_name and traceId I am setting from flauntD manifest file

config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:
      fluentforward:
        endpoint: 0.0.0.0:8006
    exporters:
      otlp:
        endpoint: http://tempo
        tls:
          insecure: true  
      logging:
        logLevel: "INFO"
        verbosity: detailed
      loki:
        endpoint: http://loki:3100/loki/api/v1/push
        tls:
          insecure: true
    processors:
      batch:
      attributes:
        actions:
          - action: insert
            key: loki.attribute.labels
            value: app_name
          - action: insert
            key: loki.attribute.labels
            value: traceId
    service:
      pipelines:
        traces:
          receivers: [otlp]
          exporters: [otlp]
        logs:
          receivers: [ fluentforward ]
          processors: [ attributes, batch  ]
          exporters: [ logging, loki ]

This is my loki dashboard:

Please help me where the configuration is wrong.

[github-actions]

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @kovrus @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mar4uk]

the attributes processor should look like this:

    processors:
      batch:
      attributes:
        actions:
          - action: insert
            key: loki.attribute.labels
            value: app_name, traceId

To promote multiple attributes to labels you need to specify them coma-separated in one action
I will update README to mention this case.

By the way, traceID is not a good candidate to label promotion. Its value doesn't have boundaries (there are infinite values of traceIDs) which leads to high cardinality issue in Loki.

The best practice is to use filter expressions to query logs for a specific traceID:

{app_name="my-app"} |= “traceID=2612c3ff044b7d02”

A few articles on Loki labels:
https://grafana.com/docs/loki/latest/best-practices/
https://grafana.com/blog/2020/08/27/the-concise-guide-to-labels-in-loki/

[saritatewari4]

@mar4uk Thank you so much for your response. I was planning to change trace id to correlation Id.

I will definitely follow the best practices which you have shared.

Thanks again.

[mar4uk]

Another way to correlate logs and traces is configuring Derived fields in Loki datasource configuration:
https://grafana.com/docs/grafana/latest/datasources/loki/#configure-derived-fields

This will parse the log message, find traceID and make it a link to the tracing datasource:

[jpkrohling]

Should this be closed?