[exporter/Loki] Attributes and Resources has a attributes_ and resources_ prefix in Loki.

[kago-dk]

What happened?

Description

I am trying to take advantage of the “new” Loki Exporter in OTEL collector v0.60.0. Do I include all my resources and attributes from the filelog receiver as either loki.attribute.labels or loki.resource.labels? The answer is no based on this article https://grafana.com/blog/2020/08/27/the-concise-guide-to-labels-in-loki/.

If not, how do I avoid them showing up with the prefix of attributes (attributes_exception_stack_trace) or the prefix resources (resources_process_pid) in Loki.

Steps to Reproduce

Test using the testfile.log and otel-config.yaml

Expected Result

I am trying to avoid the attributes_ and resources_ shown in the (dark) screenshot. Even the hardcoded values with example_http_host_name and example_pod_name end up with the prefix in Loki (dark screenshot).

Collector version

v0.60.0

Environment information

## Environment
OS: Windows Server 2019

OpenTelemetry Collector configuration

# Receivers
receivers:
  filelog:
    include:
    - "C://temp//testfile*.log"

    operators:
    - type: json_parser
      timestamp:
        parse_from: attributes.TimeStamp
        layout: '%Y-%m-%dT%H:%M:%S.%LZ'
      severity:
        parse_from: attributes.LogLevel
    - type: move
      id: Resource1
      from: attributes.Resource1
      to: resource["host_name"]
    - type: move
      id: Resource2
      from: attributes.Resource2
      to: resource["process_pid"]
    - type: move
      id: Resource3
      from: attributes.Resource3
      to: resource["process_executable_name"]
    - type: move
      id: Resource4
      from: attributes.Resource4
      to: resource["host_ip"]
    - type: move
      id: Resource5
      from: attributes.Resource5
      to: resource["service_version"]
    - type: move
      id: Attribute1
      from: attributes.Attribute1
      to: attributes["http_method"]
    - type: move
      id: Attribute2
      from: attributes.Attribute2
      to: attributes["http_status_code"]
    - type: move
      id: Attribute3
      from: attributes.Attribute3
      to: attributes["http_url"]
    - type: move
      id: Attribute4
      from: attributes.Attribute4
      to: attributes["net_peer_ip"]
    - type: move
      id: Attribute5
      from: attributes.Attribute5
      to: attributes["error.code"]
    - type: remove
      field: attributes.TimeStamp
    - type: remove
      field: attributes.LogLevel
      
# Processors
processors:
  batch:
  attributes:
    actions:
    - action: insert
      key: loki.attribute.labels
      value: http_method,example_http_status_code
    - action: insert
      key: loki.resource.labels
      value: service_version      
    # http_method is a attribute and service_version is a resource.    
      
    # the following attributes are added manually here in the example, but would
    # probably be added by other processors or straight from the source
    - action: insert
      key: example_http_status_code
      value: 500
    - action: insert
      key: example_http_status
      value: 200      
  resource:
    attributes:
    - action: insert
      key: loki.attribute.labels
      value: example_http_status
    # example_http_status is a attribute  
    - action: insert
      key: loki.resource.labels
      value: host_ip,example_host_name,example_pod_name
    # host_ip,example_host_name,example_pod_name is a resource  
      
    # the following attributes are added manually here in the example, but would
    # probably be added by other processors or straight from the source
    - action: insert
      key: example_host_name
      value: guarana
    - action: insert
      key: example_pod_name
      value: guarana-pod-01      

# Exporters
exporters:
  logging:
    logLevel: debug
  loki:
    endpoint: https://localhost:3100/loki/api/v1/push
    tls:
      insecure: true

# Services
service:
  pipelines:
    logs:
      receivers: filelog
      processors: [attributes,resource]
      exporters: loki,logging
  telemetry:
    logs:
      output_paths: ["./logs/otelcol-contrib.log"]

Log output

No response

Additional context

• testfile.log

[kago-dk]

Pinging code owners: @jpkrohling.

[github-actions]

Pinging code owners: @gramidt @gouthamve @jpkrohling @kovrus @mar4uk. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[carlosjgp]

I'm very interested in this too. I'm looking into reducing the resource footprint on the clusters and replacing Promtail with OTL receiver

@kago-dk Looks like from 0.59 onwards you need to map the attributes into loki.attributes.labels or loki.resources.labels
https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/lokiexporter#labels

I'll be testing this within the next few weeks but I haven't tried yet

[carlosjgp]

(ignore my previous comment I was looking at the filelog exporter resources and attributes)

[jpkrohling]

I believe this to be a characteristic of the filelog, not Loki, but I'll take a look at this in a couple of weeks. In the meantime, would you be able to use the logging exporter with debug level? If this shows the same type of data there, we would have a confirmation that the problem isn't at the Loki exporter.

[mar4uk]

Tried to reproduce the issue using testfile.log and config.yaml attached. But I can't reproduce it. I don't see prefixes in loki:

Does the issue still remain?

[kago-dk]

I will retest using the latest OTEL Collector later this week.

[kago-dk]

Using 63.0 and running the query "{example_http_status_code="500"} | json" gives me:

And the verbosity: detailed (in 63) gives me: