Loki exporter fails with "parse error: unexpected character inside braces: '.'" when using standard resource attributes

[cyrille-leclerc]

Component(s)

exporter/loki

What happened?

Description

Loki exporter fails with "parse error: unexpected character inside braces: '.'" when using standard resource attributes (e.g. service.name, host.name, pod.name) as explained in the Loki exporter docs.

The reason being that Loki label names don't support . when the docs suggests to use . in resource attribute names listed in loki.resource.labels and the Loki exporter doesn't do sensible escaping.

Steps to Reproduce

• Configure an OTel Collector to export logs to Loki using service.name as a Loki label. Config example:

receivers:
  otlp:
processors:
  batch:
  resource:
    attributes:
    - action: insert
      key: loki.resource.labels
      value: service.name
exporters:
  loki/cloud:
    endpoint: "https://1234567890:**@logs-prod-eu-west-0.grafana.net/loki/api/v1/push"
  logging/debug:
    loglevel: debug
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: 
        - batch
        - resource
      exporters: 
        - logging/debug
        - loki/cloud

• Send OTel logs to this OpenTelemetry collector, for example using a Java app instrumented with the OTel Java Instrumentation agent. Log message looks like the following in the OTel Collector stdout:

2023-01-09T11:10:20.346+0100    info    ResourceLog #0
Resource SchemaURL: https://opentelemetry.io/schemas/1.16.0
Resource attributes:
     -> deployment.environment: Str(production)
     -> host.arch: Str(aarch64)
     -> host.name: Str(cyrille-le-clerc-macbook.local)
     -> os.description: Str(Mac OS X 13.1)
     -> os.type: Str(darwin)
     -> process.command_line: Str(/opt/homebrew/Cellar/openjdk/19.0.1/libexec/openjdk.jdk/Contents/Home/bin/java -Xlog:gc*=info:file=/usr/local/var/log/my-shopping-cart/frontent_gc.log -javaagent:./../.otel/opentelemetry-javaagent-1.21.0.jar -Dserver.port=8080)
     -> process.executable.path: Str(/opt/homebrew/Cellar/openjdk/19.0.1/libexec/openjdk.jdk/Contents/Home/bin/java)
     -> process.pid: Int(16674)
     -> process.runtime.description: Str(Homebrew OpenJDK 64-Bit Server VM 19.0.1)
     -> process.runtime.name: Str(OpenJDK Runtime Environment)
     -> process.runtime.version: Str(19.0.1)
     -> service.name: Str(frontend)
     -> service.namespace: Str(com-shoppingcart)
     -> service.version: Str(1.0-SNAPSHOT)
     -> telemetry.auto.version: Str(1.21.0)
     -> telemetry.sdk.language: Str(java)
     -> telemetry.sdk.name: Str(opentelemetry)
     -> telemetry.sdk.version: Str(1.21.0)
     -> loki.resource.labels: Str(service.name)
ScopeLogs #0
ScopeLogs SchemaURL: 
InstrumentationScope com.mycompany.ecommerce.controller.OrderController 
LogRecord #0
ObservedTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2023-01-09 10:10:20.304 +0000 UTC
SeverityText: INFO
SeverityNumber: Info(9)
Body: Str(SUCCESS placeOrder orderId=792660 customerId=customer-36 price=200.0 paymentMethod=credit_cart shippingMethod=express shippingCountry=FR durationInNanos=1195130625)
Trace ID: ecd2d83676b15519bf60bbc6ba769e30
Span ID: d8279b572c5eb6c6
Flags: 1
ScopeLogs #1
ScopeLogs SchemaURL: 
InstrumentationScope com.mycompany.ecommerce.service.ProductServiceImpl 
        {"kind": "exporter", "data_type": "logs", "name": "logging/debug"}

Expected Result

Log message is persisted in Loki with a label using an escaped version of service.name: frontend

Actual Result

Persistence in Loki fail with the following exception message in the OTel Collector stdout. the OTel Collector Loki exporter didn't do any escaping of the unsupported .even though all resource attributes of the OTel Semantic Conventions contain dots (.).

2023-01-09T11:10:20.438+0100    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "loki/cloud", "error": "HTTP 400 \"Bad Request\": 1:40: parse error: unexpected character inside braces: '.'", "interval": "3.276474745s"}

Workaround

Create with OTel Collector processors the resource attributes used for Loki escaping the original desired resource attributes:

receivers:
  otlp:
processors:
  batch:
  resource:
    attributes:
    - action: insert
      key: service_name
      from_attribute: service.name
    - action: insert
      key: service_namespace
      from_attribute: service.namespace
    - action: insert
      key: deployment_environment
      from_attribute: deployment.environment
    - action: insert
      key: loki.resource.labels
      value: service_namespace, service_name, deployment_environment
exporters:
  loki/cloud:
    endpoint: "https://1234567890:**@logs-prod-eu-west-0.grafana.net/loki/api/v1/push"
  logging/debug:
    loglevel: debug
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: 
        - batch
        - resource
      exporters: 
        - logging/debug
        - loki/cloud

Collector version

v0.68.0

Environment information

Environment

OS: MacOS ARM

OpenTelemetry Collector configuration

receivers:
  otlp:
processors:
  batch:
  resource:
    attributes:
    - action: insert
      key: service_name
      from_attribute: service.name
    - action: insert
      key: service_namespace
      from_attribute: service.namespace
    - action: insert
      key: deployment_environment
      from_attribute: deployment.environment
    - action: insert
      key: loki.resource.labels
      value: service_namespace, service_name, deployment_environment
exporters:
  loki/cloud:
    endpoint: "https://1234567890:**@logs-prod-eu-west-0.grafana.net/loki/api/v1/push"
  logging/debug:
    loglevel: debug
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: 
        - batch
        - resource
      exporters: 
        - logging/debug
        - loki/cloud

Log output

2023-01-09T11:10:20.438+0100    info    exporterhelper/queued_retry.go:426      Exporting failed. Will retry the request after interval.        {"kind": "exporter", "data_type": "logs", "name": "loki/cloud", "error": "HTTP 400 \"Bad Request\": 1:40: parse error: unexpected character inside braces: '.'", "interval": "3.276474745s"}

Additional context

No response

[github-actions]

Pinging code owners for exporter/loki: @gramidt @gouthamve @jpkrohling @kovrus @mar4uk. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mar4uk]

Thank you Cyrille for reporting this bug!
The PR to fix README examples is created #17591

The proposal to escape labels with dots needs some investigation. To make the user experience better we should either escape labels containing dots on the Loki side (currently Loki doesn't support dots in labels, but it could change) or we should replace dots with underscores in Loki exporter to send labels in format Loki supports (the issue is already created #14113)

[cyrille-leclerc]

Thanks @mar4uk. The docs improvement is great and is good enough for the moment.
You are right that investigation is needed to provide a smoother UX related to . vs _