[exporter/loki] Loki v3.0's generated service_name is invalid when service.namespace is defined

[cyrille-leclerc]

Component(s)

Loki Exporter

What happened?

Description

Loki v3.0, as part of its OpenTelemetry support, generates a service_name label if none is provided and the current heuristic (here) contradicts the Prometheus and OpenMetrics Compatibility specification for logs emitted by the OTel Collector Loki Exporter using the value of the job label ignoring that this job label could be the composition of service.name with service.namespace.

Ideas to fix the problem:

1. Evolve the OTel Collector Loki Exporter to emit the service_name and service_namespace labels so that Loki's defaulting heuristic doesn't use the job value
2. Extend Loki's service_name default heuristic to detect messages with exporter=OTLP and then get service_name parsing the json payload and extract the field resources/service.name

Steps to Reproduce

For example, the OTel logs emitted from a service identified by (service.namespace=ecommerce, service.name=frontend) and exported by the OTel Collector Loki exporter will produce logs with:

• Labels
  • job=ecommerce/frontend
  • service_name=ecommerce/frontend <-- added by Loki's default service_name heuristic, invalid, expected value is frontend
  • exporter=OTLP
• Payload {"body": "the message", "resources":{"service.name"="frontend", "service.namespace"="ecommerce"...}}

Expected Result

The generated label service_name should be set to frontend.

Actual Result

The generated label service_name is set to ecommerce/frontend instead of frontend.

Collector version

v0.98

Environment information

Environment

OS: MacOS

OpenTelemetry Collector configuration

loki/grafana_cloud_logs:
    # https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/lokiexporter/README.md
    endpoint: "${env:GRAFANA_CLOUD_LOKI_URL}"
    auth:
      authenticator: basicauth/grafana_cloud_loki

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[aloisiobilck]

@cyrille-leclerc
How did you manage to map from service_name to ecommerce/frontend ?
My service.name is always unknown_service
In my case I am using Kubernetes.

Thanks

[cyrille-leclerc]

Hello Aloisio,

As discussed with @sandeepsukhani and the Loki team, the solution is to adopt the new Loki OTLP Endpoint as a replacement to the OpenTelemetry Collector Loki Exporter.

Using the Loki OTLP Endpoint is documented in Ingesting logs to Loki using OpenTelemetry Collector

Here is an example of OTel Collector config that sends all OTel traces, metrics, and Logs to Grafana Cloud using the Grafana Cloud OTLP endpoint rather converting to Prometheus and Loki in the OTel Collector. The same type of config works with Mimir, Loki, and Tempo; you just have to spcify one OTLP endpoint for each Telemetry type.

# Tested with Otel Collector Contrib v0.100.0

receivers:
  otlp:
    protocols:
      grpc:
      http:
  hostmetrics:
    scrapers:
      load:
      memory:
      #...

processors:
  batch:
  resourcedetection:
    # Enriches telemetry data with resource information from the host
    detectors: ["env", "system"]
    override: false
  transform/add_resource_attributes_as_metric_attributes:
    # add context labels to Prometheus metrics in order to enable group-by and filter-by on metrics
    error_mode: ignore
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["deployment.environment"], resource.attributes["deployment.environment"])
          - set(attributes["service.version"], resource.attributes["service.version"])
  transform/truncate_attributes:
    # As Mimir label and Loki metad ta values are limited to 2048 characters 
    # truncate:
    #   - metrics: attributes and resource atributes
    #   - span: resource attributes in case some metrics are derived from spans in Tempo / Grafana Cloud
    #   - logs: resource attributes
    error_mode: ignore
    trace_statements:
      - context: resource
        statements:
          - truncate_all(attributes, 2048)
      - context: span
        statements:
          - truncate_all(attributes, 2048)
    metric_statements:
      - context: resource
        statements:
        - truncate_all(attributes, 2048)
      - context: datapoint
        statements:
          - truncate_all(attributes, 2048)
    log_statements:
    - context: resource
      statements:
        - truncate_all(attributes, 2048)

exporters:
  otlphttp/grafana_cloud:
    endpoint: "${env:OTEL_EXPORTER_OTLP_ENDPOINT}"
    auth:
      authenticator: basicauth/grafana_cloud

extensions:
  basicauth/grafana_cloud:
    client_auth:
      username: "${env:GRAFANA_CLOUD_INSTANCE_ID}"
      password: "${env:GRAFANA_CLOUD_API_KEY}"

service:
  extensions: [basicauth/grafana_cloud]
  pipelines:
    traces:
      receivers: [otlp, jaeger]
      processors: 
        - resourcedetection
        - transform/truncate_attributes
        - batch
      exporters: [otlphttp/grafana_cloud]
    metrics:
      receivers: [otlp, hostmetrics]
      processors:
        - resourcedetection
        - transform/add_resource_attributes_as_metric_attributes
        - transform/truncate_attributes
        - batch
      exporters: [otlphttp/grafana_cloud]
    logs:
      receivers: [otlp]
      processors:
       - resourcedetection
       - transform/truncate_attributes
       - batch
      exporters: [otlphttp/grafana_cloud]