Loki Exporter

Status
Stability	deprecated: logs
Distributions	contrib
Issues
Code Owners	@gramidt, @jpkrohling, @mar4uk

Exports data via HTTP to Loki.

Deprecation notice

This component is deprecated: Loki now supports native OTLP ingestion starting from v3. Grafana Cloud also supports OTLP native ingestion for logs. This component will be removed in November 2024.

Benefits of the new Loki OpenTelemetry log format

The new format for OpenTelemetry logs introduced in Loki V3 brings the following benefits:

Native support for the structure of OpenTelemetry logs enabling simpler querying (no more JSON parsing).
Simplified client configuration to send OpenTelemetry data using the standard OTLP protocol.

Loki log message format changes for OpenTelemetry logs

See OpenTelemetry Logs Data Model specification here.

OpenTelemetry log field	Pre Loki V3	Loki V3 through the Loki OTLP Endpoint
`Timestamp`	`timestamp`	`timestamp`
`ObservedTimestamp`	Not available	`metadata[observed_timestamp]`
`TraceId`	`traceid` field of the Loki JSON log message	`metadata[trace_id]`
`SpanId`	`spanid` field of the Loki JSON log message	`metadata[span_id]`
`TraceFlags`	Not available	`metadata[flags]`
`SeverityText`	`severity` field of the JSON log message (e.g. `Information`) and `level` label (e.g. `ERROR`, `INFO`...), the `detected_level` label is also available	`metadata[severity_text]`, the `detected_level` label is also available
`SeverityNumber`	Not available	`metadata[severity_number]`
`Body`	`body` field of the Loki JSON log message	The Loki log message. `__line__`in LogQL functions (e.g. `line_format`)
`InstrumentationScope`	`instrumentation_scope_name` field of the JSON log message	`metadata[scope_name]`
`Attributes`	JSON fields of the Loki log message	`metadata[xyz]` Where `xyz` is the `_` version of the OTel attribute name (e.g. `thread_name` Loki metadata for the `thread.name` OpenTelemetry attribute)
`Resource`	`service.name`, `service.namespace`, and `service.instance.id` are promoted as the following labels: `job=[${service.namespace}/]${service.name}`, instance=${service.instance.id}, exporter="OTLP"`. Other resource attributes are stored as JSON fields of the Loki log message with the prefix` resources_`(e.g.`resources_k8s_namespace_name`)	Default list of resource attributes promoted as Loki labels: `cloud.availability_zone`, `cloud.region`, `container.name`, `deployment.environment`, `k8s.cluster.name`, `k8s.container.name`, `k8s.cronjob.name`, `k8s.daemonset.name`, `k8s.deployment.name`, `k8s.job.name`, `k8s.namespace.name`, `k8s.pod.name`, `k8s.replicaset.name` `k8s.statefulset.name`, `service.instance.id`, `service.name`, `service.namespace`. Other resource attributes are by default promoted as Loki message metadata. ℹ️ The list of promoted resource attributes is configurable using Loki’s distributor config parameter `default_resource_attributes_as_index_labels` when using self managed Loki (here) or opening a support request when using Grafana Cloud

ℹ️ Additional conversion rules from OpenTelemetry Logs to Loki

All special characters, including . in attribute and resource attribute names, are converted into _ when they are mapped as Loki labels or metadata.
OTel attribute values with complex data types (i.e. arrays, nested structures) are converted into JSON strings

Migration instructions

Instrumentation migration

No changes are needed in the instrumentation layer. OpenTelemetry logs sources like OpenTelemetry SDKs or the OpenTelemetry Collector File Log Receiver don’t have to be modified.

Logs collection migration

Replace the OpenTelemetry Collector Loki Exporter by the OpenTelemetry Collector OTLP HTTP Exporter as OpenTelemetry logs should now be exported as is to the Loki OTLP endpoint.

OpenTelemetry Collector configuration migration

========================
=   BEFORE MIGRATION   =
========================

extensions:
  basicauth/loki:
    client_auth:
      username: <<username>>
      password: <<password>>

exporters:
  loki:
    auth:
      authenticator: basicauth/loki
    endpoint: https://loki.example.com:3100/loki/api/v1/push

service:
  extensions: [basicauth/loki]
  pipelines:
    logs:
      receivers: [...]
      processors: [...]
      exporters: [loki, ...]


========================
=   AFTER MIGRATION    =
========================

extensions:
  basicauth/loki:
    client_auth:
      username: <<username>>
      password: <<password>>

exporters:
  otlphttp/loki:
    auth:
      authenticator: basicauth/loki
    endpoint: http://loki.example.com:3100/otlp/v1/logs

service:
  extensions: [basicauth/loki]
  pipelines:
    logs:
      receivers: [...]
      processors: [...]
      exporters: [otlphttp/loki, ...]

When using Grafana Cloud, the Grafana Cloud OTLP endpoint should be used instead of the Loki OTLP endpoint. The connection details of the Grafana Cloud OTLP endpoint, OTLP HTTP URL and credentials are available using the Grafana Cloud "OpenTelemetry Collector" connection tile.
The promotion of OpenTelemetry attributes and resource attributes to Loki labels using the loki.attribute.labels and loki.resource.labels hints is replaced by the list of promoted attributes managed centrally in Loki.
The default list of resource attributes promoted as labels (see above) should be sufficient for most use cases.
ℹ️ Changes can be made to this list using the Loki distributor configuration parameter default_resource_attributes_as_index_labels (here) for self managed instances and opening a support ticket for Grafana Cloud.

LogQL queries migration

From `job` and `instance` to `service_name`, `service_namespace`, and `service_instance_id`

The Loki labels job and instance are no longer generated and are replaced by the service_name, service_namespace, and service_instance_id labels.

Example:

BEFORE
{job="ecommerce/frontend", instance="instance-1234567890"}

AFTER
{service_name="frontend", service_namespace="ecommerce", service_instance_id="instance-1234567890"}

From `| json | an_attribute=...` to `{an_attribute=...}` or `| an_attribute=...`

OTel log attributes, resource attributes, and fields are no longer stored in the JSON message but are stored as:

Loki message labels for promoted resource attributes (see list above),
Loki message metadata for other resource attributes, log attributes, and log fields.

LogQL statements | json | an_attribute=... must be converted to:

Promoted resource attributes: {an_attribute=...}
For other resource attributes, log attributes, and log fields: | an_attribute=...

Example:

BEFORE
{exporter="OTLP", job="frontend"} | json | resources_deployment_environment="production"

AFTER
{service_name="frontend"} | deployment_environment="production"

From `| json | traceid=...` and `| json | spanid=...` to `| trace_id=...` and `| span_id=...`

The log fields SpanID and TraceId were stored as the JSON fields spanid and traceid; they are now stored as metadata[span_id] and metadata[trace_id], LogQL queries must be changed accordingly.

Example:

BEFORE
{exporter="OTLP", job="/frontend"} |= "00960a472ea5b87954ca07902d66f914"

AFTER
{service_name="frontend"} | trace_id="00960a472ea5b87954ca07902d66f914"

From `line_format {{.body}}` to `line_format {{line}}`

The {{.body}} element of the JSON payload that used to hold the OTel log message body is now the message of the Loki log line and should be referenced as {{__line__}} in line_format calls.

Example:

BEFORE
{exporter="OTLP", job="frontend"} | json | line_format `[{{.level}}] {{.body}}`

AFTER
{service_name="frontend"} | line_format `[{{.detected_level}}] {{__line__}}`

Grafana visualizations migration

Navigation between traces and logs must be updated to account for the new Loki format form OpenTelemetry logs.

Tempo data source: Trace to Logs

To enable the "trace to logs" navigation from Tempo to Loki, navigate to the Grafana Tempo data source configuration screen, in the "Trace to logs" section,

Select a Loki data source on which logs to trace is configured for the new Loki format for OTel logs as described in the next section
Select "Use custom query" and specify the query:
```
{${__tags}} | trace_id="${__span.traceId}"
```

Configuration screenshot:

Loki data source: Log to Trace

To enable the "logs to trace" navigation from Loki to Tempo, navigate to the Grafana Loki data source configuration screen, in the "Derived fields" section, update or create a derived field with:

Name: Trace ID
Type: Label (note that this Label name may be missleading because it also supports Loki message metadata)
Label: trace_id
Internal link: activated
Select the Tempo data source on which "trace to logs" is configured as described above

Configuration screenshot:

Getting Started with the deprecated OpenTelemetry Collector Loki Exporter

The following settings are required:

endpoint (no default): The target URL to send Loki log streams to (e.g.: http://loki:3100/loki/api/v1/push).
default_labels_enabled (optional): The map that allows to disable default labels: exporter, job, instance, level. If default_labels_enabled is omitted then default labels will be added. If one of the labels is omitted in default_labels_enabled then this label will be added. Important to remember: If all default labels are disabled and there are no other labels added then the log entry would be dropped because at least one label should be present to successfully put the log record into Loki. The metric otelcol_lokiexporter_send_failed_due_to_missing_labels shows how many log records were dropped because no labels were specified

Example:

exporters:
  loki:
    endpoint: https://loki.example.com:3100/loki/api/v1/push
    default_labels_enabled:
      exporter: false
      job: true

Configuration via attribute hints

Labels

The Loki exporter can convert OTLP resource and log attributes into Loki labels, which are indexed. For that, you need to configure hints, specifying which attributes should be placed as labels. The hints are themselves attributes and will be ignored when exporting to Loki. The following example uses the attributes processor to hint the Loki exporter to set the event.domain attribute as label and the resource processor to give a hint to the Loki exporter to set the service.name as label.

processors:
  attributes:
    actions:
      - action: insert
        key: loki.attribute.labels
        value: event.domain

  resource:
    attributes:
      - action: insert
        key: loki.resource.labels
        value: service.name

Currently, Loki does not support label names with dots. That's why lokiexporter normalizes label names to follow Prometheus label names standard before sending requests to Loki. More information on label normalization could be found here

The promotion of multiple resource and log attributes to labels is done with single action with comma-separated desired labels:

processors:
  attributes:
    actions:
      - action: insert
        key: loki.attribute.labels
        value: event.domain, event.name

  resource:
    attributes:
      - action: insert
        key: loki.resource.labels
        value: service.name, service.namespace

Default labels are always set unless they are disabled with default_labels_enabled setting:

job=service.namespace/service.name
instance=service.instance.id
exporter=OTLP
level=severity

If service.name and service.namespace are present then job=service.namespace/service.name is set

If service.name is present and service.namespace is not present then job=service.name is set

If service.name is not present and service.namespace is present then job label is not set

If service.instance.id is present then instance=service.instance.id is set

If service.instance.id is not present then instance label is not set

The full list of settings exposed for this exporter are documented here with detailed sample configurations here.

More information on how to send logs to Grafana Loki using the OpenTelemetry Collector could be found here

Tenant information

It is recommended to use the header_setter extension to configure the tenant information to send to Loki. In case a static tenant should be used, you can make use of the headers option for regular HTTP client settings, like the following:

exporters:
  loki:
    endpoint: http://localhost:3100/loki/api/v1/push
    headers:
      "X-Scope-OrgID": acme

It is also possible to provide the loki.tenant attribute hint that specifies which resource or log attributes value should be used as a tenant. For example:

processors:
  resource:
    attributes:
    - action: insert
      key: loki.tenant
      value: host.name

In this case the value of the host.name resource attribute is used to group logs by tenant and send requests with the X-Scope-OrgID header set to relevant tenants.

If the loki.tenant hint attribute is present in both resource and log attributes, then the look-up for a tenant value from resource attributes takes precedence.

Format

To choose the format used for writing log lines by the exporter use the loki.format hint. For example:

processors:
  resource:
    attributes:
    - action: insert
      key: loki.format
      value: logfmt

The following formats are supported:

logfmt: Write logs as logfmt lines.
json: Write logs as JSON objects. It is the default format if no hint is present.
raw: Write the body of the log message as string representation.

Severity

OpenTelemetry uses record.severityNumber to track log levels where loki uses record.attributes.level for the same. The exporter automatically maps the two, except if a "level" attribute already exists.

Advanced Configuration

Several helper files are leveraged to provide additional capabilities automatically:

HTTP settings
Queuing and retry settings

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Loki Exporter

Deprecation notice

Benefits of the new Loki OpenTelemetry log format

Loki log message format changes for OpenTelemetry logs

Migration instructions

Instrumentation migration

Logs collection migration

LogQL queries migration

From `job` and `instance` to `service_name`, `service_namespace`, and `service_instance_id`

From `| json | an_attribute=...` to `{an_attribute=...}` or `| an_attribute=...`

From `| json | traceid=...` and `| json | spanid=...` to `| trace_id=...` and `| span_id=...`

From `line_format {{.body}}` to `line_format {{line}}`

Grafana visualizations migration

Tempo data source: Trace to Logs

Loki data source: Log to Trace

See Also

Getting Started with the deprecated OpenTelemetry Collector Loki Exporter

Configuration via attribute hints

Labels

Tenant information

Format

Severity

Advanced Configuration

Files

README.md

Latest commit

History

README.md

File metadata and controls

Loki Exporter

Deprecation notice

Benefits of the new Loki OpenTelemetry log format

Loki log message format changes for OpenTelemetry logs

Migration instructions

Instrumentation migration

Logs collection migration

LogQL queries migration

From job and instance to service_name, service_namespace, and service_instance_id

From | json | an_attribute=... to {an_attribute=...} or | an_attribute=...

From | json | traceid=... and | json | spanid=... to | trace_id=... and | span_id=...

From line_format {{.body}} to line_format {{__line__}}

Grafana visualizations migration

Tempo data source: Trace to Logs

Loki data source: Log to Trace

See Also

Getting Started with the deprecated OpenTelemetry Collector Loki Exporter

Configuration via attribute hints

Labels

Tenant information

Format

Severity

Advanced Configuration

From `job` and `instance` to `service_name`, `service_namespace`, and `service_instance_id`

From `| json | an_attribute=...` to `{an_attribute=...}` or `| an_attribute=...`

From `| json | traceid=...` and `| json | spanid=...` to `| trace_id=...` and `| span_id=...`

From `line_format {{.body}}` to `line_format {{line}}`