lokiexporter: Empty logs sended from otel collector

[gillg]

Describe the bug
Otel Collector receive logs from fluent-bit, process them as batch, and export them to console + loki.
On console everything seems good, but on loki they are "empty"

All my logs have no "body" or any attributes.

Console exporter output :

LogRecord open-telemetry/opentelemetry-collector#7
Timestamp: 1614706050024387900
Severity:
ShortName:
Body: <Unknown OpenTelemetry attribute value type "NULL">
Attributes:
     -> RecordNumber: INT(134181)
     -> TimeGenerated: STRING(2021-03-02 18:27:29 +0100)
     -> TimeWritten: STRING(2021-03-02 18:27:29 +0100)
     -> EventID: INT(403)
     -> EventType: STRING(Information)
     -> EventCategory: INT(4)
     -> Channel: STRING(Windows PowerShell)
     -> SourceName: STRING(PowerShell)
     -> ComputerName: STRING([OBFUSCATED])
     -> Data: STRING()
     -> Sid: STRING()
     -> Message: STRING(Engine state is changed from Available to Stopped.

Details:
        NewEngineState=Stopped
        PreviousEngineState=Available

        SequenceNumber=15
[OBFUSCATED]
)
     -> StringInserts: STRING([OBFUSCATED])
     -> hostname: STRING([OBFUSCATED])
     -> instance: STRING([OBFUSCATED])
     -> job: STRING(winlog)
     -> fluent.tag: STRING(winevent.log)

Steps to reproduce

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
  fluentforward:
    endpoint: 0.0.0.0:8006

exporters:
  logging:
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
    # Whitelist of labels. If you try to push an unlisted label the entier log will be dropped
    labels:
      attributes:
        severity: ""
        Type: "severity"
        EventType: ""
        hostname: "instance"
        instance: ""
        job: ""
        #fluent.tag: "job"

processors:
  batch:

extensions:
  health_check:

service:
  extensions: [health_check]
  pipelines:
    logs:
      receivers: [otlp, fluentforward]
      processors: [batch]
      exporters: [logging, loki]

What did you expect to see?
Logs on loki...
Exemple with fluent-bit > fluetnd > loki

What did you see instead?
See above

What version did you use?
Loki : grafana/loki:master (2.1.0+)
Otel collector: otel/opentelemetry-collector-contrib:latest

[gillg]

Maybe we should reuse the mechanism inspired from fluentd loki plugin.
https://github.com/grafana/loki/blob/master/cmd/fluentd/lib/fluent/plugin/out_loki.rb#L151

I have the feeling than otel doesn't sends any "OTEL Attributes" to loki.

[gillg]

Hum... I understand, FluentForward protocol seems not have a "raw" log body.
It contains only attributes, but an attribute can be "message" and containing the raw log.
OTEL collector take this assumption during the collect : https://github.com/open-telemetry/opentelemetry-collector/blob/73db88faef0a99dfb3317227052fea5299d52ec5/receiver/fluentforwardreceiver/conversion.go#L164

In my case, I use Winlog input (https://github.com/fluent/fluent-bit/tree/master/plugins/in_winlog) and this plugin parse the win eventlog then forward all fields without adding a custom field message.
A serialized view of each entry (in JSON for exemple) should be the real log.

So, I have some questions to discuss.

1. Do we have to add a full attributes serialized body in this else ? https://github.com/open-telemetry/opentelemetry-collector/blob/73db88faef0a99dfb3317227052fea5299d52ec5/receiver/fluentforwardreceiver/conversion.go#L175
   Arbitrary in JSON ? What about the case where you have "message" + other attributes ?

2. Do we have to force a non empty body here ?

   opentelemetry-collector-contrib/exporter/lokiexporter/exporter.go

   Line 188 in fad1892

   func convertLogToLokiEntry(lr pdata.LogRecord) *logproto.Entry {

   
   It seems usefull to avoid unconsistent / useless entries on Loki server. Loki has no concepts of "attributes" in database or during logs push. You have "labels" which are used to query logs, but the body is raw and can be parsed during the query phase to extract attributes on the fly. Natively it supports json / logfmt / custom regex.
   I think we should always create a new serialized message combinating all attributes and the potential current body in a new attribute body (for example). If we don't do that, all enriched attributes by the collector will be losed on loki side.

[gillg]

FluentReceiver related issue here : #14718

[gramidt]

Thank you for filing this issue, @gillg!

Hum... I understand, FluentForward protocol seems not have a "raw" log body.
It contains only attributes, but an attribute can be "message" and containing the raw log.
OTEL collector take this assumption during the collect : https://github.com/open-telemetry/opentelemetry-collector/blob/73db88faef0a99dfb3317227052fea5299d52ec5/receiver/fluentforwardreceiver/conversion.go#L164

In my case, I use Winlog input (https://github.com/fluent/fluent-bit/tree/master/plugins/in_winlog) and this plugin parse the win eventlog then forward all fields without adding a custom field message.
A serialized view of each entry (in JSON for exemple) should be the real log.

So, I have some questions to discuss.

1. Do we have to add a full attributes serialized body in this else ? https://github.com/open-telemetry/opentelemetry-collector/blob/73db88faef0a99dfb3317227052fea5299d52ec5/receiver/fluentforwardreceiver/conversion.go#L175
   Arbitrary in JSON ? What about the case where you have "message" + other attributes ?

I can't speak to the 'fluentforwardreceiver', but I will find someone who can.

2. Do we have to force a non empty body here ?

   opentelemetry-collector-contrib/exporter/lokiexporter/exporter.go

   Line 188 in fad1892

   func convertLogToLokiEntry(lr pdata.LogRecord) *logproto.Entry {

   It seems usefull to avoid unconsistent / useless entries on Loki server. Loki has no concepts of "attributes" in database or during logs push. You have "labels" which are used to query logs, but the body is raw and can be parsed during the query phase to extract attributes on the fly. Natively it supports json / logfmt / custom regex.
   I think we should always create a new serialized message combinating all attributes and the potential current body in a new attribute body (for example). If we don't do that, all enriched attributes by the collector will be losed on loki side.

I was torn on this when implementing the exporter. Based on the collector design, it is the responsibility of the receiver to properly format the data into the format collector pipelines understand. In this particular case, we're relying on the 'fluentforwarderreceiver' to ensure that the log body is properly filled prior to sending the data through the remainder of the pipeline. As you mentioned, it currently looks for "message" or "key" when creating the log body. While most of the time the data sent from Fluentd/Fluentbit contain either of these fields, it is not always the case (per your experience). This can be resolved upstream within Fluentd by transforming the data prior to sending it to the collector, but sometimes operators don't have control over that particular configuration / subsystem.

Do you feel that a mechanism within the 'fluentforwarderreceiver' that allowed you to specify how to fill the log message when one isn't present would solve the empty log message issue for you? ( #14718) Or would like to have an option within the Loki exporter to dynamically fill the log message with attributes if one is not present? Or something else?

As a Loki user, I would also love to get your thoughts on #2290 .

[gillg]

@gramidt Thanks for your answer, I tried to workaround at fluentbit side... But after a long battle I abandoned and decide to learn Go to make my PR ^^
In fact, there is no way to serialize all Fluent fields in one field. And the case of windows eventlog exporter is a good case of well structured log. The "log" is the serialization of all fields if we compare it with a classic text entry.
In term of OTEL spec, the distinction between log body / attributes seems obscure. Else we consider they are redundant informations (one structured and one unstructured) else we consider attributes as an enrichment/complement of log body.

• In the first case, we could prefer attributes as raw log on the final exporter if it doesn't support both.
• In the second case, we should join attributes + body (as in my PR) to avoid loosing informations if the final exporter doesn't support both.

On OTLP exporter (for exemple) as it handle attributes + body, we have nothing to merge or drop.

[gramidt]

Thank you for the prompt followup, @gillg!

• In the first case, we could prefer attributes as raw log on the final exporter if it doesn't support both.
• In the second case, we should join attributes + body (as in my PR) to avoid loosing informations if the final exporter doesn't support both.

On OTLP exporter (for exemple) as it handle attributes + body, we have nothing to merge or drop.

Hmm..I'm going to need to think through this a bit more. At this time, I feel that if manipulation is being performed to properly fill the log data it must be done within the corresponding receiver or maybe even a processor. There are cases where it would make sense to do certain transformations on the exporter side, but only when it's a requirement of that particular destination and not a shared requirement across multiple exporters. In this particular case, I personally believe the 'fluentforwardreceiver' could be updated to provide the necessary configuration to fill the log body however you see fit, so that any downstream exporter can rely on the log without having to re-implement logic to fill the log body.

[gramidt]

@bogdandrutu @tigrannajaryan - Could you assign this to me?

[gillg]

I'm pretty aggree, but if we consider OTEL Collector as a data pipeline, it's acceptable to have a different log to export, between the receiver and the exporter. OTEL logs are defined here : https://github.com/open-telemetry/opentelemetry-collector/blob/2a743aaa117f911976049628a926a6efead5e417/internal/data/protogen/logs/v1/logs.pb.go#L268, In my opinion, if the target (after the exporter) can handle OTEL attributes it's ok, but if it can't we should adapt the log to avoid any loss of information.

What is the objective to add processors on OTEL collector if it works on attributes and if you don't export attribute on your target ?

[gramidt]

For tracking / reference: Similar issues and potential improvements discussed to the log receivers to help address the empty log/message field:

#2851

[gillg]

Hello, I can't reopen my PR, but we always need to find a solution for this issue.