Support for JSON log lines

[sandstrom]

Loki looks very promising! 🏆

Are there any plans to support ingestion of JSON log lines?

It seems to be a pretty common structure for logs these days. Here are some examples (can add more):

• https://www.loggly.com/blog/why-json-is-the-best-application-log-format-and-how-to-switch/
• https://docs.fluentd.org/v0.12/articles/parser_json
• https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#format-dictionaries
• https://docs.docker.com/config/containers/logging/json-file/
• https://jahed.io/2018/07/05/always-log-to-json/

[davkal]

What sort of support are you looking for?

Loki is format-agnostic and ingests log lines as string lines, may they be access logs, logfmt key/value pairs, or JSON.

Grafana's Explore UI shows you the log lines, and if they are JSON, some support for in-browser parsing to plot distributions of values. Notice how the fields in the log line have an orange underline, that means they were parsed successfully:

[r-moiseev]

Does it have an ability to search against json fields?

[draeron]

it would be great if from example, i could filter my logs through a level field contained in the log

[yubozhao]

@r-moiseev You can use regex as part of the query to "search" against your json fields.

@draeron I don't think this feature will be available soon. In the mean time, you can add the "key" you want to filtered as part of your label that send to the Loki server. Or You could add additional parser for promtail that parse your logs in json format.

[dano0b]

Just to share an example, in GKE/Stackdriver I can just search for key/values in the jsonPayload:
jsonPayload.@l="Warning"
Would be awesome to have some direct support for structured logging.

[4nte]

Here's a reference on how Filebeat handles structured/json logs.

[slim-bean]

With the 0.1.0 release there is included a pipeline which includes a json stage that allows extraction of json log data to be used in labels and/or metrics using JMESPath expressions.

[sandstrom]

Great! 🥇

[minhdanh]

I have a pod that emits logs in json format. But the logs are not being displayed as nested objects in Loki but a long string (the content of log field):

{"log":"{\"verb\":\"UPDATED\",\"event\":{\"metadata\":{\"name\":\"minio-backup.15c16e910ad17555\",\"namespace\":\"minio\",\"selfLink\":\"/api/v1/namespaces/minio/events/minio-backup.15c16e910ad17555\",\"uid\":\"75b2664e-d08a-11e9-aaf1-42010aa40066\",\"resourceVersion\":\"805300\",\"creationTimestamp\":\"2019-09-06T09:41:11Z\"},\"involvedObject\":{\"kind\":\"CronJob\",\"namespace\":\"minio\",\"name\":\"minio-backup\",\"uid\":\"a355d0fa-cf90-11e9-aaf1-42010aa40066\",\"apiVersion\":\"batch/v1beta1\",\"resourceVersion\":\"35722856\"},\"reason\":\"UnexpectedJob\",\"message\":\"Saw a job that the controller did not create or forgot: test-minio-backup\",\"source\":{\"component\":\"cronjob-controller\"},\"firstTimestamp\":\"2019-09-05T03:55:14Z\",\"lastTimestamp\":\"2019-09-06T10:46:18Z\",\"count\":1373,\"type\":\"Warning\"},\"old_event\":{\"metadata\":{\"name\":\"minio-backup.15c16e910ad17555\",\"namespace\":\"minio\",\"selfLink\":\"/api/v1/namespaces/minio/events/minio-backup.15c16e910ad17555\",\"uid\":\"75b2664e-d08a-11e9-aaf1-42010aa40066\",\"resourceVersion\":\"805295\",\"creationTimestamp\":\"2019-09-06T09:41:11Z\"},\"involvedObject\":{\"kind\":\"CronJob\",\"namespace\":\"minio\",\"name\":\"minio-backup\",\"uid\":\"a355d0fa-cf90-11e9-aaf1-42010aa40066\",\"apiVersion\":\"batch/v1beta1\",\"resourceVersion\":\"35722856\"},\"reason\":\"UnexpectedJob\",\"message\":\"Saw a job that the controller did not create or forgot: test-minio-backup\",\"source\":{\"component\":\"cronjob-controller\"},\"firstTimestamp\":\"2019-09-05T03:55:14Z\",\"lastTimestamp\":\"2019-09-06T10:41:14Z\",\"count\":1355,\"type\":\"Warning\"}}\n","stream":"stdout","time":"2019-09-06T10:46:18.681193448Z"}

Does Loki automatically handle json format or something else still missing?

[Lucaber]

Hi @minhdanh
i just found a solution for eventrouter :)

    - match: 
        selector: '{app="eventrouter"}'
        stages:
          - json:
              expressions:
                log:
          - json:
              source: log
              expressions:
                event_verb: verb
                event:
          - json:
              source: event
              expressions:
               event_reason: reason
               involvedObject:
               source:
          - json:
              source: involvedObject
              expressions:
                event_kind: kind
                event_namespace: namespace
                event_name: name
          - json:
              source: source
              expressions:
                event_source_host: host
                event_source_component: component
          - labels:
              event_verb:
              event_kind:
              event_reason:
              event_namespace:
              event_name:
              event_source_host:
              event_source_component:

[minhdanh]

Hi @Lucaber
Thanks for the solution. But looks like it doesn't work for me.
I added your snippet to promtail's pipelineStages config: https://github.com/grafana/loki/blob/master/production/helm/promtail/values.yaml#L29

promtail:
  pipelineStages:
    - match:
        selector: '{app="eventrouter"}'
        stages:
          - json:
              expressions:
                log:
          - json:
              source: log
              expressions:
                event_verb: verb
                event:
          - json:
              source: event
              expressions:
                event_reason: reason
                involvedObject:
                source:
          - json:
              source: involvedObject
              expressions:
                event_kind: kind
                event_namespace: namespace
                event_name: name
          - json:
              source: source
              expressions:
                event_source_host: host
                event_source_component: component
          - labels:
              event_verb:
              event_kind:
              event_reason:
              event_namespace:
              event_name:
              event_source_host:
              event_source_component:

Then deployed promtail again. But the still the same in Loki.

[slim-bean]

@minhdanh loki only knows logs as byte arrays for storage, everything is basically a string.

Your log example looks like the output of a docker log line, which has json nested inside json.

I'm not quite sure what you are ultimately looking for in Grafana? But the simplest pipeline config would just include the docker stage which will unroll the docker json, and set the log json as the log line, which should then be un-esacaped and appear like normal json.

The config @Lucaber pasted is setting a series of labels from the log but is not manipulating the output sent to Loki, you must use an output pipeline stage for this (the docker stage internally is just a json, timestamp, label, and output stage)

Also @Lucaber I believe you could make your config a little more concise and probably a little faster:

promtail:
  pipelineStages:
    - match:
        selector: '{app="eventrouter"}'
        stages:
          - docker:
          - json:
              expressions:
                event_verb: verb
                event_kind: event.involvedObject.kind
                event_reason: event.reason
                event_namespace: event.involvedObject.namespace
                event_name: event.metadata.name
                event_source_host: event.source.host
                event_source_component: event.source.component
           - labels:
               event_verb:
               event_kind:
               event_reason:
               event_namespace:
               event_name:
               event_source_host:
               event_source_component:

If all your logs are docker, you could also move that outside the match:

promtail:
  pipelineStages:
    - docker:
    - match:
        selector: '{app="eventrouter"}'
        stages:
          - json:
...

The advantage of using the docker stage is that it will set the timestamp from the log line as well as set the output to the un-escaped json of the actual log message

[Lucaber]

Ohh yes, i was looking for loki labels to easily filter the logs.
I previously tried something similar:

- match:
    selector: '{app="eventrouter"}'
    stages:
      - json:
          expressions:
            event_verb: log.verb
      - labels:
          event_verb:

I also tried verb instead of log.verb but my label was still empty (null). Maybe the docker stage does the trick, i will try this again later.

[minhdanh]

@slim-bean Thank you. Apparently I removed docker: {} in the pipeline stages and it didn't work. I added it again and it's working with correct json format in Grafana.

I'm not quite sure what you are ultimately looking for in Grafana?

With json supported by Loki I was expecting to search/query the logs using something like object.property=value. This is possible, right?