v1.3.0

With 1.3.0 we are excited to announce several improvements focusing on performance!

First and most significant is the Query Frontend:

• 1442 cyriltovena: Loki Query Frontend

The query frontend allows for sharding queries by time and dispatching them in parallel to multiple queriers, giving true horizontal scaling ability for queries. Take a look at the jsonnet changes to see how we are deploying this in our production setup. Keep an eye out for a blog post with more information on how the frontend works and more information on this exciting new feature.

In our quest to improve query performance, we discovered that gzip, while good for compression ratio, is not the best for speed. So we introduced the ability to select from several different compression algorithms:

• 1411 cyriltovena: Adds configurable compression algorithms for chunks

We are currently testing out LZ4 and snappy, LZ4 seemed like a good fit however we found that it didn't always compress the same data to the same output which was causing some troubles for another important improvement:

• 1438 pstibrany: pkg/ingester: added sync period flags

Extending on the work done by @bboreham on Cortex, @pstibrany added a few new flags and code to synchronize chunks between ingesters, which reduces the number of chunks persisted to object stores and therefore also reduces the number of chunks loaded on queries and the amount of de-duplication work which needs to be done.

As mentioned above, LZ4 was in some cases compressing the same data with a different result which was interfering with this change, we are still investigating the cause of this issue (It may be in how we implemented something, or may be in the compression code itself). For now we have switched to snappy which has seen a reduction in data written to the object store from almost 3x the source data (with a replication factor of 3) to about 1.5x, saving a lot of duplicated log storage!

Another valuable change related to chunks:

• 1406 slim-bean: allow configuring a target chunk size in compressed bytes

With this change you can set a chunk_target_size and Loki will attempt to fill a chunk to approx that size before flushing (previously a chunk size was a hard coded 10 blocks where the default block size is 262144 bytes). Larger chunks are beneficial for a few reasons, mainly on reducing API calls to your object store when performing queries, but also in reducing overhead in a few places, especially when processing very high volume log streams.

Another big improvement is the introduction of accurate rate limiting when running microservices:

• 1486 pracucci: Add ingestion rate global limit support

Previously the rate limit was applied at each distributor, however with traffic split over many distributors the limit would need to be adjusted accordingly. This meant that scaling up distributors required changing the limit. Now this information is communicated between distributors such that the limit should be applied accurately regardless of the number of distributors.

And last but not least on the notable changes list is a new feature for Promtail:

• 1275 bastjan: pkg/promtail: IETF Syslog (RFC5424) Support

With this change Promtail can receive syslogs via TCP! Thanks to @bastjan for all the hard work on this submission!

Important things to note:

• 1519 Changes a core behavior in Loki regarding logs with duplicate content AND duplicate timestamps, previously Loki would store logs with duplicate timestamps and content, moving forward logs with duplicate content AND timestamps will be silently ignored. Mainly this change is to prevent duplicates that appear when a batch is retried (the first entry in the list would be inserted again, now it will be ignored). Logs with the same timestamp and different content will still be accepted.
• 1486 Deprecated -distributor.limiter-reload-period flag / distributor's limiter_reload_period config option.

As always, the full list of changes can be found in the CHANGELOG!

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

• https://hub.docker.com/r/grafana/loki
• https://hub.docker.com/r/grafana/promtail

$ docker pull "grafana/loki:v1.3.0"
$ docker pull "grafana/promtail:v1.3.0"

Binary

We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:

$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.3.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

v1.2.0

One week has passed since the last Loki release, and it's time for a new one!

Notable Changes

We have continued our work making our API Prometheus-compatible. The key
changes centered around API compatibility are:

• 1370 slim-bean: Change /loki/api/v1/label to loki/api/v1/labels
• 1381 owen-d: application/x-www-form-urlencoded support

Meanwhile, @pstibrany has done great work ensuring that Loki handles hash
collisions properly:

• 1247 pstibrany: pkg/ingester: handle labels mapping to the same fast fingerprint.

As always, the full list of changes can be found in the CHANGELOG!

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

• https://hub.docker.com/r/grafana/loki
• https://hub.docker.com/r/grafana/promtail

$ docker pull "grafana/loki:v1.2.0"
$ docker pull "grafana/promtail:v1.2.0"

Binary

We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:

$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.2.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

v1.1.0

It's been a busy 2 weeks since the 1.0.0 release and quite a few important PR's have been merged to Loki.

The most significant:

• 1322 rfratto: Fix v1 label API to be Prometheus-compatible

Some might call this a breaking change, we are instead calling it a bug fix as our goal was to be prometheus compatible and we were not 😃

But please be aware if you are using the /loki/api/v1/label or /loki/api/v1/label/<name>/values the JSON result will be different in 1.1.0

Old result:

{
  "values": [
    "label1",
    "label2",
    "labeln"
  ]
}

New result:

{
  "status": "success",
  "data": [
    "label1",
    "label2",
    "labeln"
  ]
}

ALSO IMPORTANT

• 1160 daixiang0: replace gzip with zip

Binaries will now be zipped instead of gzipped as many people voiced their opinion that zip is likely to be installed on more systems by default.

If you had existing automation to download and install binaries this will have to be updated to use zip instead of gzip

Notable Fixes and Improvements

• Broken version info in startup log message:

  1095 pstibrany: Makefile changes to allow easy builds with or without vendoring. Also fixes version bug for both cases.

• The hashing algorithm used to calculate the hash for a stream was creating hash collisions in some instances.
  Please Note this is just one part of the fix and is only in Promtail, the second part for Loki can be tracked in PR1247 which didn't quite make the cut for 1.1.0 and will be in 1.2.0:

  1254 pstibrany: pkg/promtail/client: Handle fingerprint hash collisions

• Thank you @putrasattvika for finding and fixing an important bug where logs were some logs were missed in a query shortly after a flush!

  1299 putrasattvika: storage: fix missing logs with batched chunk iterator

• Thank you @danieldabate for helping to again improve our API to be more Prometheus compatible:

  1355 danieldabate: HTTP API: Support duration and float formats for step parameter

• LogQL will support duration formats that are not typically handled by Go like [1d] or [1w]

  1357 cyriltovena: Supports same duration format in LogQL as Prometheus

The full list of changes can be found in the CHANGELOG

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

• https://hub.docker.com/r/grafana/loki
• https://hub.docker.com/r/grafana/promtail

$ docker pull "grafana/loki:v1.1.0"
$ docker pull "grafana/promtail:v1.1.0"

Binary

We provide pre-compiled binary executables for the most common operating systems and architectures.
Choose from the assets below for the application and architecture matching your system.
Example for Loki on the linux operating system and amd64 architecture:

$ curl -O -L "https://github.com/grafana/loki/releases/download/v1.1.0/loki-linux-amd64.zip"
# extract the binary
$ unzip "loki-linux-amd64.zip"
# make sure it is executable
$ chmod a+x "loki-linux-amd64"

v1.0.0

🎉 Nearly a year since Loki was announced at KubeCon in Seattle 2018 we are very excited to announce the 1.0.0 release of Loki! 🎉

A lot has happened since the announcement, the project just recently passed 1000 commits by 138 contributors over 700+ PR's accumulating over 7700 GitHub stars!

Internally at Grafana Labs we have been using Loki to monitor all of our infrastructure and ingest around 1.5TB/10 billion log lines a day. Since the v0.2.0 release we have found Loki to be reliable and stable in our environments.

We are comfortable with the state of the project in our production environments and think it's time to promote Loki to a non-beta release to communicate to everyone that they should feel comfortable using Loki in their production environments too.
API Stability

With the 1.0.0 release our intent is to try to follow Semver rules regarding stability with some aspects of Loki, focusing mainly on the operating experience of Loki as an application. That is to say we are not planning any major changes to the HTTP API, and anything breaking would likely be accompanied by a major release with backwards compatibility support.

We are currently NOT planning on maintaining Go API stability with this release, if you are importing Loki as a library you should be prepared for any kind of change, including breaking, even in minor or bugfix releases.

Loki is still a young and active project and there might be some breaking config changes in non-major releases, rest assured this will be clearly communicated and backwards or overlapping compatibility will be provided if possible.

The full list of changes for this release can be found in the CHANGELOG

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Docker container:

• https://hub.docker.com/r/grafana/loki
• https://hub.docker.com/r/grafana/promtail

$ docker pull "grafana/loki:v1.0.0"
$ docker pull "grafana/promtail:v1.0.0"

Binary:

Choose from the assets below for the application and architecture matching your system.

Example for Loki on the linux operating system and amd64 architecture:

# curl -L "https://github.com/grafana/loki/releases/download/v1.0.0/loki-linux-amd64.gz

# extract the binary
$ gunzip "loki-linux-amd64.gz"

# make sure it is executable
$ chmod a+x "loki-linux-amd64"

v0.4.0

This is release v0.4.0 of Loki.

EDIT 2019/10/25:

PR 948 introduces more limit configurations on Loki and you may find the defaults too low for your installation resulting in rate limiting on the clients writing to Loki which was not happening prior to v0.4.0. The limits can be configured higher to avoid this, more info on limits can be found here.

Limits are mostly useful for multi-tenant installations, however they can be useful in single tenant and even single binary installations as a mechanism to help keep Loki's resources constrained. We are discussing if the current defaults are too low and/or if they should be removed entirely from the Helm installation.

Relevant issues #1200 and #1199

Notable changes:

Please see the CHANGELOG for full release notes.

• With PR 654 @cyriltovena added a really exciting new capability to Loki, a Prometheus compatible API with support for running metric style queries against your logs! Take a look at how to write metric queries for logs

  PLEASE NOTE: To use metric style queries in the current Grafana release 6.4.x you will need to add Loki as a Prometheus datasource in addition to having it as a Log datasource and you will have to select the correct source for querying logs vs metrics, coming soon Grafana will support both logs and metric queries directly to the Loki datasource!

• PR 1022 (and a few others) @joe-elliott added a new set of HTTP endpoints in conjunction with the work @cyriltovena to create a Prometheus compatible API as well as improve how labels/timestamps are handled

  IMPORTANT: The new /api/v1/* endpoints contain breaking changes on the query paths (push path is unchanged) Eventually the /api/prom/* endpoints will be removed

• PR 847 owes a big thanks to @cosmo0920 for contributing his Fluent Bit go plugin, now loki has Fluent Bit plugin support!!

• PR 982 was a couple weeks of painstaking work by @rfratto for a much needed improvement to Loki's docs! Check them out!

• PR 980 by @sh0rez improved how flags and config file's are loaded to honor a more traditional order of precedence:

  1. Defaults
  2. Config file
  3. User-supplied flag values (command line arguments)

  PLEASE NOTE: This is potentially a breaking change if you were passing command line arguments that also existed in a config file in which case the order they are given priority now has changed!

• PR 1062 and 1089 have moved Loki from Dep to Go Modules and to Go 1.13

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Binary:

# download a binary (adapt app, os and arch as needed)
$ curl -fSL -o "/usr/local/bin/loki.gz" "https://github.com/grafana/loki/releases/download/v0.4.0/loki-linux-amd64.gz"
$ gunzip "/usr/local/bin/loki.gz"

# make sure it is executable
$ chmod a+x "/usr/local/bin/loki"

Docker container:

• https://hub.docker.com/r/grafana/loki
• https://hub.docker.com/r/grafana/promtail

$ docker pull "grafana/loki:v0.4.0"
$ docker pull "grafana/promtail:v0.4.0"

v0.3.0

This is release v0.3.0 of Loki.

Notable changes:

Please see the CHANGELOG for full release notes.

Loki sees readiness probes added for the queriers and distributors (for running Loki in microservices mode), some cleanup and improvement to the live tailing code, fix panic when ingesters are removed while being live tailed, and a couple bugs around chunk handling.

The promtail amd64 image now supports reading Systemd journal files. This has been a struggle as their only exists C libraries for reading the journal file which requires using CGO and an image with systemd packages. For now we have switched to a debian image for promtail which has the necessary systemd files. Currently however cross compiling for ARM is too much of a struggle and we have deferred this for when we switch to the native image building with the drone.io CI system, at which point we will have both amd64 and arm promtail images which can read the Systemd journal.

As just mentioned there has been a big effort to offload building images from circle ci to drone.io where we have access to native arm containers for building arm images without cross compiling. Drone also seems to be building much faster. Currently we are building in both CI systems but soon will switch the image building to drone.

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Binary:

# download a binary (adapt app, os and arch as needed)
$ curl -fSL -o "/usr/local/bin/loki.gz" "https://github.com/grafana/loki/releases/download/v0.3.0/loki_linux_amd64.gz"
$ gunzip "/usr/local/bin/loki.gz"

# make sure it is executable
$ chmod a+x "/usr/local/bin/loki"

Docker container:

• https://hub.docker.com/r/grafana/loki
• https://hub.docker.com/r/grafana/promtail

$ docker pull "grafana/loki:v0.3.0"
$ docker pull "grafana/promtail:v0.3.0"

v0.2.0

This is release v0.2.0 of Loki.

Notable changes:

There were over 100 PR's merged since 0.1.0 was released. Please see the CHANGELOG for a more complete list.

A few of the most notable improvements:

There were some significant improvements in memory handling and performance of Loki:

• Loki: 713 Storage memory improvement.
• Loki: 782 Performance improvement: Query storage by iterating through chunks in batches.
• Loki: 788 Querier timeouts.
• Loki: 794 Support ingester chunk transfer on shutdown.

Additionally, labels are now fetched from storage for the requested timeframe so you are no longer limited to only retrieving labels of what was still in memory:

• Loki: 521 Query label values and names are now fetched from the store.

Please note this introduces a potential new issue if you're asking for labels across a large number of streams over a large timeframe, which causes a large amount of chunks to be loaded. A fix for this is currently being worked out.

Several additions were made to the log processing pipeline:

• Pipeline: 738 Added a template stage for manipulating label values.
• Pipeline: 732 Support for Unix timestamps.
• Pipeline: 760 Support timestamps without year.

A docker logging plugin was created to allow docker containers to send logs directly to Loki:

• Docker-Plugin: 663 Created a Docker logging driver plugin.

And starting with this release there are now multi-architecture docker containers and binaries:

• Build: 668,762 Build multiple architecture containers.

Installation:

The components of Loki are currently distributed in plain binary form and as Docker container images. Choose what fits your use-case best.

Binary:

# download a binary (adapt app, os and arch as needed)
$ curl -fSL -o "/usr/local/bin/loki.gz" "https://github.com/grafana/loki/releases/download/v0.2.0/loki_linux_amd64.gz"
$ gunzip "/usr/local/bin/loki.gz"

# make sure it is executable
$ chmod a+x "/usr/local/bin/loki"

Docker container:

• https://hub.docker.com/r/grafana/loki
• https://hub.docker.com/r/grafana/promtail

$ docker pull "grafana/loki:v0.2.0"
$ docker pull "grafana/promtail:v0.2.0"

v0.1.0

Loki Beta Release 1 🎉

Note: If you had been using the pipeline_stages in promtail which was introduced about a week ago, there were some breaking changes to this, please see here for information on the new config.

Getting Started