High cardinality labels

[sandstrom]

For many logging scenarios it's useful to lookup/query the log data on high cardinality labels such as request-ids, user-ids, ip-addresses, request-url and so on.

While I realize that high cardinality will affect indices, I think it's worth discussing whether this is something that Loki can support in the future.

There are a lot of logging use-cases where people can easily ditch full-text search that ELK and others provide. But not having easy lookup on log metadata, such as user-id or request-id, is a problem.

I should note that this doesn't necessarily need to be "make labels high cardinality", it could also be the introduction of some type of log-line metadata or properties, that are allowed to be high cardinality and can be queried for.

Example

Our services (nginx, application servers, etc) emit JSON lines[1] like this:

{ "timestamp" : "2018-12-24T18:00:00", "user-id" : "abc", "request-id" : "123", "message" : "Parsing new user data" }
{ "timestamp" : "2018-12-24T18:01:00", "user-id: " "def", "request-id" : "456", "message" : "Updating user record" }
{ "timestamp" : "2018-12-24T18:02:00", "user-id: " "abc", "request-id" : "789", "message" : "Keep alive ping from client" }

We ingest these into our log storage (which we would like to replace with Loki), and here are some common types of tasks that we currently do:

1. Bring up logs for a particular user. Usually to troubleshoot some bug they are experiencing. Mostly we know the rough timeframe (for example that it occurred during the past 2 weeks). Such a filter will usually bring up 5-200 entries. If there are more than a few entries we'll usually filter a bit more, on a stricter time intervall or based on other properties (type of request, etc).

2. Find the logs for a particular request, based on its request id. Again, we'd usually know the rough timeframe, say +/- a few days.

3. Looking at all requests that hit a particular endpoint, basically filtering on 2-3 log entry properties.

All of these, which I guess are fairly common for a logging system, require high cardinality labels (or some type of metadata/properties that are high cardinality and can be queried).

[1] http://jsonlines.org/

Updates

A few updates, since this issue was originally opened.

• This design proposal doesn't directly address high-cardinality labels, but solves some of the underlying problems.

  With LogQL you can grep over large amounts of data fairly easily, and find those needles. It assumes that the queries can run fast enough on large log corpora.

• There is some discussion in this thread, on making it easier to find unique IDs, such as trace-id and request-id

[sandstrom]

A thought:

An alternative to high cardinality labels could be to introduce a complement to labels called metadata, that is allowed to be high cardinality. Having two different things could allow us to impose other restrictions on the metadata type of values.

If grep is quick enough, maybe it would work to not index the metadata key/value pairs. But still allow them to be filtered (at "grep speed"). That would allow you to give these high-cardinality key-value pairs blessed UI, for easy filtering, whilst still avoiding the cost of indexing high cardinality fields.

[tomwilkie]

While I realize that high cardinality will affect indices, I think it's worth discussing whether this is something that Loki can support in the future.

The metadata we index for each streams has the same restrictions as it does for Prometheus labels: I don't want to say never, but I'm not sure if this is something we're ever likely to support.

An alternative to high cardinality labels could be to introduce a complement to labels called metadata, that is allowed to be high cardinality.
But not having easy lookup on log metadata such as user-id or request-id may pose a problem.

Does the support for regexp filtering allow you to filter by a given user-id or request-id and achieve the same end as a second sets of non-index labels?

I'd agree its cumbersome, so perhaps adding some jq style language here to make this more natural for json logs would be better?

[sandstrom]

@tomwilkie It's all about speed, really. For our current ELK-stack — which we would like to replace with Loki — here are some common tasks:

1. Bring up logs for a particular user. Usually to troubleshoot some bug they are experiencing. Mostly we know the rough timeframe (for example that it occurred during the past 2 weeks). Such a filter will usually bring up 5-200 entries. If there are more than a few entries we'll usually filter a bit more, on a stricter time intervall or based on other properties (type of request, etc).

2. Find the logs for a particular request, based on its request id. Again, we'd usually know the rough timeframe, say +/- a few days.

3. Looking at all requests that hit a particular endpoint, basically filtering on 2-3 log entry properties.

Could these be supported by regexp filtering, grep style? Perhaps, but it would depend on how quick that filtering/lookup would be.

Some stats:

• Storing 60 days of log data
• 26M log lines (with ~10 fields of metadata per line)
• 130 gb total
• 2-3 gb per day

Since our log volumes are so small, maybe it'll be easy to grep through?

---

Regarding your "jq style language" suggestion, I think that's a great idea! Even better would be UI support for key-value filtering on keys in the top-level of that json document. Usually people will have the log message as one key (for example message), and the metadata as other top-level keys.

Logging JSON lines seems to be common enough that it's worth building some UI/support around it:

• https://www.loggly.com/blog/why-json-is-the-best-application-log-format-and-how-to-switch/
• https://docs.fluentd.org/v0.12/articles/parser_json
• https://www.envoyproxy.io/docs/envoy/latest/configuration/access_log#format-dictionaries
• https://docs.docker.com/config/containers/logging/json-file/
• https://jahed.io/2018/07/05/always-log-to-json/

---

I've copied parts of this comment into the top-level issue, for clarity. Still kept here too, for context/history.

[mumoshu]

Would this make sense in regard to storing/searching traces and spans, too?

I think I saw that Grafana had recently announced the "LGTM" stack, where T stands for Trace/Tracing. My "impression" on the announcement was that, you may be going to use Loki with optional indices on, say "trace id" and "span id", to make traces stored in Loki searchable, so that it can be an alternative datastore for Zipkin or Jaeger.

I don't have a strong opinion if this should be in Loki or not. Just wanted to discuss and get the idea where we should head in relation to the LGTM stack 😃

[mumoshu]

Never mind on my previous comment.

I think we don't need to force Loki to blur its scope and break its project goals here.

We already have other options like Elasticsearch(obviously), Badger, RocksDB, BoltDB, and so on when it comes to a trace/span storage, as seen in Jaeger jaegertracing/jaeger#760, for example.

Regarding distributed logging, nothing prevents us from implementing our own solution with promtail + badger for example, or using Loki in combination with an another distributed logging solution w/ a more richer indexing.

I'd just use Loki for light-weight, short-term, cluster-local distributed logging solution. I use Prometheus with a similar purpose for metrics. Loki remains super useful even without high cardinality labels support or secondary indices. Just my two cents 😃

[candlerb]

I've made a suggestion in point (2) here with a simple way of dealing with high cardinality values.

In short: keep the high-cardinality fields out of the label set, but parse them as pseudo-labels which are regenerated on demand when reading the logs.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[sandstrom]

Stale bot: I don't think it makes sense to close this issue.

Although I understand that high cardinality labels may not be on the immediate roadmap, it's a common feature of log aggregation systems and by keeping it open there will at least be a place where this + workarounds can be discussed.

[candlerb]

An alternative to high cardinality labels could be to introduce a complement to labels called metadata, that is allowed to be high cardinality. Having two different things could allow us to impose other restrictions on the metadata type of values.

If grep is quick enough, maybe it would work to not index the metadata key/value pairs. But still allow them to be filtered (at "grep speed"). That would allow you to give these high-cardinality key-value pairs blessed UI, for easy filtering, whilst still avoiding the cost of indexing high cardinality fields.

This approach is very similar to influxdb, which has tags and fields. Influxdb tags are like loki labels: each set of unique tags defines a time series. Fields are just stored values.

Even linear searching would be "better than grep" speed, since if the query contains any labels, loki would already filter down to the relevant chunks.

Idea 1

Store arbitrary metadata alongside the line.

In order not to complicate the query language, I suggest that these stored-but-unindexed metadata fields look like labels with a special format: e.g. they start with a special symbol (e.g. _). Then you could do:

query {router="1.2.3.4", type="netflow", _src_ip=~"10\.10\.1\..*"} foo

In terms of the API, submitting records could be done the same way. Unfortunately, it would defeat the grouping of records with same label set, making the API more verbose and making it harder for loki to group together records belonging to the same chunk. Example:

{
  "streams": [
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\",_src_ip=\"10.10.1.50\"}",
      "entries": [{ "ts": "2018-12-18T08:28:06.801064-04:00", "line": "...netflow record 1..." }]
    },
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\",_src_ip=\"10.10.1.84\"}",
      "entries": [{ "ts": "2018-12-18T08:28:06.801065-04:00", "line": "...netflow record 2..." }]
    },
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\",_src_ip=\"10.10.1.42\"}",
      "entries": [{ "ts": "2018-12-18T08:28:06.801066-04:00", "line": "...netflow record 3..." }]
    }
  ]
}

So in the API it might be better to include the metadata inline in each record:

{
  "streams": [
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\"}",
      "entries": [
        { "ts": "2018-12-18T08:28:06.801064-04:00", "_src_ip": "10.10.1.50", "line": "...netflow record 1..." },
        { "ts": "2018-12-18T08:28:06.801065-04:00", "_src_ip": "10.10.1.84", "line": "...netflow record 2..." },
        { "ts": "2018-12-18T08:28:06.801066-04:00", "_src_ip": "10.10.1.42", "line": "...netflow record 3..." }
      ]
    }
  ]
}

In the grafana UI you could have one switch for displaying labels, and one for displaying metadata.

Wild idea 2

The same benefits as the above could be achieved if the "line" were itself a JSON record: stored in native JSON, and parsed and queried dynamically like jq.

You can do this today, if you just write serialized JSON into loki as a string. Note: this sits very well with the ELK/Beats way of doing things.

{
  "streams": [
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\"}",
      "entries": [
        { "ts": "2018-12-18T08:28:06.801064-04:00", "line":"{\"_src_ip\": \"10.10.1.50\", \"record\": \"...netflow record 1...\"}" },
        { "ts": "2018-12-18T08:28:06.801065-04:00", "line":"{\"_src_ip\": \"10.10.1.84\", \"record\": \"...netflow record 2...\"}" },
        { "ts": "2018-12-18T08:28:06.801066-04:00", "line":"{\"_src_ip\": \"10.10.1.42\", \"record\": \"...netflow record 3...\"}" }
      ]
    }
  ]
}

The only thing you can't do today is the server-side filtering of queries.

Therefore, you'd need to extend the query language to allow querying on fields of the JSON "line", in the main query part outside the label set. I think that using jq's language could be the ideal solution: the world does not need another query language, and it would allow querying down to arbitrary levels of nesting.

As an incremental improvement, you could then extend the API to natively accept a JSON object, handling the serialization internally:

{
  "streams": [
    {
      "labels": "{type=\"netflow\",router=\"1.2.3.4\"}",
      "entries": [
        { "ts": "2018-12-18T08:28:06.801064-04:00", "line":{"_src_ip": "10.10.1.50", "record": "...netflow record 1..."} },
        { "ts": "2018-12-18T08:28:06.801065-04:00", "line":{"_src_ip": "10.10.1.84", "record": "...netflow record 2..."} },
        { "ts": "2018-12-18T08:28:06.801066-04:00", "line":{"_src_ip": "10.10.1.42", "record": "...netflow record 3..."} }
      ]
    }
  ]
}

On reading records back out via the API, it could be optional whether loki returns you the raw line inside a string, or (for valid JSON) gives you the object. There are certainly cases where you'd want to retain the string version, e.g. for logcli.

Storing a 1-bit flag against each line, saying whether it's valid JSON or not, could be a useful optimisation.

The big benefits of this approach are (a) no changes to underlying data model; (b) you gain much more power dealing with log data which is already natively JSON; (c) you gain the ability to filter on numeric fields, with numeric comparison operators like < and > (where 2 < 10, unlike "2" > "10")

---

Aside: for querying IP address ranges, regexps aren't great. One option would be to enhance the query language with IP-address specific operators. Another one is to convert IP addresses to fixed-width hexadecimal (e.g. 10.0.0.1 = "0a000001") before storing them in metadata fields; prefix operations can be converted to regexp mechanically.

e.g. 10.0.0.0/22 => 0a000[0123].* or better 0a000[0123][0-9a-f]{2}