Cardinality limit exceeded; 13 unique label values

[gcotone]

Describe the bug
We're getting the following error while querying for series older than X amount of time, possibly over midnight, in spite of having a very limited number of (unique) labels:
Error doing request: Error response from server: cardinality limit exceeded for {}; 141435 entries, more than limit of 100000

To Reproduce
Steps to reproduce the behavior:

1. Started Loki 2.0.0
2. Started Promtail 2.0.0-amd64
3. Query: {job="fw-log"}

Expected behavior
If the number of unique labels remain constant over time, series should be returned

Environment:

• Infrastructure: kubernetes
• Deployment tool: N/A

Screenshots, Promtail config, or terminal output

Grafana query:

{
  "request": {
    "url": "api/datasources/proxy/176/loki/api/v1/query_range?direction=BACKWARD&limit=1860&query=%7Bjob%3D%22fw-log%22%7D&start=1604275059000000000&end=1604278660000000000&step=2",
    "hideFromInspector": false
  },
  "response": {
    "error": "",
    "response": "cardinality limit exceeded for {}; 141435 entries, more than limit of 100000\n",
    "message": "cardinality limit exceeded for {}; 141435 entries, more than limit of 100000\n"
  }
}

logcli

LALALA#logcli series '{job="fw-log"}' --analyze-labels --since=6h                                                                                                                      [0] 20-11-02 9:41:11
https://localhost/loki/api/v1/series?end=1604306474033385197&match=%7Bjob%3D%22fw-log%22%7D&start=1604284874033385197
Total Streams:  10
Unique Labels:  5

Label Name   Unique Values  Found In Streams
lvl          5              10
application  4              10
host         2              10
facility     1              10
job          1              10
LALALA#logcli series '{job="fw-log"}' --analyze-labels --since=7h                                                                                                                      [0] 20-11-02 9:41:14
https://localhost/loki/api/v1/series?end=1604306477218995589&match=%7Bjob%3D%22fw-log%22%7D&start=1604281277218995589
Total Streams:  11
Unique Labels:  5

Label Name   Unique Values  Found In Streams
lvl          5              11
application  4              11
host         2              11
facility     1              11
job          1              11
LALALA#logcli series '{job="fw-log"}' --analyze-labels --since=8h                                                                                                                      [0] 20-11-02 9:41:17
https://localhost/loki/api/v1/series?end=1604306480489553537&match=%7Bjob%3D%22fw-log%22%7D&start=1604277680489553537
Total Streams:  11
Unique Labels:  5

Label Name   Unique Values  Found In Streams
lvl          5              11
application  4              11
host         2              11
facility     1              11
job          1              11
LALALA#logcli series '{job="fw-log"}' --analyze-labels --since=9h                                                                                                                      [0] 20-11-02 9:41:21
https://localhost/loki/api/v1/series?end=1604306485172362222&match=%7Bjob%3D%22fw-log%22%7D&start=1604274085172362222
Error doing request: Error response from server: cardinality limit exceeded for {}; 141435 entries, more than limit of 100000
 (<nil>)


LALALA#logcli series '{job="fw-log"}' --analyze-labels --since=9h                                                                                                                            [0] 20-11-02 10:00:38
https://localhost/loki/api/v1/series?end=1604307698329025099&match=%7Bjob%3D%22fw-log%22%7D&start=1604275298329025099
Total Streams:  11
Unique Labels:  5

Label Name   Unique Values  Found In Streams
lvl          5              11
application  4              11
host         2              11
facility     1              11
job          1              11
LALALA#logcli series '{job="fw-log"}' --analyze-labels --since=10h                                                                                                                           [0] 20-11-02 10:01:39
https://localhost/loki/api/v1/series?end=1604307701929571537&match=%7Bjob%3D%22fw-log%22%7D&start=1604271701929571537
Error doing request: Error response from server: cardinality limit exceeded for {}; 141435 entries, more than limit of 100000
 (<nil>)
LALALA#  

[cyriltovena]

Most likely a vendor update causes this. Adding to 2.1.

[slim-bean]

@gcotone can you run the analyze labels query over all series?

logcli series '{}' --analyze-labels --since=6h

Also can you include your Loki config?

Thanks!

[gcotone]

@slim-bean now I get the error for all queries:

#logcli series '{}' --analyze-labels --since=6h                                                                                                                                       [0] 20-11-02 14:23:55
https://localhost/loki/api/v1/series?end=1604323442475853687&match=%7B%7D&start=1604301842475853687
Error doing request: Error response from server: cardinality limit exceeded for {}; 101515 entries, more than limit of 100000
 (<nil>)
#logcli series '{}' --analyze-labels --since=1m                                                                                                                                       [1] 20-11-02 14:24:03
https://localhost/loki/api/v1/series?end=1604323447135243178&match=%7B%7D&start=1604323387135243178
Error doing request: Error response from server: cardinality limit exceeded for {}; 101515 entries, more than limit of 100000
 (<nil>)
#logcli series '{job="fw-log"}' --analyze-labels --since=1m                                                                                                                         [130] 20-11-02 14:26:09
https://localhost/loki/api/v1/series?end=1604323576661875929&match=%7Bjob%3D%22fw-log%22%7D&start=1604323516661875929
Error doing request: Error response from server: cardinality limit exceeded for {}; 101515 entries, more than limit of 100000
 (<nil>)
#logcli series '{job="fw-log"}' --analyze-labels --since=6h                                                                                                                           [1] 20-11-02 14:26:17
https://localhost/loki/api/v1/series?end=1604323581387056050&match=%7Bjob%3D%22fw-log%22%7D&start=1604301981387056050
Error doing request: Error response from server: cardinality limit exceeded for {}; 101515 entries, more than limit of 100000
 (<nil>)

Here's my config:

kind: ConfigMap
metadata:
  name: loki-config
  namespace: default
apiVersion: v1
data:
  loki.yaml: |-
    # Disable multi-tenancy
    auth_enabled: false

    # Storage config
    storage_config:
      aws:
        s3: s3://eu-central-1/app-loki-s3
        dynamodb:
          dynamodb_url: dynamodb://eu-central-1
      boltdb_shipper:
        active_index_directory: /loki/index
        cache_location: /loki/boltdb-cache
        cache_ttl: 4h         # Can be increased for faster performance over longer query periods, uses more disk space
        shared_store: s3
    
    # Schema Config
    schema_config:
      configs:
      - from: 2020-05-15
        store: aws
        object_store: s3
        schema: v11
        index:
          prefix: loki_index_
          period: 24h
          tags:
            application: app
            component: loki
      - from: 2020-10-29
        store: boltdb-shipper
        object_store: s3
        schema: v11
        index:
          prefix: loki_index_
          period: 24h
    # The module to run Loki with. Supported values
    # all, distributor, ingester, querier, query-frontend, table-manager.
    server:
      http_listen_port: 8080
      grpc_listen_port: 9095
      graceful_shutdown_timeout: 5s
      grpc_server_max_recv_msg_size: 67108864
      http_server_idle_timeout: 120s
    
    # Configures how the lifecycle of the ingester will operate
    # and where it will register for discovery
    ingester:
      lifecycler:
        #address: 0.0.0.0
        ring:
          kvstore:
            store: memberlist
          replication_factor: 2
        final_sleep: 0s
      chunk_idle_period: 5m
      chunk_retain_period: 30s

    # Table Manager configuration
    table_manager:
      retention_period: 48h
      retention_deletes_enabled: true
      index_tables_provisioning:
        enable_ondemand_throughput_mode: true

    chunk_store_config:
      max_look_back_period: 0

    limits_config:
      enforce_metric_name: false
      reject_old_samples: true
      reject_old_samples_max_age: 168h
    
    memberlist:
      abort_if_cluster_join_fails: false

      # Expose this port on all distributor, ingester
      # and querier replicas.
      bind_port: 7946

      # You can use a headless k8s service for all distributor,
      # ingester and querier components.
      join_members:
      - loki-gossip-ring.default.svc.cluster.local:7946

      max_join_backoff: 1m
      max_join_retries: 10
      min_join_backoff: 1s

    compactor:
      working_directory: /loki/compactor
      shared_store: aws

[slim-bean]

You can increase this limit to at least allow you to run these queries with cardinality_limit in the limits_config section

[gcotone]

After bumping the limit to 500000, it now returns without error. However, it's not clear to me why cardinality increases if the set of label/values remain constant over time.
Here are the values for all series in the last 24-48hs:

#logcli series '{}' --analyze-labels --since=24h                                                                                                                                      [0] 20-11-02 17:22:06
https://localhost/loki/api/v1/series?end=1604334127937644445&match=%7B%7D&start=1604247727937644445
Total Streams:  11
Unique Labels:  5

Label Name   Unique Values  Found In Streams
lvl          5              11
application  4              11
host         2              11
facility     1              11
job          1              11
#logcli series '{}' --analyze-labels --since=48h                                                                                                                                      [0] 20-11-02 17:22:10
https://localhost/loki/api/v1/series?end=1604334405699267409&match=%7B%7D&start=1604161605699267409
Total Streams:  11
Unique Labels:  5

Label Name   Unique Values  Found In Streams
lvl          5              11
application  4              11
host         2              11
facility     1              11
job          1              11
#                                               

[sandeepsukhani]

The limit here is related to number of index entries a query fetches from index store, considering the number of labels and streams it looks strange that the number for you is so high.
If you are scraping loki metrics then can you please share the graph for last 24h for following metric query:
sum(rate(loki_ingester_chunks_flushed_total[1m]))

[gcotone]

@sandeepsukhani I only have values for 8AM - 8PM time range, and this is how it looks for sum(rate(loki_ingester_chunks_flushed_total[1m]))

[glebsam]

Looks like I step into the same issue.
Below is my graph for sum(rate(loki_ingester_chunks_flushed_total[1m])).
It seems troubles starts feeling after querying with specific amount of time back (with precision to seconds).

❯ while true; do logcli --addr http://127.0.0.1:3100 series '{ecs_cluster="XXX", ecs_container_name="YYY"}' --stats --since 10h1m10s --analyze-labels; sleep 1; done
http://127.0.0.1:3100/loki/api/v1/series?end=1607940065214758000&match=%7Becs_cluster%3D%22XXX%22%2C+ecs_container_name%3D%22YYY%22%7D&start=1607903995214758000
Error doing request: Error response from server: cardinality limit exceeded for {}; 131253 entries, more than limit of 100000
 (<nil>)
http://127.0.0.1:3100/loki/api/v1/series?end=1607940070964981000&match=%7Becs_cluster%3D%22XXX%22%2C+ecs_container_name%3D%22YYY%22%7D&start=1607904000964981000
Total Streams:  1
Unique Labels:  8

Label Name                   Unique Values  Found In Streams
ecs_task_definition_version  1              1
host                         1              1
image_id                     1              1
image_name                   1              1
source                       1              1
ecs_cluster                  1              1
ecs_container_name           1              1
ecs_task_definition_family   1              1

Increasing limit to 135000 did not helped, and error message now seems incorrect (or limit did not applied).

131253 entries, more than limit of 100000

Increasing limit to 500000 did not helped either, message is the same. The limit was set in loki config:

limits_config:
  cardinality_limit: 500000

[slim-bean]

@glebsam huh, thanks for reporting, and all the info you provided is great.

This is very peculiar, we haven't been able to reproduce this and are still not sure what's happening here.

Can you describe your deployment a little more, i see the two labels you list, but how many clusters and containers do you have?

Are you using boltdb-shipper? and if your log data isn't sensitive would you be willing to send us some files so we could try to run this locally?

@gcotone same question for you, would you be able to send us some files so we can try to recreate this locally?

[glebsam]

@slim-bean it is about 84 containers (26 hosts) sending its logs to Loki. Senders are Loki logging drivers. Loki deployed in monolithic mode in docker. Index storage is AWS Dynamodb, chunks storage is AWS S3. Loki server instance has 4 Gb RAM, 3200 Mb of which is available to Loki container. Unfortunately, it's impossible to send log files itself, but you can ask me questions and I will do my best answering.

Also, I am sorry, in my previous message I made a mistake trying to increase cardinality (I touched a wrong config). Now with cardinality increased in proper way (500k), it allows me to perform the series command for up to last 24h (previously it was 10h) without any RAM or CPU penalty:

❯ logcli --addr http://127.0.0.1:3100 series '{ecs_cluster="XXX", ecs_container_name="YYY"}' --stats --since 10h --analyze-labels
http://127.0.0.1:3100/loki/api/v1/series?end=1607949539128521000&match=%7Becs_cluster%3D%22XXX%22%2C+ecs_container_name%3D%22YYY%22%7D&start=1607913539128521000
Total Streams:  4
Unique Labels:  8

Label Name                   Unique Values  Found In Streams
source                       2              4
ecs_task_definition_version  2              4
image_id                     2              4
image_name                   2              4
ecs_task_definition_family   1              4
host                         1              4
ecs_container_name           1              4
ecs_cluster                  1              4

Also, (it may be related, may be not): a couple of days ago the load on loki application was increased dramatically (see the graph below), it caused loki restarts due to OOM (we've seen DynamoDB throttling and increased RAM consumption from Loki).

Cluster-wide cardinality for last 10 minutes:

❯ logcli --addr http://127.0.0.1:3100 series '{ecs_cluster=~".+"}' --stats --since 10m --analyze-labels
http://127.0.0.1:3100/loki/api/v1/series?end=1607947526420783000&match=%7Becs_cluster%3D~%22.%2B%22%7D&start=1607946926420783000
Total Streams:  115
Unique Labels:  8

Label Name                   Unique Values  Found In Streams
ecs_task_definition_family   84             115
ecs_container_name           66             115
ecs_task_definition_version  57             115
host                         26             115
image_name                   19             115
image_id                     18             115
ecs_cluster                  11             115
source                       2              115

Loki config:

auth_enabled: false

server:
  http_listen_port: 3100
  http_server_read_timeout: 4m
  http_server_write_timeout: 4m
  http_server_idle_timeout: 4m

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1
    final_sleep: 10s
  chunk_idle_period: 3m
  chunk_retain_period: 3m
  chunk_encoding: lz4

distributor:
    ring:
      kvstore:
        store: memberlist

querier:
  query_timeout: 5m
  engine:
    timeout: 4m

schema_config:
  configs:
  - from: 2020-09-28
    store: aws
    object_store: aws
    schema: v11
    index:
      prefix: loki-index_
      period: 7d

query_range:
  split_queries_by_interval: 24h

storage_config:
  aws:
    s3: s3://eu-central-1
    bucketnames: loki-chunks-
    dynamodb:
      dynamodb_url: dynamodb://eu-central-1

limits_config:
  ingestion_rate_strategy: local  # per-replica limits, not overall cluster
  enforce_metric_name: false  #  "metrics" are always logs, so do not need any names, only tags
  reject_old_samples: true  # reject samples older than X age
  reject_old_samples_max_age: 168h  # reject samples older than 7 days
  max_entries_limit_per_query: 20_000  # Maximum number of log entries that will be returned for a query. 0 to disable
  ingestion_rate_mb: 20  # Per-user ingestion rate limit in sample size per second
  ingestion_burst_size_mb: 30  # Per-user allowed ingestion burst size (in sample size)
  cardinality_limit: 500000

table_manager:
  retention_deletes_enabled: true
  retention_period: 35d
  index_tables_provisioning:
    provisioned_write_throughput: 150
    provisioned_read_throughput: 150
    inactive_write_throughput: 5
    inactive_read_throughput: 30