Ingester memory leak version 2.4.1

[timansky]

noticed memory leak with version 2.4.1

commo-loki-i4 - v2.4.1
commo-loki-i[1:3;5] - v 2.4.0

---
auth_enabled: true

common:
  storage:
    s3:
      bucketnames: loki-00,loki-01,loki-02,loki-03,loki-04,loki-05,loki-06,loki-07,loki-08,loki-09
      endpoint: ceph:7480
      region: US
      access_key_id:xxx
      secret_access_key: xxx
      insecure: true
      sse_encryption: false
      http_config:
        idle_conn_timeout: 90s
        response_header_timeout: 0s
        insecure_skip_verify: true
      s3forcepathstyle: true
  ring:
    kvstore:
      store: etcd
      etcd:
        endpoints:
          - common-loki-d1:2379
          - common-loki-d2:2379
          - common-loki-d3:2379
    heartbeat_timeout: 1m
    heartbeat_period: 15s
  replication_factor: 2

schema_config:
  configs:
    - from: 2020-12-01
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: loki_index_
        period: 24h
      chunks:
        prefix: loki_chunk_
        period: 24h

storage_config:
  boltdb_shipper:
    shared_store: s3
    active_index_directory: /var/lib/loki/active # Directory where ingesters would write boltdb files which would then be uploaded by shipper to configured storage
    cache_location: /var/lib/loki/cache          # Cache location for restoring boltDB files for queries
    cache_ttl: 168h                              # TTL for boltDB files restored in cache for queries
    resync_interval: 5m                          # Resync downloaded files with the storage
    query_ready_num_days: 1                      # Number of days of index to be kept downloaded for queries
    index_gateway_client:
      server_address: loki-index-gateway.monitoring:9095

  index_cache_validity: 2h                       # Cache validity for active index entries (Must be lower than ingester.chunk_idle_period)
  max_chunk_batch_size: 100

  index_queries_cache_config:
    enable_fifocache: false
    memcached:
      expiration: 1h
      batch_size: 256
      parallelism: 32
    memcached_client:
      addresses: loki-cache.monitoring:11213
      consistent_hash: true
      timeout: 1s
      update_interval: 5s

chunk_store_config:
  cache_lookups_older_than: 1d
  chunk_cache_config:
    enable_fifocache: false
    memcached:
      expiration: 1h
      batch_size: 256
      parallelism: 32
    memcached_client:
      addresses: loki-cache.monitoring:11211
      consistent_hash: true
      timeout: 1s
      update_interval: 5s

server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  grpc_server_max_recv_msg_size: 104857600 # 1024 * 1024 * 100
  grpc_server_max_send_msg_size: 104857600 # 1024 * 1024 * 100

ingester:
  chunk_block_size: 262144           # When this threshold is exceeded the head block will be cut and compressed inside the chunk
  chunk_target_size: 1572864         # Loki will attempt to build chunks up to 2MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  max_chunk_age: 3h                  # (1h) The maximum duration of a timeseries chunk in memory. If a timeseries runs for longer than this the current chunk will be flushed to the store and a new chunk created.
  max_transfer_retries: 0            # Disable chunk transfer which is not possible with statefulsets
  chunk_idle_period: 2h              # (30m) How long chunks should sit in-memory with no updates before being flushed if they don't hit the max block size
  chunk_retain_period: 5m            # How long chunks should be retained in-memory after they've been flushed.
  chunk_encoding: snappy             #
  sync_period: 1m
  sync_min_utilization: 0.7
  lifecycler:
    final_sleep: 10s                   # Duration to sleep before exiting to ensure metrics are scraped
    min_ready_duration: 10s
  query_store_max_look_back_period: 0 # For S3 this value must always be left as 0
  autoforget_unhealthy: false
  wal:
    enabled: true
    dir: /var/lib/loki/wal
    flush_on_shutdown: true
    replay_memory_ceiling: 2GB # Maximum memory size the WAL may use during replay. After hitting this it will flush data to storage before continuing.

limits_config:
  ingestion_rate_strategy: global
  reject_old_samples: true
  reject_old_samples_max_age: 96h
  ingestion_rate_mb: 70
  ingestion_burst_size_mb: 100
  max_query_length: 720h
  max_streams_per_user: 0 # Disabled in favor of the global limit
  max_global_streams_per_user: 100000  # 100k
  max_cache_freshness_per_query: 5m # (1m) Most recent allowed cacheable result per-tenant, to prevent caching very recent results that might still be in flux.
  enforce_metric_name: false
  max_query_series: 10000
  max_query_lookback: 720h
  unordered_writes: true

[DaveOHenry]

Memory usage on the ingester pods in our small test environment has increased after updating from 2.3 to 2.4.0, too. Maybe it has to do with the WAL?

Note to myself: see "In memory FIFO caches enabled by default" in https://grafana.com/docs/loki/latest/upgrading/#240

[timansky]

after upgrade from 2.3 to 2.4.0 there is no memory leak, noticed only with 2.4.1

rolled back to 2.4.0 and 12h normal flight

wal was enabled before

[lackhoa]

@timansky Did you try limiting the in-memory caches?

[timansky]

found from instance:3100/config that param chunk_retain_period in version 2.4.1 is not applied.
from 77e3414 it is now setting from index_cache_validity + 1m

[timansky]

index_cache_validity decreasing this param fixed issue

[JasonWangA]

I have get the same issue that the memory increased slowyly, is there any relation with it enable index cache as the boltdbshipper mode enabled ?

[timansky]

in my case decreasing index_cache_validity (decreased to 5m for us this is optimal value for mem utilization) fixed issue

[wuzhihui1123]

I have the same problem，the chunk in memory keeping growing up . Even if I manually execute chunk flush, it doesn’t work. （curl -X POST http://xxx:3100/flush ）

loki version : 2.4.1
loki config

auth_enabled: false

server:
  http_listen_port: 3100
  graceful_shutdown_timeout: 30s
  http_server_read_timeout: 60s
  http_server_write_timeout: 60s

distributor:
  ring:
    kvstore:
      store: memberlist

ingester:
  lifecycler:
    ring:
      kvstore:
        store: memberlist
      replication_factor: 1
    final_sleep: 0s
  chunk_idle_period: 5m
  chunk_retain_period: 30s
  chunk_block_size: 1048576

memberlist:
  abort_if_cluster_join_fails: false

  bind_port: 7946

  join_members:
    - loki.kube-public:7946

  max_join_retries: 10

schema_config:
  configs:
    - from: 2021-12-01
      store: boltdb-shipper
      object_store: s3
      schema: v11
      index:
        prefix: index_
        period: 24h
      chunks:
        prefix: chunk_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    shared_store: s3
  aws:
    s3: s3://<hidden>/loki
    s3forcepathstyle: true

limits_config:
  unordered_writes: true
  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 48h

chunk_store_config:
  cache_lookups_older_than: 2h

table_manager:
  retention_deletes_enabled: true
  retention_period: 168h
  poll_interval: 24h

compactor:
  working_directory: /data/compactor
  shared_store: s3
  compaction_interval: 5m

[wuzhihui1123]

in my case decreasing index_cache_validity (decreased to 5m for us this is optimal value for mem utiliz

my index_cache_validity values is 5m0s @timansky

[varac]

Same here:

I looked into the index_cache_validity setting but it was already set to 5m. So we'll try with a downgrade to 2.4.0 now and wil report back after a while if this solved the issue for us.

[varac]

Unfortunatly, after downgrading to 2.4.0 the loki pod is restarting quite often (~twice per hour) because of panic: runtime error: invalid memory address or nil pointer dereference (see #4707). So we'll upgrade again to 2.4.1 and kill the pod once a night to prevent this mem leak. Not great, but as a workaround until this gets fixed properly. Or is there anything else we could try ?

[wuzhihui1123]

after downgrading to 2.4.0 the loki pod always oom

[slim-bean]

I believe the initial issue as reported @timansky was resolved by reducing the index_cache_validity to 5m (the default value) and based on that I'd like to close this issue.

I suspect if others are seeing increased memory usage with 2.4.x it's because of the FIFO cache's enabled by default which have a default size of 1GB, thus it's likely to see an increase of 1GB of memory over any previous value after updating (assuming you did not enable the fifocache before)

To avoid confusion I think if anyone is still seeing memory issues to please open a new issue!