Skip to content

[bug] loki bloom builder and gateway don't perform TTL eviction of bloomshipper data #15865

Closed
@davidk-zx

Description

Describe the bug
Bloom builder and bloom gateway don't perform TTL block eviction every minute. In production this causes bloom builder to exhaust the ephemeral disk space limits of 15 GB every hour.

To Reproduce
Run bloom builder with the loki helm chart

Expected behavior
Loki by code should periodically evict bloom shipper cache entries every minute

func (c *BlocksCache) runTTLEvictJob(interval, ttl time.Duration) {
	if interval == 0 || ttl == 0 {
		return
	}

	level.Info(c.logger).Log("msg", "run ttl evict job")
	ticker := time.NewTicker(interval)
	defer ticker.Stop()

	for {
		select {
		case <-c.done:
			return
		case <-ticker.C:
			c.evictExpiredItems(ttl)
		}
	}
}

Environment:

  • EKS Kubernetes
  • Loki helm chart version 6.23.0

Screenshots, Promtail config, or terminal output
Loki version
caller=main.go:126 msg="Starting Loki" version="(version=release-3.3.x-60f2af3, branch=release-3.3.x, revision=60f2af32)"

Only once upon a startup TTL eviction job is performed which should be happening every minute.
Image

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions