[bug] loki bloom builder and gateway don't perform TTL eviction of bloomshipper data #15865
Closed
Description
Describe the bug
Bloom builder and bloom gateway don't perform TTL block eviction every minute. In production this causes bloom builder to exhaust the ephemeral disk space limits of 15 GB every hour.
To Reproduce
Run bloom builder with the loki helm chart
Expected behavior
Loki by code should periodically evict bloom shipper cache entries every minute
func (c *BlocksCache) runTTLEvictJob(interval, ttl time.Duration) {
if interval == 0 || ttl == 0 {
return
}
level.Info(c.logger).Log("msg", "run ttl evict job")
ticker := time.NewTicker(interval)
defer ticker.Stop()
for {
select {
case <-c.done:
return
case <-ticker.C:
c.evictExpiredItems(ttl)
}
}
}
Environment:
- EKS Kubernetes
- Loki helm chart version
6.23.0
Screenshots, Promtail config, or terminal output
Loki version
caller=main.go:126 msg="Starting Loki" version="(version=release-3.3.x-60f2af3, branch=release-3.3.x, revision=60f2af32)"
Only once upon a startup TTL eviction job is performed which should be happening every minute.