feat: limit verdict cache size

[keejon]

Limits the VerdictCache size by removing unused keys (last access older than x minutes) and capping the total cache size to a percentage of the heap, preventing OOM. The cache size is estimated rather than precisely measured to avoid performance overhead.

[ivanyu]

As big numbers of transactions make the verdict cache problematic, limiting the cache size directly will make the same big streams of transaction flush out cached verdicts about other types of entities (like e.g. topics). We probably should consider separating the cache for transaction and making it bound to solve the original problem and prevent the new one.

[keejon]

As big numbers of transactions make the verdict cache problematic, limiting the cache size directly will make the same big streams of transaction flush out cached verdicts about other types of entities (like e.g. topics). We probably should consider separating the cache for transaction and making it bound to solve the original problem and prevent the new one.

Although caffeine will invalidate least frequently used keys so shouldn't the transactions be the first keys that are flushed out? They should also be the ones that are invalidated by expireAfterAccess.

[ivanyu]

That's true, but I'm afraid it's possible that new transaction will first flush out stuff from the cache, before sooner be replaced by newer transactions. I.e. we'll get cache thrashing

[keejon]

That's true, but I'm afraid it's possible that new transaction will first flush out stuff from the cache, before sooner be replaced by newer transactions. I.e. we'll get cache thrashing

Hm what we could maybe also do is use the same cache instance but have a short expire on transaction keys. like 60-120 seconds maybe?

[ivanyu]

I don't see how it could help... newly created transactions will still be removing more stable entities, regardless of their TTL

[tvainika]

Caffeine cache is using more accurate cache flushing than simple LRU, so I don't think cache trashing becomes an issue unless cache size is really small. Actually I wonder if time based expiry is even needed at all? But anyway in my opinion this can go in as is now.

[keejon]

I don't see how it could help... newly created transactions will still be removing more stable entities, regardless of their TTL

I would think that it is just less likely that this will ever happen since transactions most likely take up most space but are expired and evicted at a faster and more appropriate rate. This means it is very unlikely that it will ever hit cache size limit and thus we don't have to worry as much about it. IMHO if we have such high rate of transactions that low expire of ~120 seconds is not enough the cluster will probably already be in trouble.

Caffeine cache is using more accurate cache flushing than simple LRU, so I don't think cache trashing becomes an issue unless cache size is really small. Actually I wonder if time based expiry is even needed at all? But anyway in my opinion this can go in as is now.

Instead of getting rid of expiry i would actually rather just do TTL and get rid of cache size limit for the reasons above :-)

[tvainika]

LGTM