NS-CACHE: Eviction from the cache

[guptsanj]

Explain the changes

Cached chunks are evicted when a namespace bucket with cache starts to fill up.
The tier ttf worker will periodically perform the eviction.
When bucket reaches a predefined limit, entries are no longer cached
Any in transit objects/chunks are also dropped.

Issues: Fixed #xxx / Gap #xxx

Testing Instructions:

[guymguym]

Still reviewing, but sending out my comments so far.

[guymguym]

Another note was about the call to usage_aggregator.get_bandwidth_report() in the worker here:

noobaa-core/src/server/bg_services/tier_ttf_worker.js

Lines 74 to 79 in 0f73225

	const reports = await usage_aggregator.get_bandwidth_report({
	bucket: bucket._id,
	since: now - (1000 * 60 * 60),
	till: now,
	time_range: 'hour'
	});

@nb-ohad suggested to use lower time_range and sum up the results.

[guymguym]

@guptsanj See changes requested. Please let me know if I can help.

Comments are not relevant in updated PR

[guymguym]

@jackyalbo Please review and change your review to approved if you have no additional comments - we are trying to complete this PR this week. Thanks

[guymguym]

Hi @guptsanj Good job, this is almost done.
I found a couple more things - see below.
Also what about usage_aggregator.get_bandwidth_report() - didn't you need to change it?

[guptsanj]

Hi @guymguym

Re: Also what about usage_aggregator.get_bandwidth_report() - didn't you need to change it?
I will start looking at this today.

[guptsanj]

@guymguym Should we not do this cached object aging check at the start of the map builder for the eviction case? This check is just marking object entries as deleted when they are aged out AND objects have no parts.
The chunks, parts and blocks related to eviction have already been deleted by this point.

[guymguym]

Good question - here are my thoughts:

For chunks (with their parts and blocks) we trigger eviction based on capacity and evict chunks from the cache based on LRU. FIFO is even easier to implement just by not moving chunks to the front of the LRU on hit - and FIFO is age based. So if we will prefer simple age based vs access based caching, we can do that - actually we should just add this for dev/test.

For objects_md cleanup - keep in mind that even if we are limited on cache capacity and have to evict, it is still very much likely that cached object_mds have not been deleted or overwritten in the hub, so keeping them is not a bad thing at all. But we do need to evict eventually if the workload will read too many of these to the cache than we can store in the DB. Today we do not have a capacity limit for object_mds and we do not order them in LRU. However we can easily iterate object_mds in FIFO order by traversing the _id index in order or creation in the cache and then check if this object has no parts and then delete.

Ultimately I think object_md cleanup calls for a separate worker - because the iteration subject and order are completely different than tier_ttf_worker. In its own worker we should check how close we are to the object_mds limit we can store in the DB (either config or calculate based on PV), and cleanup in FIFO order.

For now I thought we can piggy back the chunks eviction, BUT adding the age condition I suggested is presenting an issue - if we evict all chunks/parts of an object but it is too young and we filter it out from objects_to_gc, we may never come back to evict more chunks for this same object_md and it will never be evicted. So I would remove the filter for now, but in any case to make it work well we need to move this to its own worker.

[guymguym]

@guptsanj Great work. See my comment above - I think removing the age filter for objects_to_gc is better, but in any case this is not a real issue and we can go ahead with this PR and improve object_md eviction in next PR.