Replicated cache deletion behavior when the 3rd node joins the cluster.

[rezigned]

Problem

Let's say we have 3 nodes A, B and C with the following sequence of events:

1. A starts and write a cache Cache.put(:a, 1)
2. B joins the cluster. It can still see the cache Cache.all() #=> [:a]
3. C joins the cluster. All cache are gone (Cache.all() #=> []) in all nodes.

From what I understand (after reading through the code).

The current design of generational cache has 2 generations at most. e.g. [new, old]. Each time when a new node starts (or joins the cluster) it will always create a new generation first before copying the data (as seen in the code below)

nebulex/lib/nebulex/adapters/replicated.ex

Lines 767 to 770 in b20cd77

	with :ok <- maybe_run_on_nodes(adapter_meta, nodes, :new_generation),
	:ok <- copy_entries_from_nodes(adapter_meta, nodes),
	:ok <- maybe_run_on_nodes(adapter_meta, [node()], :new_generation) do
	maybe_run_on_nodes(adapter_meta, nodes, :reset_generation_timer)

This seems to cause the issue I mentioned above. When A first started, its generational cache become

[a1] # a1 is holding the cache data `:a => 1`

When B joins, A's cache become

[a2, a1] # a2 is the new one, a1 is the old

Now, when C joins, all caches are gone.

[a3, a2] # a1 is gone, including the data `:a => 1` (technically, it's in `deprecated` table)

Solution/Suggestion?

I've modified the code above to run copy_entries_from_nodes first whenever a new node joins the cluster and it seemed to fix the issue. Is this the correct way to fix this?

[cabol]

Hey 👋 !! The fix you propose seems valid. However, that makes me wonder if we really need to push a new generation. The original idea was to copy only the data from the newer generation. Hence, to make it so, a new generation is created first, the older is removed, and when the data is copied, we ensure it comes from the previous newer generation. With the approach you suggest, the issue I see is that we will be copying the data from the new and old generations, but maybe it is not necessarily an issue? Perhaps, the best way to handle this would be to avoid pushing a new generation and somehow copy the data from the new generation in the remote node, but that will require some changes. In the meantime, I think your fix seems valid to me. Like I said, it may not be a big deal to copy all the data from both generations, so we can avoid the issue you are highlighting here.

Summing up, the fix you are suggesting looks good to me in the meantime, while we work on something better, like I mentioned before. There is one thing I need to do before you can open the PR. I need to create a new branch to keep maintaining version 2, because right now the main branch is with version 3. Let me ping you when this is done, so you can open the PR against that branch.

Thanks 🙇 !!

[rezigned]

Thanks for your quick response and detailed explanation! In the mean time, while waiting for a new v2 branch, I will prepare a PR and tests for this.

Thank you 🙏

[cabol]

A new version v2.6.5 has been released with this fix!

[rezigned]

@cabol Thank you 🥳. Although it's a little odd; for some reason, I can't seem to see those changes in v2.6.5? (it seemed to be v3 if i'm not mistaken) v2.6.4...v2.6.5

[rezigned]

FYI, I just found that it shows the correct diff on Hex registry https://diff.hex.pm/diff/nebulex/2.6.4..2.6.5 (but somehow the current https://github.com/elixir-nebulex/nebulex/tree/v2.6.5 points to v3 instead)

[cabol]

Hum, ok, I'll check, Thx!

[cabol]

@rezigned I've fixed the issue. Thx!

[kung-rpt]

Thx 🎉!