JedisClusterInfoCache: refresh the nodes map

[stillerrr]

Expected behavior

Jedis would refresh the elements of nodes map when the cluster nodes changed

Actual behavior

this nodes parameter just add nodes if not exist in map, but not delete nodes when cluster nodes are deleted

Steps to reproduce:

it would cause a problem in this situation:

1. create a redis cluster A with 10 master nodes and delete 9 nodes from cluster;
2. use the 9 nodes create a new cluster B
3. delete some slots from the cluster or give a large latency between client and the cluster A， make sure jedis throw JedisConnectionException;

JedisConnectionException would cause a slot refresh in JedisClusterCommand:

this method would use JedisClusterInfoCache paramter: nodes map values to refresh slot in JedisClusterInfoCache.renewClusterSlots():

There is a high probability(90%) that the jedis client will connect to cluster B because jedis use the cluster B nodes to refresh slot

Jedis version:

3.6.0

[sazzad16]

@StillerRen In such massive reconfiguration of Redis Cluster, you should recreate JedisCluster with proper nodes.

[stillerrr]

Node adding and deleting is normal operation for cluster and client don't perceive the change.
I think jedis should determine cluster nodes based on the cluster nodes/slots command.

[sazzad16]

Jedis is supporting node adding and deleting well enough.
Only issue it doesn't handle (in JedisCluster A) yet is when a node is moved from Redis Cluster A to Redis Cluster B.

[stillerrr]

This is a common operation, especially when resources are reused.
but this would cause the client connect to a wrong cluster and hard to find out reason

[brunogbv]

Bumping this old issue, sorry in advance.

We just had an incident in production related with this issue where, during Elasticache maintenance window, AWS reused some IP's from a node in cluster A to another one in cluster B.

It was tricky to understand what was going on since it was a silent error - data inconsistency and unexpected behaviour without clear indication of where the problem was. We're still exploring our possible solutions, but without fixing this issue I don't think Jedis is safe for production IMHO. Especially in cloud environments where the IP's are not on your control.

Let me know your thoughts and thanks in advance.

Also huge thanks to @ivkalita for finding this tricky issue

[yangbodong22011]

Bumping this old issue, sorry in advance.

We just had an incident in production related with this issue where, during Elasticache maintenance window, AWS reused some IP's from a node in cluster A to another one in cluster B.

It was tricky to understand what was going on since it was a silent error - data inconsistency and unexpected behaviour without clear indication of where the problem was. We're still exploring our possible solutions, but without fixing this issue I don't think Jedis is safe for production IMHO. Especially in cloud environments where the IP's are not on your control.

Let me know your thoughts and thanks in advance.

Also huge thanks to @ivkalita for finding this tricky issue

Your issue may be similar to this: #3595