Redis cluster connection fails when slave goes offline

[theblazehen]

Version: What redis-py and what redis version is the issue happening on?
redis-py: 4.3.4
redis: keydb 6.3.1

Platform: Python 3.8.10 on Ubuntu 20.04

Description:
I have a 6 node cluster, 3 masters and 3 slaves.
I initialize the client with

rc = redis.cluster.RedisCluster(
    host="keydb-staging-1.lxd",
    port=6379,
    decode_responses=True,
    retry=redis.retry.Retry(redis.backoff.ConstantBackoff(1), 20),
)

When I shut down one master then everything keeps running fine:

>>> rc.ping()
True

However, when I shut down one slave (All masters are online) I get the following when attempting to run a ping:

>>> rc.ping()
<class 'redis.exceptions.ConnectionError'>
Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 1393, in get_connection
    if connection.can_read():
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 809, in can_read
    return self._parser.can_read(timeout)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 411, in can_read
    return self.read_from_socket(timeout=timeout, raise_on_timeout=False)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 423, in read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 611, in connect
    sock = self.retry.call_with_retry(
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 612, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 677, in _connect
    raise err
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 665, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 1067, in _execute_command
    connection = get_connection(redis_node, *args, **kwargs)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 50, in get_connection
    return redis_node.connection or redis_node.connection_pool.get_connection(
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 1397, in get_connection
    connection.connect()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 617, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to 240.159.209.101:6380. Connection refused.
<class 'redis.exceptions.ConnectionError'>
Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 611, in connect
    sock = self.retry.call_with_retry(
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 612, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 677, in _connect
    raise err
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 665, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 1067, in _execute_command
    connection = get_connection(redis_node, *args, **kwargs)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 50, in get_connection
    return redis_node.connection or redis_node.connection_pool.get_connection(
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 1387, in get_connection
    connection.connect()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 617, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to 240.159.209.101:6380. Connection refused.
<class 'redis.exceptions.ConnectionError'>
Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 611, in connect
    sock = self.retry.call_with_retry(
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 612, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 677, in _connect
    raise err
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 665, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 1067, in _execute_command
    connection = get_connection(redis_node, *args, **kwargs)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 50, in get_connection
    return redis_node.connection or redis_node.connection_pool.get_connection(
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 1387, in get_connection
    connection.connect()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 617, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to 240.159.209.101:6380. Connection refused.
<class 'redis.exceptions.ConnectionError'>
Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 611, in connect
    sock = self.retry.call_with_retry(
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 612, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 677, in _connect
    raise err
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 665, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 1067, in _execute_command
    connection = get_connection(redis_node, *args, **kwargs)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 50, in get_connection
    return redis_node.connection or redis_node.connection_pool.get_connection(
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 1387, in get_connection
    connection.connect()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 617, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to 240.159.209.101:6380. Connection refused.
<class 'redis.exceptions.ConnectionError'>
Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 611, in connect
    sock = self.retry.call_with_retry(
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/retry.py", line 46, in call_with_retry
    return do()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 612, in <lambda>
    lambda: self._connect(), lambda error: self.disconnect(error)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 677, in _connect
    raise err
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 665, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 1067, in _execute_command
    connection = get_connection(redis_node, *args, **kwargs)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 50, in get_connection
    return redis_node.connection or redis_node.connection_pool.get_connection(
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 1387, in get_connection
    connection.connect()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/connection.py", line 617, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to 240.159.209.101:6380. Connection refused.
Exception ignored in: <function ClusterNode.__del__ at 0x7f8033910940>
Traceback (most recent call last):
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 1220, in __del__
    if self.redis_connection is not None:
AttributeError: 'ClusterNode' object has no attribute 'redis_connection'
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    rc.ping()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/commands/core.py", line 1132, in ping
    return self.execute_command("PING", **kwargs)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 1029, in execute_command
    raise e
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 1019, in execute_command
    res[node.name] = self._execute_command(node, *args, **kwargs)
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 1103, in _execute_command
    self.nodes_manager.initialize()
  File "/home/ubuntu/venv/deepalert/lib/python3.8/site-packages/redis/cluster.py", line 1486, in initialize
    raise RedisClusterException(
redis.exceptions.RedisClusterException: ERROR sending "cluster slots" command to redis server 240.159.209.101:6380. error: cannot pickle '_thread.lock' object

240.159.209.101:6380 is the slave that I shut down.

I would expect the client to continue operating against the master nodes in the absence of the slave

[theblazehen]

I got my cluster into a weird state, and what I thought was a slave was in fact a master. Issue on my end, closing