Redis Cluster implementation is broken

[felixwang9817]

Expected Behavior

Current Behavior

Recently a Redis cluster bug was fixed in #2311 and tests were added in #2317. However, these tests revealed other issues with our Redis cluster implementation.

One issue is the way we currently scan for rows to delete:

client.scan_iter(b"".join([prefix, b"*", config.project.encode("utf8")]))

First, scan_iter only hits a single master node by default, so this line should instead be something like

client.scan_iter(b"".join([prefix, b"*", config.project.encode("utf8")]), target_nodes=RedisCluster.PRIMARIES)

However, this is still broken. After some digging, it seems that scan_iter relies on scan, which is itself broken - I believe this is an issue with redis-py. I filed an issue here.

Another issue seems to be with our current pipelining logic. We have with client.pipeline(transaction=False) as pipe:, but this leads to the following error:

redis.exceptions.ResponseError: Command # 1 (HMGET drivertest_universal_cli_26a064e4 _ts:driver_locations) of pipeline caused error: MOVED 12671 127.0.0.1:6003

Steps to reproduce

Specifications

• Version:
• Platform:
• Subsystem:

Possible Solution

This issue tracks a resolution to the existing Redis cluster bugs. After they've been fixed, we should reenable Redis cluster tests, which were disabled in #2327 to unblock other PRs.

[kevjumba]

@felixwang9817 I think this issue can be closed pending further upstream changes from redis? The core issue with our tests is fixed.