[GROW-3598] make initialize to loop through replica nodes per cluster slots row

[zach-iee]

Pull Request check-list

Please make sure to review and check all of these items:

• Does $ tox pass with this change (including linting)?
• Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
• Is the new or changed code fully tested?
• Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
• Is there an example added to the examples folder (if applicable)?
• Was the change added to CHANGES file?

NOTE: these things are not required to open a PR and can be done
afterwards / while the PR is open.

Description of change

previously initialize method was iterating over replica nodes for 16,384 times (number of slots), so it was slow. This PR improves the speed by making the method to iterate for the number of CLUSTER SLOTS row.

I've ran test on hinge's actual CLUSTER SLOTS response and the result and code is show below
result:

original implementation: number=1000, 15.134753834019648
new implementation: number=1000, 3.0220605000213254

test code:

    def test_initialize_speed(self):
        cluster_slots = [
            [0,
             2,
             ["10.81.110.8", 6379, "0159248445fa2b4db225afc6fad21ab4c5295bf3", ["hostname", ""]],
             ["10.81.110.166", 6379, "447b487165023e1f84d49fbc0515bcf76fabb8f9", ["hostname", ""]]],
            [3,
             186,
             ["10.81.110.137", 6379, "ff77be7a8adb43bf6b10366365f041ed5fc0e5b", ["hostname", ""]],
             ["10.81.110.72", 6379, "78e6fb1b41eac11e4ff34dc8d528dc48738519da", ["hostname", ""]]],
            [187,
             816,
             ["10.81.110.69", 6379, "9eb526cb05f6bb282b1f08aecb30126529f44067", ["hostname", ""]],
             ["10.81.110.159", 6379, "6257e0b08902f7f268eb3096e7a53cbf82db72c4", ["hostname", ""]]],
            [817,
             911,
             ["10.81.110.137", 6379, "ff77be7a8adb43bf6b10366365f041ed5fc0e5b", ["hostname", ""]],
             ["10.81.110.72", 6379, "78e6fb1b41eac11e4ff34dc8d528dc48738519da", ["hostname", ""]]],
            [912,
             1541,
             ["10.81.110.176", 6379, "fcdff65590b0440e39f6cab456eb26f47b0ac228", ["hostname", ""]],
             ["10.81.110.52", 6379, "4f0fba418c16ba855cdccbd507650e2974323419", ["hostname", ""]]],
            [1542,
             1725,
             ["10.81.110.137", 6379, "ff77be7a8adb43bf6b10366365f041ed5fc0e5b", ["hostname", ""]],
             ["10.81.110.72", 6379, "78e6fb1b41eac11e4ff34dc8d528dc48738519da", ["hostname", ""]]],
            [1726,
             2355,
             ["10.81.110.141", 6379, "b0d18e74f404e6de01546b5fb7a3f9f32a0c39c2", ["hostname", ""]],
             ["10.81.110.38", 6379, "df186272a93ae8033ac9bb89798287315f62c1a8", ["hostname", ""]]],
            [2356,
             2423,
             ["10.81.110.137", 6379, "ff77be7a8adb43bf6b10366365f041ed5fc0e5b", ["hostname", ""]],
             ["10.81.110.72", 6379, "78e6fb1b41eac11e4ff34dc8d528dc48738519da", ["hostname", ""]]],
            [2424,
             3054,
             ["10.81.110.184", 6379, "b0c0812a182f4e998fbd8225b544e824d8beaf02", ["hostname", ""]],
             ["10.81.110.39", 6379, "d390f31082c5c829325960a414cc0d49ffe246ae", ["hostname", ""]]],
            [3055,
             3081,
             ["10.81.110.137", 6379, "ff77be7a8adb43bf6b10366365f041ed5fc0e5b", ["hostname", ""]],
             ["10.81.110.72", 6379, "78e6fb1b41eac11e4ff34dc8d528dc48738519da", ["hostname", ""]]],
            [3082,
             3549,
             ["10.81.110.182", 6379, "cedf7808383c79ddc8b9a088010589bb9894e8d6", ["hostname", ""]],
             ["10.81.110.59", 6379, "a5293952ca4b3072727751cb19f72de4ece3f529", ["hostname", ""]]],
            [3550,
             3980,
             ["10.81.110.108", 6379, "d31312cf21f250c42c0a42a230f73e050a7160a1", ["hostname", ""]],
             ["10.81.110.133", 6379, "f166a71180e625e265639b0fa012e80c9b2d6f7", ["hostname", ""]]],
            [3981,
             4573,
             ["10.81.110.94", 6379, "2b4eee7d594fb76887edfd57e0ec4e7f383acbb5", ["hostname", ""]],
             ["10.81.110.254", 6379, "7bb8b643fd9f44f7a1634dc77e7eee6944a6c824", ["hostname", ""]]],
            [4574,
             4792,
             ["10.81.110.75", 6379, "3b3e0e415f04f4de8c909deab27441f34b9be726", ["hostname", ""]],
             ["10.81.110.210", 6379, "dea6c0219f2896a605228851c398b857e4ebede3", ["hostname", ""]]],
            [4793,
             5422,
             ["10.81.110.208", 6379, "fa5837ad09afcecfaa5fe25b2be7b023e5ca7416", ["hostname", ""]],
             ["10.81.110.4", 6379, "fefcf8780f83612214dbf309ea3ad97436b7cb63", ["hostname", ""]]],
            [5423,
             5833,
             ["10.81.110.75", 6379, "3b3e0e415f04f4de8c909deab27441f34b9be726", ["hostname", ""]],
             ["10.81.110.210", 6379, "dea6c0219f2896a605228851c398b857e4ebede3", ["hostname", ""]]],
            [5834,
             5870,
             ["10.81.110.94", 6379, "2b4eee7d594fb76887edfd57e0ec4e7f383acbb5", ["hostname", ""]],
             ["10.81.110.254", 6379, "7bb8b643fd9f44f7a1634dc77e7eee6944a6c824", ["hostname", ""]]],
            [5871,
             5941,
             ["10.81.110.108", 6379, "d31312cf21f250c42c0a42a230f73e050a7160a1", ["hostname", ""]],
             ["10.81.110.133", 6379, "f166a71180e625e265639b0fa012e80c9b2d6f7", ["hostname", ""]]],
            [5942,
             6318,
             ["10.81.110.80", 6379, "d6001f842500be13c32679f10c57c34c25b9c8ca", ["hostname", ""]],
             ["10.81.110.250", 6379, "2a903500f54dd7b151da3e2ce122d34df359ca6d", ["hostname", ""]]],
            [6319,
             6649,
             ["10.81.110.145", 6379, "18da195cca7271000e3814154fd3c7e543ff00c1", ["hostname", ""]],
             ["10.81.110.37", 6379, "ce74c0de873051f363a04ceb8d01963434d279ac", ["hostname", ""]]],
            [6650,
             7279,
             ["10.81.110.138", 6379, "9c89da6b76931322dfb3abb64044b4bee6a7999d", ["hostname", ""]],
             ["10.81.110.100", 6379, "565b81145a24b8e4dab61452c773fae7b58f8cc3", ["hostname", ""]]],
            [7280,
             7578,
             ["10.81.110.145", 6379, "18da195cca7271000e3814154fd3c7e543ff00c1", ["hostname", ""]],
             ["10.81.110.37", 6379, "ce74c0de873051f363a04ceb8d01963434d279ac", ["hostname", ""]]],
            [7579,
             7723,
             ["10.81.110.80", 6379, "d6001f842500be13c32679f10c57c34c25b9c8ca", ["hostname", ""]],
             ["10.81.110.250", 6379, "2a903500f54dd7b151da3e2ce122d34df359ca6d", ["hostname", ""]]],
            [7724,
             8353,
             ["10.81.110.202", 6379, "085594987ea3eaeeab384e75df9f9f628124ff91", ["hostname", ""]],
             ["10.81.110.9", 6379, "600e448e1b762103050b4e4ca7ea6d69efc0572", ["hostname", ""]]],
            [8354,
             8436,
             ["10.81.110.80", 6379, "d6001f842500be13c32679f10c57c34c25b9c8ca", ["hostname", ""]],
             ["10.81.110.250", 6379, "2a903500f54dd7b151da3e2ce122d34df359ca6d", ["hostname", ""]]],
            [8437,
             8904,
             ["10.81.110.172", 6379, "9d13f634ab4282ad6a1cdaba4d22b9295bfbae20", ["hostname", ""]],
             ["10.81.110.117", 6379, "d06f237bc9d89ecb812afb5ee00dcc46458175c3", ["hostname", ""]]],
            [8905,
             9534,
             ["10.81.110.11", 6379, "8438a147c6a358601a816c2df9049ee11da53bf7", ["hostname", ""]],
             ["10.81.110.163", 6379, "43db90add13aa9be8db60c755f2544dafbb01a19", ["hostname", ""]]],
            [9535,
             9696,
             ["10.81.110.172", 6379, "9d13f634ab4282ad6a1cdaba4d22b9295bfbae20", ["hostname", ""]],
             ["10.81.110.117", 6379, "d06f237bc9d89ecb812afb5ee00dcc46458175c3", ["hostname", ""]]],
            [9697,
             9721,
             ["10.81.110.80", 6379, "d6001f842500be13c32679f10c57c34c25b9c8ca", ["hostname", ""]],
             ["10.81.110.250", 6379, "2a903500f54dd7b151da3e2ce122d34df359ca6d", ["hostname", ""]]],
            [9722,
             9849,
             ["10.81.110.108", 6379, "d31312cf21f250c42c0a42a230f73e050a7160a1", ["hostname", ""]],
             ["10.81.110.133", 6379, "f166a71180e625e265639b0fa012e80c9b2d6f7", ["hostname", ""]]],
            [9850,
             9881,
             ["10.81.110.182", 6379, "cedf7808383c79ddc8b9a088010589bb9894e8d6", ["hostname", ""]],
             ["10.81.110.59", 6379, "a5293952ca4b3072727751cb19f72de4ece3f529", ["hostname", ""]]],
            [9882,
             10255,
             ["10.81.110.112", 6379, "e7c3bba886096bb2a945ee74d2132e421811cf32", ["hostname", ""]],
             ["10.81.110.136", 6379, "ca432a75923192ca4841ba8cdf527677d9fa95bc", ["hostname", ""]]],
            [10256,
             10886,
             ["10.81.110.124", 6379, "ba495edae8472356c44d833eb19d58d82b88ba88", ["hostname", ""]],
             ["10.81.110.161", 6379, "02fb03a026027980c79a38723f77f6a0f8959bd1", ["hostname", ""]]],
            [10887,
             11142,
             ["10.81.110.112", 6379, "e7c3bba886096bb2a945ee74d2132e421811cf32", ["hostname", ""]],
             ["10.81.110.136", 6379, "ca432a75923192ca4841ba8cdf527677d9fa95bc", ["hostname", ""]]],
            [11143,
             11272,
             ["10.81.110.182", 6379, "cedf7808383c79ddc8b9a088010589bb9894e8d6", ["hostname", ""]],
             ["10.81.110.59", 6379, "a5293952ca4b3072727751cb19f72de4ece3f529", ["hostname", ""]]],
            [11273,
             11344,
             ["10.81.110.137", 6379, "ff77be7a8adb43bf6b10366365f041ed5fc0e5b", ["hostname", ""]],
             ["10.81.110.72", 6379, "78e6fb1b41eac11e4ff34dc8d528dc48738519da", ["hostname", ""]]],
            [11345,
             11809,
             ["10.81.110.8", 6379, "0159248445fa2b4db225afc6fad21ab4c5295bf3", ["hostname", ""]],
             ["10.81.110.166", 6379, "447b487165023e1f84d49fbc0515bcf76fabb8f9", ["hostname", ""]]],
            [11810,
             12439,
             ["10.81.110.199", 6379, "8ea5a8ae58fda71ba33f1a56a6af0b7c3cd6cb8a", ["hostname", ""]],
             ["10.81.110.32", 6379, "908c046680a58d5541c9326cea1866aff58b2f7c", ["hostname", ""]]],
            [12440,
             12601,
             ["10.81.110.8", 6379, "0159248445fa2b4db225afc6fad21ab4c5295bf3", ["hostname", ""]],
             ["10.81.110.166", 6379, "447b487165023e1f84d49fbc0515bcf76fabb8f9", ["hostname", ""]]],
            [12602,
             12822,
             ["10.81.110.61", 6379, "3bf03cc67633ce9d4a162b97f40f37f50e866b01", ["hostname", ""]],
             ["10.81.110.252", 6379, "5b7b3140f6505cc210c1e23bbfb5b462e5d3614f", ["hostname", ""]]],
            [12823,
             13453,
             ["10.81.110.190", 6379, "9aff50e31615d0e839a4c00d3c7ae2b2b16bdefc", ["hostname", ""]],
             ["10.81.110.101", 6379, "fbb7520d250268d07f487630be6048c48d811d74", ["hostname", ""]]],
            [13454,
             13606,
             ["10.81.110.61", 6379, "3bf03cc67633ce9d4a162b97f40f37f50e866b01", ["hostname", ""]],
             ["10.81.110.252", 6379, "5b7b3140f6505cc210c1e23bbfb5b462e5d3614f", ["hostname", ""]]],
            [13607,
             14236,
             ["10.81.110.194", 6379, "00036c3710844350efa23344c1475c85056109cd", ["hostname", ""]],
             ["10.81.110.97", 6379, "288e0e7478ace40eea3140055cb6e32dca3fed3c", ["hostname", ""]]],
            [14237,
             14492,
             ["10.81.110.61", 6379, "3bf03cc67633ce9d4a162b97f40f37f50e866b01", ["hostname", ""]],
             ["10.81.110.252", 6379, "5b7b3140f6505cc210c1e23bbfb5b462e5d3614f", ["hostname", ""]]],
            [14493,
             15122,
             ["10.81.110.125", 6379, "7820fb8237accba727b2f29187f7d80fabc1e946", ["hostname", ""]],
             ["10.81.110.150", 6379, "ae4bd5a5dbafceb775daaae50a49669e1a851df5", ["hostname", ""]]],
            [15123,
             15123,
             ["10.81.110.61", 6379, "3bf03cc67633ce9d4a162b97f40f37f50e866b01", ["hostname", ""]],
             ["10.81.110.252", 6379, "5b7b3140f6505cc210c1e23bbfb5b462e5d3614f", ["hostname", ""]]],
            [15124,
             15414,
             ["10.81.110.70", 6379, "e4f285e094ac23738f13c9afe753da5e29d55ccc", ["hostname", ""]],
             ["10.81.110.225", 6379, "25b89ed611a398837e0e62e159bb79c1c99bc3fc", ["hostname", ""]]],
            [15415,
             16044,
             ["10.81.110.156", 6379, "5514cb5d334d05becc205b862dba11f296dbf7f6", ["hostname", ""]],
             ["10.81.110.78", 6379, "a97577e747ba1e2ed2addb84685d8f90f6a7de15", ["hostname", ""]]],
            [16045,
             16383,
             ["10.81.110.70", 6379, "e4f285e094ac23738f13c9afe753da5e29d55ccc", ["hostname", ""]],
             ["10.81.110.225", 6379, "25b89ed611a398837e0e62e159bb79c1c99bc3fc", ["hostname", ""]]]
        ]

        with patch.object(Redis, "execute_command") as execute_command_mock:

            def execute_command(*_args, **_kwargs):
                if _args[0] == "CLUSTER SLOTS":
                    mock_cluster_slots = cluster_slots
                    return mock_cluster_slots
                elif _args[0] == "COMMAND":
                    return {"get": [], "set": []}
                elif _args[0] == "INFO":
                    return {"cluster_enabled": True}
                elif len(_args) > 1 and _args[1] == "cluster-require-full-coverage":
                    return {"cluster-require-full-coverage": False}
                else:
                    return execute_command_mock(*_args, **_kwargs)

            execute_command_mock.side_effect = execute_command

            nm = NodesManager(
                startup_nodes=[ClusterNode(host=default_host, port=default_port)],
                from_url=False,
                require_full_coverage=False,
                dynamic_startup_nodes=True,
            )

            time_defaultdict = timeit.Timer(nm.initialize).timeit(number=1000)
            print(time_defaultdict)

[codecov-commenter]

Codecov Report

Patch coverage: 100.00% and project coverage change: -0.02% ⚠️

Comparison is base (16f87bf) 92.36% compared to head (2e8fa0c) 92.34%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master      #17      +/-   ##
==========================================
- Coverage   92.36%   92.34%   -0.02%     
==========================================
  Files         119      119              
  Lines       30682    30684       +2     
==========================================
- Hits        28338    28335       -3     
- Misses       2344     2349       +5     

Files Changed	Coverage Δ
redis/cluster.py	90.73% <100.00%> (+0.02%)	⬆️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[zach-iee]

test_init_slots_cache test should be good enough to cover the edited code in this PR

[bellatoris]

replica_nodes = slot[3:]

if we change this line like above, how much does performance change?

[bellatoris]

because all below lines should not affect performance much, I believe. All of its time complexity is O(1)

[zach-iee]

replica_nodes = slot[3:] improves the time spent up to 11.043858208024176

the bigger difference came from not iterating through replica nodes by the number of slots

[bellatoris]

i know your change is more neat, just curious for python performance

[bellatoris]

the bigger difference came from not iterating through replica nodes by the number of slots

Hmm? but aren't replica nodes traversed by the number of slots still in your change?

redis-py/redis/cluster.py

Lines 1577 to 1582 in 2e8fa0c

	for i in range(int(slot[0]), int(slot[1]) + 1):
	if i not in tmp_slots:
	tmp_slots[i] = []
	tmp_slots[i].append(target_node)
	for target_replica_node in target_replica_nodes:
	tmp_slots[i].append(target_replica_node)

[zach-iee]

that part is true, but looking for node doesn't have to be done in those iterations

host = str_if_bytes(replica_node[0])
port = replica_node[1]
host, port = self.remap_host_port(host, port)

target_replica_node = self._get_or_create_cluster_node(host, port, REPLICA, tmp_nodes_cache)

[bellatoris]

🤔 🤔 🤔 What's wrong with you python. why the 16384 * above operations bring that much performance difference 🐍

[zach-iee]

@bellatoris i've submitted redis#2943 to redis/redis-py repo as well