Skip to content

remove COMMAND GETKEYS call from ZUNION and ZUNIONSTORE #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 4, 2023

Conversation

zach-iee
Copy link

@zach-iee zach-iee commented Jun 30, 2023

Pull Request check-list

Please make sure to review and check all of these items:

  • Does $ tox pass with this change (including linting)?
  • Do the CI tests pass with this change (enable it first in your forked repo and wait for the github action build to finish)?
  • Is the new or changed code fully tested?
  • Is a documentation update included (if this change modifies existing APIs, or introduces new ones)?
  • Is there an example added to the examples folder (if applicable)?
  • Was the change added to CHANGES file?

NOTE: these things are not required to open a PR and can be done
afterwards / while the PR is open.

Description of change

Redis 7 has a bug, which causes the server to crash with large size of keys in COMMAND GETKEYS
redis/redis#12380

In order for us to execute ZUNIONSTORE without a crash, this PR manually extracts keys from the ZUNIONSTORE without calling COMMAND GETKEYS

@zach-iee zach-iee self-assigned this Jun 30, 2023
@codecov-commenter
Copy link

codecov-commenter commented Jun 30, 2023

Codecov Report

Patch coverage: 93.75% and project coverage change: -0.01 ⚠️

Comparison is base (308c82b) 92.36% compared to head (4b228cf) 92.35%.

Additional details and impacted files
@@            Coverage Diff             @@
##           master       #4      +/-   ##
==========================================
- Coverage   92.36%   92.35%   -0.01%     
==========================================
  Files         119      119              
  Lines       30600    30615      +15     
==========================================
+ Hits        28265    28276      +11     
- Misses       2335     2339       +4     
Impacted Files Coverage Δ
redis/cluster.py 90.75% <83.33%> (-0.07%) ⬇️
tests/test_cluster.py 96.94% <100.00%> (-0.09%) ⬇️

... and 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@zach-iee zach-iee force-pushed the zach/remove_command_getkeys branch 2 times, most recently from d47dbe7 to 82ec943 Compare July 4, 2023 02:50
@zach-iee zach-iee changed the title [WIP] remove COMMAND GETKEYS call from ZUNION and ZUNIONSTORE remove COMMAND GETKEYS call from ZUNION and ZUNIONSTORE Jul 4, 2023
@zach-iee zach-iee force-pushed the zach/remove_command_getkeys branch from 82ec943 to 4b228cf Compare July 4, 2023 04:14
@zach-iee zach-iee requested review from bellatoris and noah-chae July 4, 2023 05:44
@@ -968,6 +968,14 @@ def determine_slot(self, *args):
if len(eval_keys) == 0:
return random.randrange(0, REDIS_CLUSTER_HASH_SLOTS)
keys = eval_keys
elif command == "ZUNIONSTORE":

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

georadius
georadiusbymember
migrate
sort
sort_ro
zdiffstore
zinterstore

shouldn't we raise an exception for above commands?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I assume that we are not using those commands, but just in case.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeap, i'm trying to block those commands in soda code, let me write the PR link once it's ready

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

got it, please be careful that we should block direct call like execute_command(sort)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link

@noah-chae noah-chae left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀

@zach-iee zach-iee merged commit 68c3505 into master Jul 4, 2023
@zach-iee zach-iee deleted the zach/remove_command_getkeys branch July 4, 2023 09:58
zach-iee added a commit that referenced this pull request Jul 4, 2023
* make EVALSHA and EVALSHA_RO to avoid command_keys as well

* make ZUNION, ZUNIONSTORE to avoid command_keys as well
zach-iee added a commit that referenced this pull request Jul 4, 2023
* make EVALSHA and EVALSHA_RO to avoid command_keys as well

* make ZUNION, ZUNIONSTORE to avoid command_keys as well
zach-iee added a commit that referenced this pull request Jul 7, 2023
* make EVALSHA and EVALSHA_RO to avoid command_keys as well

* make ZUNION, ZUNIONSTORE to avoid command_keys as well
zach-iee added a commit that referenced this pull request Aug 21, 2023
* make EVALSHA and EVALSHA_RO to avoid command_keys as well

* make ZUNION, ZUNIONSTORE to avoid command_keys as well
zach-iee added a commit that referenced this pull request Aug 21, 2023
* make EVALSHA and EVALSHA_RO to avoid command_keys as well

* make ZUNION, ZUNIONSTORE to avoid command_keys as well
zach-iee added a commit that referenced this pull request Sep 14, 2023
* make EVALSHA and EVALSHA_RO to avoid command_keys as well

* make ZUNION, ZUNIONSTORE to avoid command_keys as well
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants