Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Enhancement] Rank topn prune redundant rows #8108

Merged
merged 6 commits into from
Jul 4, 2022

Conversation

liuyehcf
Copy link
Contributor

@liuyehcf liuyehcf commented Jun 30, 2022

What type of PR is this:

  • bug
  • feature
  • enhancement
  • refactor
  • others

Which issues of this PR fixes :

Fixes #5885

Enhancement

In previous work #6120, I support rank topn. Comparing to normal topn, rank-topn should take equal range into consideration, but the result is not well-processed, it's result may contain more rows than we need.

For example, given input (1 2 3 3 3 4 5) with limit=3, the expected output should be (1 2 3 3 3), but the actual result may be (1 2 3 3 3)(1 2 3 3 3 4)(1 2 3 3 3 4 5), which has the prefix of the expected result and with redundant elements.

Although it's result is not correct, the plan still works fine, because the late full sort and filter will get the correct results.

In this pr, I add pruning mechanism to remove the redundant rows

Change list

  • Pruning after done rank topn
  • Simplify some conditions of rank topn.
  • Add ut for rank topn

@liuyehcf liuyehcf force-pushed the bugfix branch 5 times, most recently from 670a7b1 to aabed87 Compare June 30, 2022 11:01
@liuyehcf liuyehcf changed the title [Bugfix] Fix topn assertion error [Enhancement] Rank topn prune redundant rows Jun 30, 2022
@liuyehcf liuyehcf force-pushed the bugfix branch 2 times, most recently from 5b02071 to 92ce23f Compare June 30, 2022 14:08
@liuyehcf liuyehcf requested review from Pslydhh, trueeyu and murphyatwork and removed request for Pslydhh and trueeyu July 1, 2022 00:13
@liuyehcf liuyehcf requested a review from murphyatwork July 1, 2022 08:44
Pslydhh
Pslydhh previously approved these changes Jul 1, 2022
murphyatwork
murphyatwork previously approved these changes Jul 1, 2022
murphyatwork
murphyatwork previously approved these changes Jul 4, 2022
@kangkaisen kangkaisen merged commit 7833a32 into StarRocks:main Jul 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Enhancement] Rank window function optimization
4 participants