Skip to content

[Improvement](segment iterator) Optimize column row reservation to reduce overhead#42060

Merged
airborne12 merged 3 commits intoapache:masterfrom
airborne12:improve-perf
Oct 22, 2024
Merged

[Improvement](segment iterator) Optimize column row reservation to reduce overhead#42060
airborne12 merged 3 commits intoapache:masterfrom
airborne12:improve-perf

Conversation

@airborne12
Copy link
Member

@airborne12 airborne12 commented Oct 18, 2024 •
edited
Loading

Proposed changes

This PR improves the segment iterator by reducing the reserved rows for a column when the row bitmap is smaller than the block row. This optimization aims to enhance memory efficiency and improve performance by reducing unnecessary row allocations.

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12 airborne12 changed the title [Improve](segment iterator) reserve less rows for column when row bit… [Improvement](segment iterator) Optimize column row reservation to reduce overhead Oct 18, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Oct 18, 2024

clang-tidy review says "All clean, LGTM! 👍"

@airborne12
Copy link
Member Author

airborne12 commented Oct 18, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 18, 2024

TeamCity be ut coverage result:
Function Coverage: 37.47% (9711/25919)
Line Coverage: 28.74% (80642/280617)
Region Coverage: 28.17% (41713/148088)
Branch Coverage: 24.75% (21209/85700)
Coverage Report: http://coverage.selectdb-in.cc/coverage/820e3984e22499a150acfa8a322cfd59cfc815cc_820e3984e22499a150acfa8a322cfd59cfc815cc/report/index.html

eldenmoon
eldenmoon reviewed Oct 18, 2024
View reviewed changes
@airborne12
Copy link
Member Author

airborne12 commented Oct 21, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Oct 21, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Oct 21, 2024

TeamCity be ut coverage result:
Function Coverage: 37.47% (9708/25912)
Line Coverage: 28.75% (80651/280523)
Region Coverage: 28.18% (41718/148045)
Branch Coverage: 24.77% (21213/85656)
Coverage Report: http://coverage.selectdb-in.cc/coverage/48b8cd544d27ec4cc4715d0b7631a9a4f7fe57df_48b8cd544d27ec4cc4715d0b7631a9a4f7fe57df/report/index.html

@github-actions
Copy link
Contributor

github-actions bot commented Oct 22, 2024

clang-tidy review says "All clean, LGTM! 👍"

@airborne12
Copy link
Member Author

airborne12 commented Oct 22, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 22, 2024

TeamCity be ut coverage result:
Function Coverage: 37.45% (9703/25907)
Line Coverage: 28.71% (80547/280527)
Region Coverage: 28.14% (41656/148014)
Branch Coverage: 24.72% (21168/85644)
Coverage Report: http://coverage.selectdb-in.cc/coverage/fb2fc0ee44681e1b5ec219f5009c1c7431dd82c8_fb2fc0ee44681e1b5ec219f5009c1c7431dd82c8/report/index.html

zzzxl1993
zzzxl1993 approved these changes Oct 22, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Oct 22, 2024

PR approved by anyone and no changes requested.

qidaye
qidaye approved these changes Oct 22, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Oct 22, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 22, 2024
@airborne12
Copy link
Member Author

airborne12 commented Oct 22, 2024

run p1

@airborne12
Copy link
Member Author

airborne12 commented Oct 22, 2024

run cloud_p1

@airborne12 airborne12 merged commit 8653616 into apache:master Oct 22, 2024
@airborne12 airborne12 deleted the improve-perf branch October 22, 2024 10:53
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Oct 24, 2024
@airborne12
[Improvement](segment iterator) Optimize column row reservation to re…
b4b1b2c 
…duce overhead (apache#42060)

## Proposed changes

This PR improves the segment iterator by reducing the reserved rows for
a column when the row bitmap is smaller than the block row. This
optimization aims to enhance memory efficiency and improve performance
by reducing unnecessary row allocations.
@airborne12 airborne12 mentioned this pull request Oct 24, 2024
Merged
airborne12 added a commit that referenced this pull request Oct 24, 2024
@airborne12
[Improvement](segment iterator) Optimize column row reservation to re…
f96ac18 
…duce overhead #42060 (#42372)

cherry pick from #42060
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eldenmoon eldenmoon eldenmoon left review comments

@qidaye qidaye qidaye approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@airborne12 @doris-robot @qidaye @zzzxl1993 @eldenmoon