Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Pipeline] Fix PipScannerContext::can_finish return wrong status #15259

Merged
merged 4 commits into from
Jan 9, 2023
Merged

[Pipeline] Fix PipScannerContext::can_finish return wrong status #15259

merged 4 commits into from
Jan 9, 2023

Conversation

liutang123
Copy link
Contributor

@liutang123 liutang123 commented Dec 21, 2022 •
edited
Loading

Proposed changes

Issue Number: close #xxx

Problem summary

Now in ScannerContext::push_back_scanner_and_reschedule, _num_running_scanners-- is before _num_scheduling_ctx++.
InPipScannerContext::can_finish, we check _num_running_scanners == 0 && _num_scheduling_ctx == 0 without obtaining _transfer_lock.
In follow case, PipScannerContext::can_finish will return wrong result.

  1. _num_running_scanners--
  2. Check _num_running_scanners == 0 && _num_scheduling_ctx == 0` return true.
  3. _num_scheduling_ctx++

So, we can set _num_running_scanners-- in the last of this func.

Describe your changes.

  1. PipScannerContext::get_block_from_queue not block.
  2. Set _num_running_scanners-- in the last of ScannerContext::push_back_scanner_and_reschedule.

Checklist(Required)

  1. Does it affect the original behavior:
    • Yes
    • No
    • I don't know
  2. Has unit tests been added:
    • Yes
    • No
    • No Need
  3. Has document been added or modified:
    • Yes
    • No
    • No Need
  4. Does it need to update dependencies:
    • Yes
    • No
  5. Are there any changes that cannot be rolled back:
    • Yes (If Yes, please explain WHY)
    • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 21, 2022 •
edited
Loading

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 36.52 seconds
load time: 650 seconds
storage size: 17123681856 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221230065551_clickbench_pr_71499.html

@liutang123 liutang123 force-pushed the fix-pipline-scan-pending-finish-error branch from 4fd9fa7 to f143dff Compare December 23, 2022 08:52
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@liutang123 liutang123 force-pushed the fix-pipline-scan-pending-finish-error branch from 2f788f6 to 9711cb7 Compare December 25, 2022 09:30
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@liutang123 liutang123 force-pushed the fix-pipline-scan-pending-finish-error branch from 9711cb7 to e127f05 Compare December 26, 2022 12:15
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@liutang123 liutang123 force-pushed the fix-pipline-scan-pending-finish-error branch from e127f05 to 3a9fe5c Compare December 27, 2022 09:17
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@liutang123 liutang123 force-pushed the fix-pipline-scan-pending-finish-error branch from 3a9fe5c to 6e67296 Compare December 30, 2022 04:02
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@liutang123 liutang123 force-pushed the fix-pipline-scan-pending-finish-error branch from 2f448ca to 6e67296 Compare December 30, 2022 14:27
@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

HappenLee
HappenLee approved these changes Jan 5, 2023
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 5, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jan 5, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 5, 2023

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit c57fa7c into apache:master Jan 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@HappenLee HappenLee HappenLee approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. area/vectorization reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@liutang123 @hello-stephen @HappenLee @yiguolei