Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SR-4078] Add exchange merge sort operator #205

Merged
merged 11 commits into from
Sep 15, 2021
Merged

[SR-4078] Add exchange merge sort operator #205

merged 11 commits into from
Sep 15, 2021

Conversation

Pslydhh
Copy link
Contributor

@Pslydhh Pslydhh commented Sep 14, 2021

The transmission thread stores data through a queue and notifies the computing thread to grab the data. However, in pipeline, this step needs to be observed by the dispatch thread. Because computing thread summarizes multiple sorted data, the compute thread does not read the queue directly, it reads the data through an indirect minimum heap and constantly adjusts the minimum heap.
Computing thread works when cursor has datas or there is chunk in chunk_queue(need to grab), it Blocking in the opposite case.

  • get_next_merging in ExchangeMergeSortSourceOperator is almost like it in ExchangeNode except it willn't blocking through should_exit.
  • we add is_pipeline to DataStreamRecvr to use in pipeline.
  • add_chunks_for_pipeline is almost like add_chunks except it don't notify compute thread.
  • add chunk_probe_supplier and chunk_has_supplier for pipeline, because chunk_supplier is a blocking interface.
  • and chunk_has_supplier is used in operator's has_output, chunk_probe_supplier is used to get chunk from _chunk_queue.

@satanson
Copy link
Contributor

run starrocks_be_unittest

be/src/exec/exchange_node.cpp Outdated Show resolved Hide resolved
be/src/exec/exchange_node.cpp Outdated Show resolved Hide resolved
be/src/runtime/data_stream_recvr.cc Outdated Show resolved Hide resolved
be/src/runtime/data_stream_recvr.cc Show resolved Hide resolved
be/src/runtime/data_stream_recvr.cc Outdated Show resolved Hide resolved
be/src/runtime/data_stream_recvr.cc Outdated Show resolved Hide resolved
be/src/runtime/data_stream_recvr.cc Outdated Show resolved Hide resolved
satanson
satanson previously approved these changes Sep 14, 2021
satanson
satanson previously approved these changes Sep 14, 2021
satanson
satanson previously approved these changes Sep 14, 2021
liuyehcf
liuyehcf previously approved these changes Sep 14, 2021
@Pslydhh Pslydhh dismissed stale reviews from liuyehcf and satanson via 2158d05 September 15, 2021 02:24
satanson
satanson previously approved these changes Sep 15, 2021
@CLAassistant
Copy link

CLAassistant commented Sep 15, 2021

CLA assistant check
All committers have signed the CLA.

@kangkaisen kangkaisen merged commit 5c43663 into StarRocks:main Sep 15, 2021
jaogoy pushed a commit that referenced this pull request Jul 14, 2022
* update docs

including:
other faqs
CREATE TABLE
ROUTINE LOAD

* release 2 english docs

configure a time zone
synchronize data from mysql

* Update Flink_cdc_load.md

* Update Flink_cdc_load.md

* Update Flink_cdc_load.md

* fix url

fix url

* add and update docs

add restore FEs
update release note 2.2

* Update release-2.2.md

* Update release-2.2.md

FIX URL

* Update Metadata_recovery.md

MODIFY FORMAT
caneGuy pushed a commit to caneGuy/starrocks that referenced this pull request Mar 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants