[improvement](executor) Support task group schedule in pipeline engine #17615

liutang123 · 2023-03-09T09:53:09Z

Proposed changes

Issue Number: close #xxx

Problem summary

Describe your changes.

Checklist(Required)

Does it affect the original behavior
Has unit tests been added
Has document been added or modified
Does it need to update dependencies
Is this PR support rollback (If NO, please explain WHY)

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

github-actions

clang-tidy made some suggestions

github-actions

clang-tidy made some suggestions

be/src/pipeline/pipeline_task.cpp

github-actions · 2023-03-10T02:38:42Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-03-10T07:27:01Z

clang-tidy review says "All clean, LGTM! 👍"

be/src/runtime/task_group/task_group_manager.h

be/src/runtime/task_group/task_group.h

wangbo · 2023-03-10T07:50:45Z

be/src/runtime/task_group/task_group.h

+
+class TaskGroup;
+
+class TaskGroupEntity {


What 's the difference between TaskGroupEntity and TaskGroup?
How about combining both to TaskGroup ?

TaskGroupEntity is a sched entity, maybe a scan task, exec task or task group in the future.

be/src/runtime/task_group/task_group.h

be/src/runtime/task_group/task_group.cpp

levy5307 · 2023-03-11T07:39:36Z

be/src/runtime/task_group/task_group.h

+class TaskGroupEntity {
+public:
+    explicit TaskGroupEntity(taskgroup::TaskGroup* rs) : _rs(rs) {}
+    void push_back(pipeline::PipelineTask* task);


It’s better to use smart pointer instead of raw pointer

levy5307 · 2023-03-11T07:45:03Z

be/src/runtime/task_group/task_group_manager.h

+    // TODO pipeline task group
+    TaskGroupPtr get_or_create_task_group(uint64_t id);
+
+    static constexpr uint64_t DEFAULT_RG_ID = 0;


What about using enum class to define resource group id?

be/src/runtime/task_group/task_group.h

be/src/runtime/task_group/task_group_manager.h

be/src/runtime/query_fragments_ctx.h

be/src/runtime/task_group/task_group_manager.cpp

be/src/pipeline/pipeline_fragment_context.h

be/src/pipeline/pipeline_task.cpp

be/src/pipeline/pipeline_task.h

be/src/pipeline/task_queue.cpp

be/src/runtime/fragment_mgr.cpp

github-actions · 2023-03-16T07:44:10Z

clang-tidy review says "All clean, LGTM! 👍"

be/src/pipeline/task_queue.h

be/src/pipeline/task_queue.cpp

be/src/runtime/task_group/task_group_manager.cpp

be/src/runtime/task_group/task_group_manager.h

github-actions · 2023-03-17T06:05:35Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-03-17T06:14:00Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-03-17T08:10:48Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2023-03-18T09:38:04Z

clang-tidy review says "All clean, LGTM! 👍"

liutang123 · 2023-03-18T10:17:16Z

run buildall

hello-stephen · 2023-03-18T11:43:18Z

TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 35.13 seconds
stream load tsv: 444 seconds loaded 74807831229 Bytes, about 160 MB/s
stream load json: 21 seconds loaded 2358488459 Bytes, about 107 MB/s
stream load orc: 73 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 30 seconds loaded 861443392 Bytes, about 27 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230329104516_clickbench_pr_122298.html

liutang123 · 2023-03-20T02:11:20Z

run p0

liutang123 · 2023-03-20T03:01:24Z

run p0

liutang123 · 2023-03-20T14:20:54Z

run buildall

github-actions · 2023-03-20T14:29:02Z

clang-tidy review says "All clean, LGTM! 👍"

liutang123 · 2023-03-29T09:12:57Z

run buildall

github-actions · 2023-03-29T09:21:01Z

clang-tidy review says "All clean, LGTM! 👍"

wangbo

LGTM

github-actions · 2023-03-30T01:51:20Z

PR approved by at least one committer and no changes requested.

Gabriel39 · 2023-03-30T02:50:32Z

Pls complete the description

apache#17615)

github-actions bot reviewed Mar 9, 2023

View reviewed changes

be/src/pipeline/pipeline_task.cpp Outdated Show resolved Hide resolved

be/src/pipeline/pipeline_task.cpp Outdated Show resolved Hide resolved

apache deleted a comment from github-actions bot Mar 10, 2023

wangbo reviewed Mar 10, 2023

View reviewed changes

be/src/runtime/task_group/task_group_manager.h Outdated Show resolved Hide resolved

wangbo reviewed Mar 10, 2023

View reviewed changes

be/src/runtime/task_group/task_group.h Show resolved Hide resolved

wangbo reviewed Mar 10, 2023

View reviewed changes

levy5307 reviewed Mar 11, 2023

View reviewed changes

liutang123 force-pushed the resource-iso branch from 4cefb2d to 6d3a9ef Compare March 16, 2023 07:36

levy5307 reviewed Mar 16, 2023

View reviewed changes

be/src/pipeline/task_queue.h Outdated Show resolved Hide resolved

be/src/pipeline/task_queue.cpp Outdated Show resolved Hide resolved

wangbo reviewed Mar 17, 2023

View reviewed changes

be/src/runtime/task_group/task_group_manager.cpp Outdated Show resolved Hide resolved

wangbo reviewed Mar 17, 2023

View reviewed changes

be/src/runtime/task_group/task_group_manager.h Outdated Show resolved Hide resolved

liutang123 force-pushed the resource-iso branch from 56d80ff to 053e473 Compare March 17, 2023 06:04

liutang123 changed the title ~~Init task Group func~~ [improvement](executor) Support task group schedule in pipeline engine Mar 18, 2023

liutang123 marked this pull request as ready for review March 18, 2023 10:16

liutang123 force-pushed the resource-iso branch from c067e55 to 9b81302 Compare March 20, 2023 14:18

liutang123 added 17 commits March 29, 2023 17:10

tmp

65ff114

tmp

c40b6b0

format code

72b5ff1

fix bug

c74357c

fix compile

3fa8865

fix compile 1

60f332f

Remove unneeded comment.

3dbe37c

review 1

a66de68

fix compile

71c4221

debug log

9c21d9c

Fix TaskGroupSchedEntityComparator bug

787b572

timeout_second

13739b9

code format

ae5fb95

unify query ctx creation

8873566

Reformat

1566cc7

Fix compile

821f689

Change regression be conf

fd93d03

liutang123 force-pushed the resource-iso branch from 90d4127 to fd93d03 Compare March 29, 2023 09:12

wangbo approved these changes Mar 29, 2023

View reviewed changes

luozenglin approved these changes Mar 30, 2023

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 30, 2023

Gabriel39 merged commit 2ee1468 into apache:master Mar 30, 2023

yagagagaga pushed a commit to yagagagaga/doris that referenced this pull request Mar 31, 2023

[improvement](executor) Support task group schedule in pipeline engine (

ca42de7

apache#17615)

yagagagaga pushed a commit to yagagagaga/doris that referenced this pull request Mar 31, 2023

[improvement](executor) Support task group schedule in pipeline engine (

87c088e

apache#17615)

yagagagaga pushed a commit to yagagagaga/doris that referenced this pull request Mar 31, 2023

[improvement](executor) Support task group schedule in pipeline engine (

ef7dfae

apache#17615)

gnehil pushed a commit to gnehil/doris that referenced this pull request Apr 21, 2023

[improvement](executor) Support task group schedule in pipeline engine (

0f12afc

apache#17615)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[improvement](executor) Support task group schedule in pipeline engine #17615

[improvement](executor) Support task group schedule in pipeline engine #17615

liutang123 commented Mar 9, 2023

github-actions bot left a comment

github-actions bot left a comment

github-actions bot commented Mar 10, 2023

github-actions bot commented Mar 10, 2023

wangbo Mar 10, 2023

levy5307 Mar 11, 2023

liutang123 Mar 15, 2023

levy5307 Mar 11, 2023

levy5307 Mar 11, 2023

github-actions bot commented Mar 16, 2023

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 18, 2023

liutang123 commented Mar 18, 2023

hello-stephen commented Mar 18, 2023 •

edited

Loading

liutang123 commented Mar 20, 2023

liutang123 commented Mar 20, 2023

liutang123 commented Mar 20, 2023

github-actions bot commented Mar 20, 2023

liutang123 commented Mar 29, 2023

github-actions bot commented Mar 29, 2023

wangbo left a comment

github-actions bot commented Mar 30, 2023

Gabriel39 commented Mar 30, 2023


		class TaskGroup;

		class TaskGroupEntity {

[improvement](executor) Support task group schedule in pipeline engine #17615

[improvement](executor) Support task group schedule in pipeline engine #17615

Conversation

liutang123 commented Mar 9, 2023

Proposed changes

Problem summary

Checklist(Required)

Further comments

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 10, 2023

github-actions bot commented Mar 10, 2023

wangbo Mar 10, 2023

Choose a reason for hiding this comment

levy5307 Mar 11, 2023

Choose a reason for hiding this comment

liutang123 Mar 15, 2023

Choose a reason for hiding this comment

levy5307 Mar 11, 2023

Choose a reason for hiding this comment

levy5307 Mar 11, 2023

Choose a reason for hiding this comment

github-actions bot commented Mar 16, 2023

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 17, 2023

github-actions bot commented Mar 18, 2023

liutang123 commented Mar 18, 2023

hello-stephen commented Mar 18, 2023 • edited Loading

liutang123 commented Mar 20, 2023

liutang123 commented Mar 20, 2023

liutang123 commented Mar 20, 2023

github-actions bot commented Mar 20, 2023

liutang123 commented Mar 29, 2023

github-actions bot commented Mar 29, 2023

wangbo left a comment

Choose a reason for hiding this comment

github-actions bot commented Mar 30, 2023

Gabriel39 commented Mar 30, 2023

hello-stephen commented Mar 18, 2023 •

edited

Loading