Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pipelineX](dependency) Wake by task by read dependency #27260

Merged
merged 8 commits into from
Nov 20, 2023
Merged

[pipelineX](dependency) Wake by task by read dependency #27260

merged 8 commits into from
Nov 20, 2023

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.51% (8448/23137)
Line Coverage: 28.86% (68669/237901)
Region Coverage: 27.84% (35505/127541)
Branch Coverage: 24.60% (18107/73604)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1a688a40ff6c6e9caa94beb8bedf75a5f6536329_1a688a40ff6c6e9caa94beb8bedf75a5f6536329/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.52% (8449/23137)
Line Coverage: 28.87% (68688/237901)
Region Coverage: 27.85% (35524/127541)
Branch Coverage: 24.61% (18112/73604)
Coverage Report: http://coverage.selectdb-in.cc/coverage/34e9a946840ae6a9bb447b16b0c2415c4ba50553_34e9a946840ae6a9bb447b16b0c2415c4ba50553/report/index.html

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.54% (8449/23124)
Line Coverage: 28.88% (68682/237849)
Region Coverage: 27.86% (35521/127521)
Branch Coverage: 24.61% (18110/73596)
Coverage Report: http://coverage.selectdb-in.cc/coverage/42b7206e7b969067d043ff2f7cbc0280506f7ce4_42b7206e7b969067d043ff2f7cbc0280506f7ce4/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.99 seconds
stream load tsv: 563 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17099240624 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.54% (8450/23124)
Line Coverage: 28.88% (68684/237854)
Region Coverage: 27.85% (35521/127528)
Branch Coverage: 24.61% (18111/73602)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ef58282b2239ee4839d525b704e1011fafadd68a_ef58282b2239ee4839d525b704e1011fafadd68a/report/index.html

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.57 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17100495634 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.54% (8450/23125)
Line Coverage: 28.88% (68692/237853)
Region Coverage: 27.86% (35527/127523)
Branch Coverage: 24.61% (18114/73596)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e4742db13eeb84d0c8a2bec31365f498b79e9abc_e4742db13eeb84d0c8a2bec31365f498b79e9abc/report/index.html

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit e4742db13eeb84d0c8a2bec31365f498b79e9abc, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4900	4647	4650	4647
q2	358	157	161	157
q3	2052	1942	1949	1942
q4	1391	1296	1221	1221
q5	3971	3961	4046	3961
q6	247	129	126	126
q7	1443	874	895	874
q8	2755	2775	2758	2758
q9	9796	10187	9596	9596
q10	3488	3544	3537	3537
q11	373	241	251	241
q12	435	294	298	294
q13	4578	3825	3818	3818
q14	309	286	283	283
q15	586	526	530	526
q16	658	586	586	586
q17	1122	974	955	955
q18	7809	7402	7372	7372
q19	1660	1698	1695	1695
q20	529	309	289	289
q21	4350	3947	3941	3941
q22	479	366	372	366
Total cold run time: 53289 ms
Total hot run time: 49185 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4553	4555	4576	4555
q2	340	233	248	233
q3	4018	3994	3989	3989
q4	2698	2686	2681	2681
q5	9699	9554	9556	9554
q6	246	118	124	118
q7	3017	2490	2465	2465
q8	4509	4492	4479	4479
q9	13203	13103	13066	13066
q10	4129	4209	4194	4194
q11	753	660	648	648
q12	980	811	812	811
q13	4285	3597	3591	3591
q14	377	362	358	358
q15	582	525	517	517
q16	747	679	669	669
q17	3904	3905	3912	3905
q18	9639	9075	9103	9075
q19	1813	1740	1770	1740
q20	2412	2053	2070	2053
q21	8856	8789	8666	8666
q22	852	886	811	811
Total cold run time: 81612 ms
Total hot run time: 78178 ms

@Gabriel39
Copy link
Contributor Author

run buildall

@Gabriel39
Copy link
Contributor Author

run buildall

yiguolei
yiguolei previously approved these changes Nov 20, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

HappenLee
HappenLee previously approved these changes Nov 20, 2023
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 20, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.81 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17097840772 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit c94eca7fbd505522429f3a966e03b41d7e6fed0e, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4923	4685	4681	4681
q2	356	166	159	159
q3	2033	1891	1964	1891
q4	1411	1298	1263	1263
q5	4000	3950	4017	3950
q6	250	131	130	130
q7	1409	883	894	883
q8	2765	2804	2782	2782
q9	10003	9670	9584	9584
q10	3477	3531	3563	3531
q11	380	236	252	236
q12	437	293	298	293
q13	4589	3866	3815	3815
q14	331	289	288	288
q15	597	534	535	534
q16	669	585	584	584
q17	1152	976	967	967
q18	7824	7261	7275	7261
q19	1676	1689	1682	1682
q20	531	302	292	292
q21	4358	3994	3917	3917
q22	469	365	376	365
Total cold run time: 53640 ms
Total hot run time: 49088 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4607	4620	4616	4616
q2	338	235	261	235
q3	4014	4008	4006	4006
q4	2710	2704	2703	2703
q5	9666	9754	9794	9754
q6	239	121	125	121
q7	3038	2462	2491	2462
q8	4449	4472	4473	4472
q9	13170	13109	13193	13109
q10	4127	4202	4233	4202
q11	756	683	632	632
q12	975	820	830	820
q13	4292	3560	3554	3554
q14	371	344	350	344
q15	579	527	520	520
q16	730	661	677	661
q17	3888	3855	3829	3829
q18	9595	9067	8929	8929
q19	1807	1812	1765	1765
q20	2401	2054	2061	2054
q21	8890	8599	8662	8599
q22	936	818	809	809
Total cold run time: 81578 ms
Total hot run time: 78196 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.93 seconds
stream load tsv: 580 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17099258812 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.53% (8448/23125)
Line Coverage: 28.86% (68665/237886)
Region Coverage: 27.83% (35506/127562)
Branch Coverage: 24.58% (18097/73620)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c94eca7fbd505522429f3a966e03b41d7e6fed0e_c94eca7fbd505522429f3a966e03b41d7e6fed0e/report/index.html

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.54% (8450/23125)
Line Coverage: 28.87% (68689/237886)
Region Coverage: 27.85% (35524/127562)
Branch Coverage: 24.60% (18111/73620)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1cfcf774c3cc776f18e462e8af0aa17e6c6db5f4_1cfcf774c3cc776f18e462e8af0aa17e6c6db5f4/report/index.html

@Gabriel39 Gabriel39 dismissed stale reviews from HappenLee and yiguolei via 0715bb6 November 20, 2023 11:41
@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 20, 2023
yiguolei
yiguolei approved these changes Nov 20, 2023
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 20, 2023
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 0715bb6b99b75a94cdae54031fd0b1482f0bdf7d, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4907	4682	4707	4682
q2	357	163	158	158
q3	2041	1986	1918	1918
q4	1405	1229	1262	1229
q5	3981	3959	4003	3959
q6	249	136	133	133
q7	1405	880	903	880
q8	2757	2782	2772	2772
q9	9783	9745	9564	9564
q10	3470	3539	3535	3535
q11	378	246	244	244
q12	439	299	305	299
q13	4567	3812	3822	3812
q14	316	286	287	286
q15	588	538	533	533
q16	668	586	573	573
q17	1137	939	937	937
q18	7805	7275	7290	7275
q19	1675	1700	1692	1692
q20	563	309	298	298
q21	4383	3949	3966	3949
q22	469	376	385	376
Total cold run time: 53343 ms
Total hot run time: 49104 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4577	4582	4563	4563
q2	338	230	294	230
q3	4034	4011	4000	4000
q4	2684	2680	2693	2680
q5	9755	9786	9721	9721
q6	248	125	128	125
q7	3025	2461	2458	2458
q8	4487	4452	4465	4452
q9	13304	13126	13126	13126
q10	4083	4186	4213	4186
q11	734	657	717	657
q12	977	823	814	814
q13	4279	3559	3541	3541
q14	382	344	351	344
q15	574	519	528	519
q16	721	645	672	645
q17	3900	3896	3875	3875
q18	9416	8960	9063	8960
q19	1771	1787	1771	1771
q20	2409	2083	2046	2046
q21	8883	8705	8434	8434
q22	878	800	772	772
Total cold run time: 81459 ms
Total hot run time: 77919 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.02 seconds
stream load tsv: 579 seconds loaded 74807831229 Bytes, about 123 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.6 seconds inserted 10000000 Rows, about 349K ops/s
storage size: 17099264298 Bytes

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.53% (8448/23124)
Line Coverage: 28.86% (68661/237872)
Region Coverage: 27.83% (35504/127553)
Branch Coverage: 24.59% (18103/73612)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0715bb6b99b75a94cdae54031fd0b1482f0bdf7d_0715bb6b99b75a94cdae54031fd0b1482f0bdf7d/report/index.html

@Gabriel39 Gabriel39 merged commit 840f3b6 into apache:master Nov 20, 2023
superdiaodiao pushed a commit to superdiaodiao/doris that referenced this pull request Nov 21, 2023
@Gabriel39 @superdiaodiao
[pipelineX](dependency) Wake by task by read dependency (apache#27260)
77786a9
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
@Gabriel39 @seawinde
[pipelineX](dependency) Wake by task by read dependency (apache#27260)
72b5b1a
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@Gabriel39
[pipelineX](dependency) Wake by task by read dependency (apache#27260)
81c9c4d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@Mryange Mryange Mryange approved these changes

@HappenLee HappenLee HappenLee left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Gabriel39 @doris-robot @yiguolei @HappenLee @Mryange