Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](pipelineX) add mem control in local exchange sink #31982

Merged
merged 1 commit into from
Mar 11, 2024

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Mar 8, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@Mryange
Copy link
Contributor Author

Mryange commented Mar 8, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.37% (8556/24189)
Line Coverage: 27.15% (69525/256065)
Region Coverage: 26.36% (36073/136835)
Branch Coverage: 23.21% (18440/79446)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bcf36e6f2f672820a4f712e5b4ba18a8646b2b7b_bcf36e6f2f672820a4f712e5b4ba18a8646b2b7b/report/index.html

@Mryange
Copy link
Contributor Author

Mryange commented Mar 8, 2024

run p0 10

@Mryange
Copy link
Contributor Author

Mryange commented Mar 8, 2024

run p0

2 similar comments
@Mryange
Copy link
Contributor Author

Mryange commented Mar 9, 2024

run p0

@Mryange
Copy link
Contributor Author

Mryange commented Mar 9, 2024

run p0

@Mryange
Copy link
Contributor Author

Mryange commented Mar 10, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.36% (8556/24197)
Line Coverage: 27.14% (69529/256140)
Region Coverage: 26.37% (36082/136835)
Branch Coverage: 23.22% (18447/79440)
Coverage Report: http://coverage.selectdb-in.cc/coverage/18daa8402f2d1984965e0ed48a6eeae432f606d9_18daa8402f2d1984965e0ed48a6eeae432f606d9/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38245 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 18daa8402f2d1984965e0ed48a6eeae432f606d9, data reload: false

------ Round 1 ----------------------------------
q1	17630	4049	4009	4009
q2	2025	150	145	145
q3	10551	947	942	942
q4	4664	938	940	938
q5	7608	2870	3000	2870
q6	182	126	124	124
q7	1297	846	814	814
q8	9477	2064	2074	2064
q9	7311	6527	6507	6507
q10	8226	2594	2596	2594
q11	421	227	219	219
q12	750	320	317	317
q13	17976	2983	2967	2967
q14	276	256	260	256
q15	501	470	468	468
q16	470	395	411	395
q17	939	881	861	861
q18	6761	5943	6022	5943
q19	1556	1519	1510	1510
q20	549	291	277	277
q21	7577	3728	3732	3728
q22	837	297	297	297
Total cold run time: 107584 ms
Total hot run time: 38245 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4026	3975	3978	3975
q2	319	220	226	220
q3	2972	2907	2908	2907
q4	1860	1833	1829	1829
q5	5237	5247	5222	5222
q6	211	115	116	115
q7	2271	1829	1826	1826
q8	3202	3258	3262	3258
q9	8525	8518	8563	8518
q10	6122	3715	3674	3674
q11	530	443	438	438
q12	680	536	536	536
q13	13718	2833	2819	2819
q14	269	253	264	253
q15	486	447	450	447
q16	451	395	410	395
q17	1693	1661	1633	1633
q18	7771	7520	7294	7294
q19	1706	1623	1612	1612
q20	1881	1686	1700	1686
q21	5011	4776	4835	4776
q22	516	438	439	438
Total cold run time: 69457 ms
Total hot run time: 53871 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186198 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 18daa8402f2d1984965e0ed48a6eeae432f606d9, data reload: false

query1	922	361	339	339
query2	7297	1923	1990	1923
query3	6705	216	213	213
query4	24280	20909	20746	20746
query5	4325	434	434	434
query6	262	192	192	192
query7	4645	312	319	312
query8	230	174	168	168
query9	8411	2270	2271	2270
query10	443	231	235	231
query11	15034	14439	14480	14439
query12	133	88	85	85
query13	1635	430	454	430
query14	10776	9901	9632	9632
query15	240	196	200	196
query16	7845	282	284	282
query17	1874	599	569	569
query18	2073	292	284	284
query19	204	164	168	164
query20	97	94	90	90
query21	206	128	133	128
query22	4660	4485	4466	4466
query23	31389	30771	30880	30771
query24	11230	3029	3093	3029
query25	658	399	398	398
query26	938	162	170	162
query27	2812	362	364	362
query28	6262	1865	1863	1863
query29	1030	619	628	619
query30	312	151	151	151
query31	914	724	733	724
query32	107	60	60	60
query33	760	264	254	254
query34	1041	473	486	473
query35	834	633	639	633
query36	917	817	863	817
query37	113	75	77	75
query38	3379	3162	3201	3162
query39	1443	1377	1376	1376
query40	211	120	119	119
query41	62	59	56	56
query42	109	102	104	102
query43	424	414	401	401
query44	1096	717	714	714
query45	273	261	262	261
query46	1045	796	776	776
query47	1639	1562	1551	1551
query48	420	356	355	355
query49	1131	351	349	349
query50	789	381	379	379
query51	6845	6706	6691	6691
query52	105	99	93	93
query53	357	284	284	284
query54	288	268	252	252
query55	85	84	84	84
query56	247	243	244	243
query57	1076	1005	987	987
query58	250	223	221	221
query59	2476	2290	2367	2290
query60	282	257	311	257
query61	115	119	112	112
query62	662	416	433	416
query63	306	286	281	281
query64	5257	3023	3053	3023
query65	3006	3018	3012	3012
query66	831	324	321	321
query67	14938	14526	14358	14358
query68	6073	548	565	548
query69	603	391	382	382
query70	1190	1100	1157	1100
query71	504	277	266	266
query72	6449	2710	2551	2551
query73	754	326	320	320
query74	6899	6534	6642	6534
query75	9273	7955	7984	7955
query76	4292	1102	1152	1102
query77	625	262	273	262
query78	10440	9625	9656	9625
query79	7765	513	521	513
query80	1725	392	387	387
query81	547	214	218	214
query82	1402	203	195	195
query83	226	145	147	145
query84	283	81	76	76
query85	1572	369	329	329
query86	477	287	285	285
query87	3464	3282	3310	3282
query88	4214	2403	2342	2342
query89	547	355	364	355
query90	1956	176	172	172
query91	164	158	127	127
query92	63	55	47	47
query93	6437	548	527	527
query94	1197	196	191	191
query95	457	349	345	345
query96	611	265	269	265
query97	4400	4286	4274	4274
query98	235	214	219	214
query99	1144	762	758	758
Total cold run time: 296749 ms
Total hot run time: 186198 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.18 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 18daa8402f2d1984965e0ed48a6eeae432f606d9, data reload: false

query1	0.05	0.03	0.03
query2	0.06	0.02	0.03
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.50	0.50	0.49
query6	1.31	0.68	0.68
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.54	0.52	0.52
query10	0.57	0.57	0.58
query11	0.13	0.10	0.10
query12	0.12	0.10	0.10
query13	0.58	0.58	0.56
query14	0.74	0.75	0.76
query15	0.83	0.81	0.80
query16	0.37	0.39	0.36
query17	0.97	0.94	0.98
query18	0.28	0.26	0.26
query19	1.82	1.68	1.69
query20	0.01	0.01	0.01
query21	15.40	0.62	0.65
query22	2.89	4.00	3.65
query23	17.62	1.05	0.92
query24	2.11	0.60	0.58
query25	0.24	0.04	0.09
query26	0.19	0.14	0.14
query27	0.03	0.03	0.04
query28	11.96	0.83	0.85
query29	12.71	3.32	3.46
query30	0.65	0.53	0.59
query31	2.79	0.33	0.35
query32	3.40	0.43	0.44
query33	2.84	2.78	2.78
query34	15.50	4.31	4.29
query35	4.30	4.33	4.32
query36	1.08	1.01	1.01
query37	0.07	0.05	0.05
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.18	0.14	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.02	0.02	0.03
Total cold run time: 105 s
Total hot run time: 32.18 s

@Mryange
Copy link
Contributor Author

Mryange commented Mar 10, 2024

run p0

2 similar comments
@Mryange
Copy link
Contributor Author

Mryange commented Mar 10, 2024

run p0

@Mryange
Copy link
Contributor Author

Mryange commented Mar 10, 2024

run p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 11, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HappenLee HappenLee merged commit 1303c9b into apache:master Mar 11, 2024
24 of 29 checks passed
Gabriel39 pushed a commit to Gabriel39/incubator-doris that referenced this pull request Mar 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants