Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](spill) disable partitioned agg when group by limit opt is set #37873

Merged
merged 1 commit into from
Jul 18, 2024

Conversation

mrhhsg
Copy link
Member

@mrhhsg mrhhsg commented Jul 16, 2024

Proposed changes

Disable partitioned agg when group by limit opt is set(#29641)

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mrhhsg
Copy link
Member Author

mrhhsg commented Jul 16, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@mrhhsg
Copy link
Member Author

mrhhsg commented Jul 16, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39683 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa3348f79dd3e51bc9d36127e6c6ed2c918e95e9, data reload: false

------ Round 1 ----------------------------------
q1	17627	4333	4261	4261
q2	2024	193	190	190
q3	10448	1163	1054	1054
q4	10191	717	780	717
q5	7513	2686	2645	2645
q6	223	142	141	141
q7	956	598	595	595
q8	9206	2068	2067	2067
q9	8862	6505	6528	6505
q10	8917	3763	3739	3739
q11	457	241	235	235
q12	462	228	227	227
q13	17765	2940	2979	2940
q14	287	231	242	231
q15	527	469	491	469
q16	516	374	377	374
q17	961	663	742	663
q18	7977	7438	7335	7335
q19	6475	1484	1430	1430
q20	680	331	326	326
q21	4855	3291	3260	3260
q22	341	288	279	279
Total cold run time: 117270 ms
Total hot run time: 39683 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4498	4239	4230	4230
q2	388	279	258	258
q3	2985	2814	2890	2814
q4	1999	1753	1674	1674
q5	5698	5516	5539	5516
q6	223	138	138	138
q7	2205	1849	1879	1849
q8	3258	3419	3421	3419
q9	8827	8859	8875	8859
q10	3986	3960	3793	3793
q11	562	504	504	504
q12	826	651	626	626
q13	16218	3213	3137	3137
q14	318	286	289	286
q15	519	500	487	487
q16	493	443	438	438
q17	1820	1534	1504	1504
q18	8070	7968	7777	7777
q19	1742	1580	1606	1580
q20	2163	1860	1850	1850
q21	9115	4899	4793	4793
q22	621	535	501	501
Total cold run time: 76534 ms
Total hot run time: 56033 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173649 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa3348f79dd3e51bc9d36127e6c6ed2c918e95e9, data reload: false

query1	913	371	372	371
query2	6440	1869	1815	1815
query3	6638	200	212	200
query4	28255	17536	17214	17214
query5	3596	474	486	474
query6	267	156	159	156
query7	4577	287	286	286
query8	240	210	204	204
query9	8570	2415	2424	2415
query10	437	289	263	263
query11	12277	10111	10042	10042
query12	116	89	82	82
query13	1645	391	363	363
query14	10177	7480	7230	7230
query15	246	166	168	166
query16	7752	320	317	317
query17	1774	565	531	531
query18	1967	288	285	285
query19	194	154	153	153
query20	90	82	84	82
query21	208	131	129	129
query22	4345	4090	4108	4090
query23	34110	33832	33916	33832
query24	10883	2957	2904	2904
query25	652	422	409	409
query26	802	169	153	153
query27	2275	289	289	289
query28	6795	2091	2083	2083
query29	928	671	646	646
query30	264	160	160	160
query31	998	794	777	777
query32	103	54	61	54
query33	781	306	311	306
query34	997	486	504	486
query35	728	591	595	591
query36	1097	1001	1003	1001
query37	151	91	85	85
query38	2971	2854	2859	2854
query39	919	872	856	856
query40	229	126	122	122
query41	47	47	49	47
query42	114	103	101	101
query43	522	492	446	446
query44	1223	728	753	728
query45	206	166	163	163
query46	1083	804	717	717
query47	1869	1782	1790	1782
query48	385	305	291	291
query49	845	418	414	414
query50	768	375	399	375
query51	6854	6823	6863	6823
query52	108	90	97	90
query53	356	292	294	292
query54	866	445	452	445
query55	75	75	71	71
query56	287	266	260	260
query57	1148	1073	1085	1073
query58	230	266	269	266
query59	2892	2554	2683	2554
query60	309	286	278	278
query61	93	93	93	93
query62	806	628	642	628
query63	326	283	280	280
query64	9130	2201	1651	1651
query65	3198	3096	3126	3096
query66	720	337	327	327
query67	15606	15013	14996	14996
query68	8774	559	559	559
query69	739	496	362	362
query70	1350	1196	1157	1157
query71	528	302	274	274
query72	9205	5530	5475	5475
query73	2185	328	329	328
query74	6056	5673	5682	5673
query75	5179	2691	2698	2691
query76	5391	952	920	920
query77	781	292	306	292
query78	9835	9046	8951	8951
query79	8648	536	528	528
query80	1087	477	478	477
query81	574	219	226	219
query82	325	132	137	132
query83	329	161	164	161
query84	271	84	87	84
query85	932	312	296	296
query86	357	312	334	312
query87	3353	3124	3174	3124
query88	4703	2389	2383	2383
query89	514	381	386	381
query90	2025	187	190	187
query91	129	99	100	99
query92	62	50	50	50
query93	6078	524	520	520
query94	1366	219	212	212
query95	402	315	325	315
query96	614	269	266	266
query97	3194	3005	3081	3005
query98	210	199	194	194
query99	1503	1259	1288	1259
Total cold run time: 303172 ms
Total hot run time: 173649 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.45 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa3348f79dd3e51bc9d36127e6c6ed2c918e95e9, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.68	0.07	0.06
query5	0.48	0.54	0.47
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.48
query10	0.53	0.54	0.54
query11	0.16	0.12	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.77	0.79	0.77
query15	0.85	0.83	0.82
query16	0.36	0.37	0.36
query17	0.95	0.95	1.03
query18	0.23	0.22	0.22
query19	1.74	1.76	1.67
query20	0.01	0.01	0.01
query21	15.39	0.71	0.65
query22	4.31	7.24	1.91
query23	18.31	1.39	1.29
query24	2.11	0.23	0.22
query25	0.14	0.08	0.08
query26	0.30	0.22	0.21
query27	0.45	0.22	0.22
query28	13.26	1.02	1.01
query29	12.65	3.32	3.28
query30	0.25	0.06	0.05
query31	2.90	0.39	0.39
query32	3.25	0.48	0.47
query33	2.87	2.83	3.05
query34	17.18	4.33	4.41
query35	4.40	4.39	4.42
query36	0.66	0.48	0.48
query37	0.19	0.17	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.04
query40	0.14	0.13	0.12
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.73 s
Total hot run time: 30.45 s

@mrhhsg mrhhsg changed the title [fix](spill) disable limit opt in partitioned agg operator [fix](spill) disable partitioned agg when group by limit opt is set Jul 17, 2024
@mrhhsg
Copy link
Member Author

mrhhsg commented Jul 17, 2024

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39825 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f26f8cf948a78d9e7c9a76d1d30c851f87ecb93c, data reload: false

------ Round 1 ----------------------------------
q1	17620	4346	4255	4255
q2	2011	189	185	185
q3	10465	1153	1070	1070
q4	10194	780	831	780
q5	7542	2686	2743	2686
q6	219	140	139	139
q7	947	603	610	603
q8	9217	2075	2051	2051
q9	8755	6542	6511	6511
q10	8858	3740	3725	3725
q11	448	235	235	235
q12	407	228	233	228
q13	17765	2987	2990	2987
q14	286	245	243	243
q15	537	485	497	485
q16	504	382	379	379
q17	957	679	695	679
q18	8043	7438	7339	7339
q19	7302	1421	1490	1421
q20	654	317	330	317
q21	4873	3218	3221	3218
q22	352	289	294	289
Total cold run time: 117956 ms
Total hot run time: 39825 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4442	4255	4291	4255
q2	364	277	265	265
q3	3046	2909	2855	2855
q4	1983	1726	1735	1726
q5	5667	5547	5502	5502
q6	240	137	138	137
q7	2212	1884	1856	1856
q8	3276	3407	3414	3407
q9	8772	8848	8780	8780
q10	4044	3825	3841	3825
q11	615	527	535	527
q12	805	666	644	644
q13	17076	3172	3230	3172
q14	322	284	293	284
q15	521	488	487	487
q16	503	452	462	452
q17	1841	1532	1515	1515
q18	8167	7825	7799	7799
q19	1810	1674	1477	1477
q20	2931	1883	1872	1872
q21	7782	4906	4725	4725
q22	606	520	506	506
Total cold run time: 77025 ms
Total hot run time: 56068 ms

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 17, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-DS: Total hot run time: 173977 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f26f8cf948a78d9e7c9a76d1d30c851f87ecb93c, data reload: false

query1	939	377	365	365
query2	6460	1870	1857	1857
query3	6657	216	225	216
query4	23238	17470	17410	17410
query5	3587	488	472	472
query6	263	168	160	160
query7	4576	287	286	286
query8	235	191	191	191
query9	8516	2395	2395	2395
query10	442	301	279	279
query11	10733	9999	10015	9999
query12	117	86	86	86
query13	1636	372	358	358
query14	10203	8076	7936	7936
query15	215	166	175	166
query16	7355	306	311	306
query17	1812	545	531	531
query18	1922	270	269	269
query19	191	144	149	144
query20	88	79	82	79
query21	201	129	129	129
query22	4363	4032	3960	3960
query23	34120	33857	33537	33537
query24	7500	2933	2909	2909
query25	591	399	386	386
query26	702	155	152	152
query27	2130	276	276	276
query28	5712	2045	2056	2045
query29	904	643	622	622
query30	253	157	154	154
query31	969	780	767	767
query32	97	55	56	55
query33	574	298	290	290
query34	906	494	514	494
query35	684	569	582	569
query36	1141	968	1002	968
query37	146	90	84	84
query38	2946	2892	2860	2860
query39	920	853	872	853
query40	203	118	119	118
query41	44	42	47	42
query42	111	101	95	95
query43	504	476	476	476
query44	1123	726	729	726
query45	193	168	157	157
query46	1074	731	722	722
query47	1810	1733	1766	1733
query48	351	292	290	290
query49	823	404	422	404
query50	778	383	383	383
query51	6869	6816	6809	6809
query52	117	95	90	90
query53	350	281	276	276
query54	576	442	441	441
query55	75	77	75	75
query56	305	270	280	270
query57	1153	1031	1046	1031
query58	258	249	263	249
query59	2830	2640	2716	2640
query60	303	276	278	276
query61	97	93	94	93
query62	774	665	650	650
query63	312	279	282	279
query64	9155	2311	1738	1738
query65	3232	3134	3172	3134
query66	731	331	332	331
query67	16196	15135	14702	14702
query68	4510	539	544	539
query69	515	397	361	361
query70	1198	1089	1164	1089
query71	368	285	275	275
query72	7340	6004	5863	5863
query73	743	326	330	326
query74	6062	5657	5715	5657
query75	3360	2682	2693	2682
query76	2193	932	900	900
query77	440	302	315	302
query78	9576	8957	8966	8957
query79	2010	515	527	515
query80	1908	480	532	480
query81	594	218	227	218
query82	1016	133	127	127
query83	275	170	169	169
query84	260	89	85	85
query85	1157	320	305	305
query86	458	329	332	329
query87	3297	3118	3068	3068
query88	3333	2396	2381	2381
query89	478	378	387	378
query90	1696	198	194	194
query91	130	102	105	102
query92	59	50	49	49
query93	2090	511	499	499
query94	1126	215	210	210
query95	415	326	324	324
query96	598	273	272	272
query97	3176	3046	3014	3014
query98	230	196	202	196
query99	1772	1266	1331	1266
Total cold run time: 266774 ms
Total hot run time: 173977 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.87 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f26f8cf948a78d9e7c9a76d1d30c851f87ecb93c, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.22	0.05	0.05
query4	1.69	0.08	0.07
query5	0.50	0.49	0.48
query6	1.14	0.72	0.73
query7	0.01	0.02	0.01
query8	0.06	0.05	0.05
query9	0.56	0.50	0.50
query10	0.55	0.55	0.54
query11	0.15	0.12	0.11
query12	0.14	0.13	0.13
query13	0.59	0.59	0.58
query14	0.75	0.79	0.78
query15	0.86	0.81	0.82
query16	0.37	0.35	0.36
query17	0.97	1.00	1.04
query18	0.22	0.21	0.22
query19	1.82	1.78	1.77
query20	0.01	0.01	0.01
query21	15.42	0.76	0.66
query22	4.51	7.25	2.01
query23	18.31	1.48	1.36
query24	2.13	0.23	0.23
query25	0.17	0.09	0.09
query26	0.30	0.21	0.21
query27	0.45	0.23	0.23
query28	13.25	1.03	1.02
query29	12.66	3.31	3.28
query30	0.25	0.06	0.05
query31	2.88	0.39	0.40
query32	3.26	0.47	0.46
query33	2.87	2.90	2.86
query34	17.12	4.32	4.34
query35	4.40	4.45	4.40
query36	0.65	0.47	0.47
query37	0.19	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.09 s
Total hot run time: 30.87 s

@mrhhsg
Copy link
Member Author

mrhhsg commented Jul 17, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40050 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f26f8cf948a78d9e7c9a76d1d30c851f87ecb93c, data reload: false

------ Round 1 ----------------------------------
q1	17603	4843	4303	4303
q2	2022	196	183	183
q3	10455	1204	1157	1157
q4	10193	809	822	809
q5	7547	2702	2711	2702
q6	223	138	136	136
q7	975	595	597	595
q8	9228	2070	2091	2070
q9	8861	6578	6557	6557
q10	8869	3790	3787	3787
q11	448	239	233	233
q12	405	226	224	224
q13	17759	2974	2994	2974
q14	273	235	230	230
q15	534	476	492	476
q16	503	384	380	380
q17	954	622	722	622
q18	8123	7521	7380	7380
q19	7485	1486	1409	1409
q20	667	324	351	324
q21	4910	3213	3222	3213
q22	345	290	286	286
Total cold run time: 118382 ms
Total hot run time: 40050 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4406	4255	4269	4255
q2	375	273	258	258
q3	3013	2953	2942	2942
q4	1956	1688	1751	1688
q5	5586	5569	5452	5452
q6	234	137	131	131
q7	2236	1845	1803	1803
q8	3223	3378	3407	3378
q9	8733	8721	8891	8721
q10	4091	3896	3754	3754
q11	627	491	503	491
q12	805	643	635	635
q13	15977	3174	3168	3168
q14	305	279	308	279
q15	528	490	477	477
q16	482	443	445	443
q17	1812	1527	1539	1527
q18	8128	7853	7795	7795
q19	1726	1548	1557	1548
q20	2170	1850	1849	1849
q21	8230	4927	4610	4610
q22	557	525	512	512
Total cold run time: 75200 ms
Total hot run time: 55716 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172597 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f26f8cf948a78d9e7c9a76d1d30c851f87ecb93c, data reload: false

query1	912	372	368	368
query2	6445	1906	1868	1868
query3	6636	206	211	206
query4	23347	17540	17233	17233
query5	3692	482	455	455
query6	269	169	164	164
query7	4583	283	282	282
query8	240	201	201	201
query9	8611	2437	2430	2430
query10	459	289	269	269
query11	12036	10163	10034	10034
query12	119	85	85	85
query13	1650	360	375	360
query14	10163	7696	7545	7545
query15	224	165	168	165
query16	7866	318	321	318
query17	1803	573	536	536
query18	2027	279	289	279
query19	208	151	161	151
query20	92	82	89	82
query21	210	128	128	128
query22	4229	4116	4178	4116
query23	33993	33746	33513	33513
query24	9148	2993	2905	2905
query25	607	427	405	405
query26	708	160	155	155
query27	2113	336	274	274
query28	6035	2082	2078	2078
query29	891	677	643	643
query30	261	157	157	157
query31	976	755	760	755
query32	97	56	54	54
query33	635	314	306	306
query34	882	486	492	486
query35	698	594	569	569
query36	1147	1004	992	992
query37	146	90	81	81
query38	2925	2894	2827	2827
query39	919	853	819	819
query40	202	121	117	117
query41	49	43	42	42
query42	117	97	101	97
query43	511	464	483	464
query44	1077	723	752	723
query45	196	162	158	158
query46	1087	727	735	727
query47	1864	1765	1768	1765
query48	373	295	285	285
query49	847	401	410	401
query50	782	384	381	381
query51	6859	6946	6745	6745
query52	108	102	92	92
query53	353	284	284	284
query54	878	440	438	438
query55	74	75	72	72
query56	289	260	271	260
query57	1131	1070	1037	1037
query58	254	245	260	245
query59	3060	2543	2535	2535
query60	296	278	267	267
query61	99	97	97	97
query62	777	640	652	640
query63	318	285	287	285
query64	9149	2203	1671	1671
query65	3172	3095	3108	3095
query66	738	316	339	316
query67	15587	14937	14894	14894
query68	5508	525	534	525
query69	745	477	328	328
query70	1214	1111	1062	1062
query71	446	302	297	297
query72	8716	5249	5030	5030
query73	780	323	317	317
query74	6176	5678	5732	5678
query75	4436	2670	2664	2664
query76	3442	900	904	900
query77	710	303	305	303
query78	11522	9645	8888	8888
query79	7768	519	521	519
query80	982	477	470	470
query81	574	224	218	218
query82	765	138	139	138
query83	318	171	167	167
query84	275	84	89	84
query85	792	332	380	332
query86	358	310	285	285
query87	3289	3106	3106	3106
query88	5307	2390	2365	2365
query89	495	386	378	378
query90	1757	191	186	186
query91	127	101	103	101
query92	61	53	50	50
query93	6059	512	494	494
query94	808	221	214	214
query95	394	318	304	304
query96	617	275	268	268
query97	3237	3039	3073	3039
query98	210	200	193	193
query99	1555	1267	1270	1267
Total cold run time: 287692 ms
Total hot run time: 172597 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.27 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f26f8cf948a78d9e7c9a76d1d30c851f87ecb93c, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.66	0.08	0.10
query5	0.51	0.49	0.49
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.53	0.53	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.58	0.59	0.59
query14	0.75	0.77	0.78
query15	0.86	0.80	0.82
query16	0.37	0.35	0.34
query17	0.94	1.00	0.95
query18	0.22	0.21	0.22
query19	1.76	1.67	1.69
query20	0.02	0.01	0.01
query21	15.38	0.75	0.67
query22	4.25	7.76	1.72
query23	18.28	1.30	1.23
query24	2.18	0.24	0.21
query25	0.15	0.10	0.08
query26	0.28	0.21	0.20
query27	0.46	0.22	0.23
query28	13.23	1.02	0.99
query29	12.63	3.32	3.32
query30	0.25	0.06	0.05
query31	2.88	0.39	0.39
query32	3.27	0.48	0.49
query33	2.90	2.91	2.91
query34	17.11	4.33	4.35
query35	4.42	4.37	4.40
query36	0.66	0.46	0.47
query37	0.18	0.16	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.15	0.14	0.12
query41	0.10	0.05	0.04
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 109.65 s
Total hot run time: 30.27 s

@yiguolei yiguolei merged commit b6cd230 into apache:master Jul 18, 2024
28 of 31 checks passed
@mrhhsg mrhhsg deleted the fix_agg_limit branch July 18, 2024 02:33
dataroaring pushed a commit that referenced this pull request Jul 19, 2024
…37873)

## Proposed changes

Disable partitioned agg when group by limit opt is set(#29641)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants