Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](cloud) Fix constant folding for requests sent to BE without dis… #42865

Merged
merged 4 commits into from
Nov 1, 2024

Conversation

deardeng
Copy link
Contributor

…tinguishing clusters

Proposed changes

Issue Number: close #xxx

replace function getAllBackendIds by getBackendsByCurrentCluster

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@deardeng
Copy link
Contributor Author

run buildall

@deardeng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41040 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f35612f797fd16bb6190de0e1cafebd2722ad9e0, data reload: false

------ Round 1 ----------------------------------
q1	17579	7432	7307	7307
q2	2044	163	172	163
q3	10556	1148	1206	1148
q4	10225	830	814	814
q5	7752	3088	3020	3020
q6	235	151	144	144
q7	1009	609	601	601
q8	9375	1928	2004	1928
q9	6558	6423	6454	6423
q10	7075	2449	2414	2414
q11	436	247	244	244
q12	406	207	207	207
q13	17770	3033	2990	2990
q14	255	211	227	211
q15	566	519	524	519
q16	627	582	578	578
q17	971	491	554	491
q18	7189	6624	6706	6624
q19	1324	979	993	979
q20	470	184	179	179
q21	3954	3295	3079	3079
q22	1108	991	977	977
Total cold run time: 107484 ms
Total hot run time: 41040 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7349	7240	7251	7240
q2	329	233	227	227
q3	2893	2770	2848	2770
q4	1916	1745	1662	1662
q5	5429	5471	5562	5471
q6	218	138	137	137
q7	2110	1719	1707	1707
q8	3267	3399	3370	3370
q9	8495	8539	8501	8501
q10	3493	3450	3425	3425
q11	581	499	476	476
q12	784	621	606	606
q13	8183	3048	3020	3020
q14	289	259	251	251
q15	576	505	515	505
q16	676	641	639	639
q17	1817	1595	1546	1546
q18	7849	7524	7505	7505
q19	1676	1581	1428	1428
q20	2069	1826	1838	1826
q21	5310	5221	5330	5221
q22	1068	1005	989	989
Total cold run time: 66377 ms
Total hot run time: 58522 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191737 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f35612f797fd16bb6190de0e1cafebd2722ad9e0, data reload: false

query1	962	378	399	378
query2	6529	2018	2111	2018
query3	6785	211	219	211
query4	34231	23531	23481	23481
query5	4357	461	452	452
query6	275	181	178	178
query7	4595	298	307	298
query8	291	239	231	231
query9	9503	2732	2723	2723
query10	476	252	250	250
query11	18075	15439	15195	15195
query12	170	103	101	101
query13	1668	447	413	413
query14	10394	7148	6831	6831
query15	262	173	188	173
query16	7963	451	447	447
query17	1615	568	531	531
query18	2121	290	281	281
query19	368	142	148	142
query20	123	110	106	106
query21	206	100	102	100
query22	4520	4433	4226	4226
query23	34816	34105	34142	34105
query24	11131	2805	2776	2776
query25	657	369	388	369
query26	1199	155	153	153
query27	2771	291	291	291
query28	8116	2431	2431	2431
query29	858	409	406	406
query30	337	169	159	159
query31	1008	789	801	789
query32	97	54	56	54
query33	762	258	266	258
query34	1004	503	531	503
query35	1056	890	862	862
query36	1092	943	975	943
query37	133	75	69	69
query38	4525	4259	4382	4259
query39	1482	1426	1409	1409
query40	204	96	98	96
query41	47	43	43	43
query42	112	110	94	94
query43	534	478	473	473
query44	1280	798	812	798
query45	181	166	162	162
query46	1122	704	719	704
query47	1954	1837	1854	1837
query48	430	327	334	327
query49	1161	407	413	407
query50	834	390	395	390
query51	7132	6992	7092	6992
query52	103	91	88	88
query53	263	184	180	180
query54	1086	424	406	406
query55	82	79	88	79
query56	285	247	242	242
query57	1332	1157	1191	1157
query58	245	207	210	207
query59	3325	3088	2929	2929
query60	300	261	253	253
query61	127	122	117	117
query62	860	680	682	680
query63	215	192	190	190
query64	4569	738	732	732
query65	3326	3249	3242	3242
query66	1463	335	312	312
query67	16078	15589	15771	15589
query68	4470	563	557	557
query69	427	269	263	263
query70	1244	1178	1194	1178
query71	332	265	263	263
query72	6492	4049	4031	4031
query73	778	359	360	359
query74	10313	9234	9003	9003
query75	3464	2672	2685	2672
query76	2701	1030	1034	1030
query77	386	274	290	274
query78	10487	9652	9486	9486
query79	2525	604	611	604
query80	1158	422	420	420
query81	574	250	244	244
query82	916	117	112	112
query83	212	142	139	139
query84	235	75	70	70
query85	1256	312	296	296
query86	432	305	299	299
query87	4860	4762	4775	4762
query88	4195	2210	2180	2180
query89	396	295	285	285
query90	2054	188	183	183
query91	142	102	104	102
query92	59	47	48	47
query93	1627	568	546	546
query94	981	296	296	296
query95	347	245	249	245
query96	616	278	291	278
query97	2924	2685	2677	2677
query98	224	197	191	191
query99	1531	1317	1361	1317
Total cold run time: 303541 ms
Total hot run time: 191737 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f35612f797fd16bb6190de0e1cafebd2722ad9e0, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.10
query5	0.42	0.41	0.40
query6	1.19	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.05	0.04
query9	0.57	0.51	0.52
query10	0.54	0.55	0.55
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.60	0.58	0.60
query14	2.72	2.73	2.73
query15	0.89	0.82	0.82
query16	0.38	0.41	0.40
query17	1.01	1.04	1.00
query18	0.24	0.22	0.21
query19	1.96	1.87	1.96
query20	0.02	0.02	0.01
query21	15.36	0.58	0.58
query22	2.52	1.96	1.67
query23	17.06	0.96	0.87
query24	2.94	1.87	0.46
query25	0.23	0.26	0.08
query26	0.47	0.14	0.14
query27	0.04	0.03	0.04
query28	10.56	1.11	1.06
query29	12.59	3.28	3.31
query30	0.25	0.06	0.06
query31	2.87	0.39	0.38
query32	3.28	0.46	0.46
query33	3.00	3.05	3.04
query34	16.89	4.44	4.40
query35	4.48	4.47	4.47
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.03
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106.49 s
Total hot run time: 31.97 s

@deardeng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41429 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1237a87f6b940ad787ee55c362c792316c524eba, data reload: false

------ Round 1 ----------------------------------
q1	17564	7435	7343	7343
q2	2039	171	153	153
q3	10573	1136	1214	1136
q4	10230	802	897	802
q5	7716	3156	3120	3120
q6	236	148	150	148
q7	1033	611	610	610
q8	9374	1999	1987	1987
q9	6630	6450	6468	6450
q10	7029	2418	2470	2418
q11	436	247	253	247
q12	406	222	209	209
q13	17793	3052	3035	3035
q14	245	215	219	215
q15	558	517	525	517
q16	646	589	586	586
q17	984	538	569	538
q18	7507	6684	6763	6684
q19	1325	979	1067	979
q20	466	183	177	177
q21	4015	3306	3071	3071
q22	1131	1004	1021	1004
Total cold run time: 107936 ms
Total hot run time: 41429 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7344	7281	7261	7261
q2	345	236	238	236
q3	2966	2815	2829	2815
q4	1963	1771	1735	1735
q5	5425	5539	5534	5534
q6	220	138	139	138
q7	2139	1785	1723	1723
q8	3252	3395	3442	3395
q9	8626	8656	8676	8656
q10	3504	3489	3444	3444
q11	580	489	486	486
q12	810	582	577	577
q13	10195	3007	3042	3007
q14	303	262	278	262
q15	579	513	516	513
q16	692	634	650	634
q17	1843	1608	1587	1587
q18	7937	7538	7554	7538
q19	1681	1605	1631	1605
q20	2050	1809	1820	1809
q21	5290	5129	5283	5129
q22	1112	1016	1008	1008
Total cold run time: 68856 ms
Total hot run time: 59092 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192445 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1237a87f6b940ad787ee55c362c792316c524eba, data reload: false

query1	974	372	384	372
query2	6528	2128	2037	2037
query3	6785	214	219	214
query4	34095	23535	23632	23535
query5	4317	446	438	438
query6	263	168	169	168
query7	4599	295	297	295
query8	288	243	232	232
query9	9381	2688	2681	2681
query10	476	252	251	251
query11	18140	15164	15300	15164
query12	162	101	100	100
query13	1663	431	396	396
query14	9904	7439	7484	7439
query15	291	177	177	177
query16	8140	456	467	456
query17	1715	572	573	572
query18	2148	297	304	297
query19	359	153	154	153
query20	117	109	106	106
query21	208	101	100	100
query22	4440	4511	4246	4246
query23	35629	34230	34344	34230
query24	11087	2780	2804	2780
query25	587	416	412	412
query26	1134	161	156	156
query27	2781	287	289	287
query28	7883	2432	2449	2432
query29	665	432	435	432
query30	326	171	167	167
query31	1031	791	819	791
query32	101	56	60	56
query33	777	290	280	280
query34	942	514	531	514
query35	1037	906	874	874
query36	1107	943	957	943
query37	122	80	77	77
query38	4400	4264	4270	4264
query39	1464	1429	1443	1429
query40	281	105	103	103
query41	50	48	47	47
query42	107	100	102	100
query43	527	500	495	495
query44	1229	816	811	811
query45	186	171	170	170
query46	1134	687	683	683
query47	1924	1822	1843	1822
query48	441	334	340	334
query49	1260	460	405	405
query50	811	385	403	385
query51	7118	7026	7086	7026
query52	102	94	91	91
query53	253	179	183	179
query54	1277	422	430	422
query55	81	75	83	75
query56	267	254	248	248
query57	1347	1180	1155	1155
query58	228	202	202	202
query59	3184	3040	2818	2818
query60	276	246	244	244
query61	106	105	109	105
query62	861	678	684	678
query63	219	183	182	182
query64	5209	643	607	607
query65	3291	3257	3207	3207
query66	1435	299	344	299
query67	16471	15857	15804	15804
query68	5068	563	552	552
query69	432	258	255	255
query70	1190	1145	1038	1038
query71	431	257	252	252
query72	6246	4076	4011	4011
query73	776	362	370	362
query74	10402	9033	9030	9030
query75	3415	2686	2651	2651
query76	2949	1037	949	949
query77	396	280	270	270
query78	10578	9580	9751	9580
query79	1203	592	602	592
query80	922	432	440	432
query81	535	245	241	241
query82	1174	114	118	114
query83	216	138	138	138
query84	234	71	69	69
query85	1288	305	296	296
query86	365	300	294	294
query87	4722	4691	4745	4691
query88	3111	2227	2199	2199
query89	393	289	286	286
query90	2029	190	182	182
query91	133	100	99	99
query92	57	48	53	48
query93	1047	538	529	529
query94	969	294	296	294
query95	350	245	253	245
query96	608	277	285	277
query97	2875	2718	2746	2718
query98	213	199	195	195
query99	1550	1307	1306	1306
Total cold run time: 301618 ms
Total hot run time: 192445 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.01 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1237a87f6b940ad787ee55c362c792316c524eba, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.04
query3	0.23	0.07	0.07
query4	1.65	0.10	0.10
query5	0.42	0.39	0.41
query6	1.17	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.56	0.51	0.49
query10	0.56	0.55	0.56
query11	0.15	0.12	0.11
query12	0.16	0.10	0.10
query13	0.60	0.60	0.60
query14	2.82	2.86	2.71
query15	0.89	0.84	0.83
query16	0.38	0.36	0.39
query17	1.03	1.05	0.97
query18	0.24	0.21	0.22
query19	1.85	1.81	1.93
query20	0.01	0.01	0.01
query21	15.37	0.59	0.60
query22	2.34	2.36	2.41
query23	16.85	0.83	0.86
query24	3.75	0.94	1.80
query25	0.32	0.17	0.15
query26	0.36	0.13	0.15
query27	0.04	0.04	0.06
query28	10.04	1.09	1.07
query29	12.62	3.32	3.27
query30	0.25	0.06	0.06
query31	2.88	0.37	0.38
query32	3.28	0.48	0.45
query33	2.99	3.00	3.04
query34	17.03	4.44	4.50
query35	4.49	4.47	4.54
query36	0.64	0.48	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.13	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.58 s
Total hot run time: 33.01 s

@deardeng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41362 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c17764401e04c9a0af38280c56baa999adfce0c4, data reload: false

------ Round 1 ----------------------------------
q1	17587	7435	7287	7287
q2	2050	170	155	155
q3	10616	1096	1196	1096
q4	10574	843	910	843
q5	7733	3037	3089	3037
q6	232	143	148	143
q7	1045	616	608	608
q8	9357	1960	1985	1960
q9	6657	6426	6458	6426
q10	7059	2457	2409	2409
q11	455	250	245	245
q12	400	214	202	202
q13	17792	3006	3063	3006
q14	238	209	223	209
q15	572	512	508	508
q16	653	587	582	582
q17	970	602	562	562
q18	7165	6741	6730	6730
q19	1327	936	1087	936
q20	461	178	177	177
q21	3937	3213	3268	3213
q22	1124	1032	1028	1028
Total cold run time: 108004 ms
Total hot run time: 41362 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7283	7933	7268	7268
q2	316	228	221	221
q3	3046	2985	2978	2978
q4	2123	1861	1804	1804
q5	5682	5783	5782	5782
q6	229	137	139	137
q7	2251	1826	1786	1786
q8	3369	3501	3457	3457
q9	8890	8857	8874	8857
q10	3581	3546	3584	3546
q11	592	510	481	481
q12	819	618	593	593
q13	11092	3178	3211	3178
q14	311	302	277	277
q15	574	529	524	524
q16	693	632	675	632
q17	1828	1641	1616	1616
q18	8385	7880	7622	7622
q19	1680	1546	1536	1536
q20	2081	1875	1865	1865
q21	5607	5399	5503	5399
q22	1181	1109	1039	1039
Total cold run time: 71613 ms
Total hot run time: 60598 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197282 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c17764401e04c9a0af38280c56baa999adfce0c4, data reload: false

query1	1248	991	967	967
query2	6252	2074	2030	2030
query3	11625	4659	4919	4659
query4	34015	23587	23710	23587
query5	4900	451	434	434
query6	285	173	171	171
query7	3975	291	302	291
query8	273	221	234	221
query9	9585	2718	2707	2707
query10	490	246	250	246
query11	18133	15193	15515	15193
query12	150	105	99	99
query13	1572	424	398	398
query14	9398	7043	7778	7043
query15	262	188	190	188
query16	8088	476	502	476
query17	1570	607	583	583
query18	2139	309	300	300
query19	364	154	154	154
query20	123	110	118	110
query21	203	111	115	111
query22	4678	4483	4463	4463
query23	34958	34310	34306	34306
query24	11063	2825	2836	2825
query25	638	401	402	401
query26	1332	158	159	158
query27	2571	278	288	278
query28	7377	2442	2412	2412
query29	861	422	431	422
query30	252	163	161	161
query31	1000	782	812	782
query32	91	55	55	55
query33	784	275	268	268
query34	937	496	516	496
query35	1042	878	881	878
query36	1072	960	969	960
query37	131	77	67	67
query38	4389	4257	4284	4257
query39	1494	1431	1437	1431
query40	247	99	104	99
query41	46	50	45	45
query42	112	103	98	98
query43	537	494	487	487
query44	1245	810	821	810
query45	184	168	167	167
query46	1147	694	709	694
query47	1940	1840	1826	1826
query48	417	321	323	321
query49	901	412	412	412
query50	814	404	385	385
query51	7016	6979	7021	6979
query52	102	85	84	84
query53	260	179	179	179
query54	1095	402	402	402
query55	79	76	79	76
query56	279	242	223	223
query57	1269	1173	1133	1133
query58	230	200	200	200
query59	3151	2990	2952	2952
query60	277	248	247	247
query61	109	118	112	112
query62	843	668	678	668
query63	215	186	178	178
query64	4023	664	652	652
query65	3286	3213	3212	3212
query66	962	302	308	302
query67	16016	15747	15500	15500
query68	4509	569	565	565
query69	434	260	252	252
query70	1194	1180	1175	1175
query71	326	250	237	237
query72	6296	4045	4016	4016
query73	756	344	360	344
query74	10475	9114	9001	9001
query75	3419	2635	2737	2635
query76	2651	1106	1039	1039
query77	421	281	278	278
query78	10661	9572	9562	9562
query79	1140	605	602	602
query80	854	440	425	425
query81	539	246	240	240
query82	1321	117	108	108
query83	209	137	134	134
query84	240	70	71	70
query85	1119	319	304	304
query86	329	309	305	305
query87	4782	4651	4715	4651
query88	3466	2186	2155	2155
query89	415	296	298	296
query90	1918	192	187	187
query91	133	99	99	99
query92	59	48	48	48
query93	1210	540	540	540
query94	750	282	283	282
query95	346	246	245	245
query96	613	283	283	283
query97	2960	2698	2695	2695
query98	206	199	195	195
query99	1513	1283	1303	1283
Total cold run time: 301386 ms
Total hot run time: 197282 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.04 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c17764401e04c9a0af38280c56baa999adfce0c4, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.43	0.40	0.41
query6	1.17	0.66	0.68
query7	0.02	0.01	0.02
query8	0.04	0.03	0.02
query9	0.58	0.50	0.49
query10	0.56	0.54	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.62	0.59	0.60
query14	2.72	2.83	2.86
query15	0.89	0.82	0.83
query16	0.39	0.38	0.37
query17	1.05	1.05	1.06
query18	0.20	0.19	0.19
query19	1.83	1.79	1.81
query20	0.01	0.01	0.01
query21	15.35	0.60	0.59
query22	2.80	2.02	1.32
query23	17.02	0.97	0.83
query24	3.09	1.00	0.86
query25	0.28	0.14	0.08
query26	0.42	0.15	0.14
query27	0.04	0.04	0.04
query28	10.93	1.09	1.08
query29	12.58	3.28	3.24
query30	0.25	0.07	0.06
query31	2.86	0.38	0.39
query32	3.28	0.46	0.45
query33	2.96	3.02	3.02
query34	17.09	4.46	4.57
query35	4.60	4.52	4.59
query36	0.66	0.47	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.41 s
Total hot run time: 32.04 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 31, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@CalvinKirs CalvinKirs merged commit 7e90e66 into apache:master Nov 1, 2024
25 of 27 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 1, 2024
#42865)

…tinguishing clusters

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

replace function getAllBackendIds by getBackendsByCurrentCluster
dataroaring pushed a commit that referenced this pull request Nov 7, 2024
… without dis… (#43110)

PR Body: …tinguishing clusters

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->

replace function getAllBackendIds by getBackendsByCurrentCluster

 
 Cherry-picked from #42865

Co-authored-by: deardeng <565620795@qq.com>
@gavinchou gavinchou added the cir label Nov 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. cir dev/3.0.3-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants