Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](merge-cloud) Fix cloud cluster checker cant get backends #30467

Merged
merged 4 commits into from
Jan 29, 2024
Merged

[feature](merge-cloud) Fix cloud cluster checker cant get backends #30467

merged 4 commits into from
Jan 29, 2024

Conversation

deardeng
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@deardeng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37569 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 436ec0052a5670b303faaea962af9aaba174047d, data reload: false

------ Round 1 ----------------------------------
q1	17683	4945	5038	4945
q2	2050	137	127	127
q3	10725	919	931	919
q4	4715	710	717	710
q5	7683	2775	2878	2775
q6	184	119	118	118
q7	1114	725	718	718
q8	9356	1997	2030	1997
q9	7215	6312	6344	6312
q10	8111	2413	2436	2413
q11	402	209	188	188
q12	765	267	271	267
q13	17988	3331	3276	3276
q14	279	240	249	240
q15	538	492	483	483
q16	470	397	396	396
q17	931	525	508	508
q18	6807	6144	6084	6084
q19	1553	1362	1378	1362
q20	577	329	316	316
q21	6971	3144	3102	3102
q22	823	314	313	313
Total cold run time: 106940 ms
Total hot run time: 37569 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5214	5210	4922	4922
q2	357	245	238	238
q3	2990	2717	2725	2717
q4	1778	1678	1599	1599
q5	5381	5142	5258	5142
q6	202	116	115	115
q7	2126	1754	1784	1754
q8	3077	3173	3159	3159
q9	8728	8605	15366	8605
q10	5703	3532	3399	3399
q11	558	466	470	466
q12	760	614	578	578
q13	16973	3076	3056	3056
q14	278	261	247	247
q15	533	488	488	488
q16	518	477	470	470
q17	2007	1621	1743	1621
q18	9008	9246	8805	8805
q19	23921	1592	1567	1567
q20	4576	1930	1883	1883
q21	15241	4847	4869	4847
q22	1058	471	500	471
Total cold run time: 110987 ms
Total hot run time: 56149 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172861 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 436ec0052a5670b303faaea962af9aaba174047d, data reload: false

query1	938	328	322	322
query2	6580	1992	1942	1942
query3	6705	204	196	196
query4	33694	22079	22004	22004
query5	4453	415	346	346
query6	251	150	153	150
query7	4601	265	252	252
query8	242	173	171	171
query9	8619	2259	2230	2230
query10	405	206	214	206
query11	17922	15498	15395	15395
query12	126	67	66	66
query13	1671	382	384	382
query14	9096	6428	6419	6419
query15	208	183	181	181
query16	5699	254	248	248
query17	952	477	471	471
query18	1807	264	247	247
query19	174	131	130	130
query20	70	77	70	70
query21	201	135	136	135
query22	4866	4743	4543	4543
query23	31110	30353	30441	30353
query24	12456	2743	2763	2743
query25	586	306	311	306
query26	1944	135	137	135
query27	3177	280	277	277
query28	7641	1817	1808	1808
query29	1921	621	609	609
query30	288	138	135	135
query31	917	696	719	696
query32	79	47	52	47
query33	692	209	213	209
query34	1177	453	466	453
query35	854	756	739	739
query36	1269	1190	1165	1165
query37	94	56	62	56
query38	3283	3178	3153	3153
query39	1302	1270	1253	1253
query40	339	86	83	83
query41	37	37	35	35
query42	89	78	80	78
query43	531	489	477	477
query44	1137	670	676	670
query45	189	177	178	177
query46	1048	639	655	639
query47	1542	1531	1519	1519
query48	375	296	304	296
query49	1186	285	279	279
query50	685	300	309	300
query51	5269	5191	5153	5153
query52	95	78	71	71
query53	318	253	257	253
query54	257	185	183	183
query55	83	72	71	71
query56	170	154	167	154
query57	978	880	888	880
query58	189	167	152	152
query59	2829	2680	2680	2680
query60	196	183	179	179
query61	84	85	86	85
query62	606	363	352	352
query63	287	271	268	268
query64	5987	1751	1726	1726
query65	3299	3233	3204	3204
query66	1266	329	311	311
query67	14626	14237	14057	14057
query68	3808	496	485	485
query69	425	301	313	301
query70	1539	1543	1477	1477
query71	252	209	214	209
query72	3757	2863	2882	2863
query73	694	306	306	306
query74	6622	6228	6298	6228
query75	2749	2323	2321	2321
query76	2646	993	1025	993
query77	353	226	221	221
query78	8998	8775	8528	8528
query79	2178	480	477	477
query80	1929	320	310	310
query81	515	196	196	196
query82	681	87	81	81
query83	241	119	114	114
query84	284	77	71	71
query85	2129	345	334	334
query86	500	346	413	346
query87	3413	3276	3247	3247
query88	3745	2163	2154	2154
query89	408	361	357	357
query90	1928	188	182	182
query91	172	121	118	118
query92	58	45	46	45
query93	2195	418	426	418
query94	1367	161	159	159
query95	498	471	449	449
query96	602	310	310	310
query97	4225	4143	4108	4108
query98	215	200	195	195
query99	1045	683	707	683
Total cold run time: 281938 ms
Total hot run time: 172861 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 436ec0052a5670b303faaea962af9aaba174047d, data reload: false

query1	0.04	0.03	0.02
query2	0.06	0.02	0.03
query3	0.23	0.06	0.07
query4	1.68	0.10	0.10
query5	0.52	0.52	0.51
query6	1.20	0.65	0.60
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.55	0.51	0.50
query10	0.54	0.57	0.55
query11	0.12	0.09	0.09
query12	0.11	0.09	0.10
query13	0.61	0.61	0.61
query14	0.79	0.80	0.80
query15	0.81	0.76	0.77
query16	0.35	0.38	0.37
query17	1.01	1.01	1.02
query18	0.23	0.26	0.24
query19	1.91	1.82	1.79
query20	0.01	0.00	0.01
query21	15.40	0.59	0.60
query22	2.20	2.51	1.40
query23	17.40	0.88	0.71
query24	2.30	1.65	1.61
query25	0.29	0.16	0.14
query26	0.75	0.14	0.13
query27	0.06	0.07	0.04
query28	10.99	0.85	0.84
query29	12.50	3.26	3.26
query30	0.69	0.55	0.54
query31	2.78	0.34	0.35
query32	3.37	0.48	0.49
query33	3.24	3.23	3.20
query34	15.66	4.44	4.23
query35	4.34	4.38	4.38
query36	1.12	1.06	1.07
query37	0.07	0.05	0.05
query38	0.03	0.02	0.03
query39	0.02	0.02	0.02
query40	0.16	0.12	0.14
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.03
Total cold run time: 104.31 s
Total hot run time: 31.35 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 436ec0052a5670b303faaea962af9aaba174047d with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@deardeng
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37969 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7c14a589ed15a1aa91953da377480a1df83c1e60, data reload: false

------ Round 1 ----------------------------------
q1	17947	5117	5020	5020
q2	2568	147	146	146
q3	11263	960	953	953
q4	5187	821	767	767
q5	7831	2939	2896	2896
q6	189	131	122	122
q7	1197	746	716	716
q8	9277	2008	2029	2008
q9	7239	6339	6327	6327
q10	8092	2425	2421	2421
q11	414	219	210	210
q12	768	275	292	275
q13	18004	3288	3298	3288
q14	270	244	253	244
q15	526	494	490	490
q16	468	402	416	402
q17	944	462	535	462
q18	7158	6179	6153	6153
q19	1595	1489	1358	1358
q20	622	344	359	344
q21	7218	3089	3168	3089
q22	816	301	278	278
Total cold run time: 109593 ms
Total hot run time: 37969 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5191	5027	5021	5021
q2	337	252	235	235
q3	3063	2760	2703	2703
q4	1796	1609	1581	1581
q5	5313	5294	5225	5225
q6	202	114	115	114
q7	2197	1853	1863	1853
q8	3110	3178	3226	3178
q9	8612	8572	12160	8572
q10	5532	3452	3542	3452
q11	556	476	458	458
q12	769	572	610	572
q13	16939	3110	3072	3072
q14	282	251	265	251
q15	544	491	490	490
q16	540	485	509	485
q17	1924	1756	1736	1736
q18	9042	9731	8675	8675
q19	24046	1612	1567	1567
q20	4596	1934	1910	1910
q21	14830	4948	4907	4907
q22	1009	474	481	474
Total cold run time: 110430 ms
Total hot run time: 56531 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173246 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7c14a589ed15a1aa91953da377480a1df83c1e60, data reload: false

query1	944	325	336	325
query2	6546	1985	1959	1959
query3	6694	202	198	198
query4	34100	22086	22068	22068
query5	4467	367	348	348
query6	246	152	151	151
query7	4610	267	260	260
query8	275	178	179	178
query9	8525	2268	2251	2251
query10	414	210	198	198
query11	17909	15465	15338	15338
query12	125	68	66	66
query13	1680	368	380	368
query14	9182	6505	6494	6494
query15	221	178	186	178
query16	5698	258	259	258
query17	942	476	466	466
query18	1811	263	254	254
query19	177	129	131	129
query20	69	73	62	62
query21	207	133	137	133
query22	4808	4699	4670	4670
query23	31362	30512	30448	30448
query24	12857	2833	2807	2807
query25	609	313	303	303
query26	1960	140	139	139
query27	3749	299	282	282
query28	7717	1813	1805	1805
query29	2300	622	611	611
query30	290	137	138	137
query31	893	698	713	698
query32	83	49	48	48
query33	706	208	202	202
query34	1366	459	464	459
query35	847	742	768	742
query36	1343	1224	1250	1224
query37	172	56	61	56
query38	3290	3189	3105	3105
query39	1312	1252	1260	1252
query40	345	90	84	84
query41	38	34	34	34
query42	89	78	86	78
query43	527	507	496	496
query44	1112	691	681	681
query45	185	179	175	175
query46	1066	644	647	644
query47	1620	1517	1518	1517
query48	391	300	304	300
query49	1225	285	286	285
query50	677	317	318	317
query51	5212	5214	5241	5214
query52	87	82	73	73
query53	315	251	273	251
query54	242	195	191	191
query55	80	73	73	73
query56	171	153	165	153
query57	989	870	928	870
query58	178	150	154	150
query59	2841	2605	2675	2605
query60	212	179	180	179
query61	84	86	82	82
query62	612	362	367	362
query63	277	256	270	256
query64	6005	1794	1743	1743
query65	3273	3207	3200	3200
query66	1462	323	322	322
query67	14482	14123	14094	14094
query68	3715	478	475	475
query69	425	303	294	294
query70	1561	1420	1433	1420
query71	264	214	214	214
query72	3736	2831	2854	2831
query73	678	311	309	309
query74	6647	6291	6266	6266
query75	2788	2363	2279	2279
query76	2643	1034	995	995
query77	338	233	230	230
query78	9041	8760	8560	8560
query79	2888	485	488	485
query80	2021	316	312	312
query81	513	199	202	199
query82	695	90	79	79
query83	241	112	120	112
query84	287	71	79	71
query85	2312	343	329	329
query86	521	370	388	370
query87	3447	3253	3274	3253
query88	4243	2163	2148	2148
query89	421	342	343	342
query90	1946	189	185	185
query91	154	119	118	118
query92	52	43	43	43
query93	4659	435	407	407
query94	1362	159	164	159
query95	507	450	461	450
query96	617	311	316	311
query97	4258	4106	4111	4106
query98	202	187	181	181
query99	1045	679	614	614
Total cold run time: 288530 ms
Total hot run time: 173246 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.68 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7c14a589ed15a1aa91953da377480a1df83c1e60, data reload: false

query1	0.03	0.04	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.52	0.51	0.51
query6	1.18	0.63	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.54	0.51	0.49
query10	0.55	0.56	0.56
query11	0.12	0.08	0.09
query12	0.12	0.10	0.09
query13	0.61	0.61	0.61
query14	0.79	0.79	0.80
query15	0.79	0.78	0.78
query16	0.39	0.37	0.38
query17	1.03	1.01	1.05
query18	0.23	0.25	0.25
query19	1.90	1.76	1.79
query20	0.02	0.01	0.01
query21	15.41	0.56	0.57
query22	2.70	2.52	2.22
query23	17.23	0.88	0.83
query24	2.51	1.54	1.18
query25	0.32	0.23	0.14
query26	0.63	0.14	0.13
query27	0.05	0.05	0.05
query28	10.88	0.87	0.85
query29	12.48	3.16	3.10
query30	0.66	0.56	0.52
query31	2.79	0.33	0.36
query32	3.34	0.47	0.48
query33	3.24	3.20	3.21
query34	15.78	4.32	4.27
query35	4.36	4.34	4.35
query36	1.09	1.06	1.06
query37	0.07	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.02	0.02
query40	0.16	0.12	0.13
query41	0.08	0.01	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.7 s
Total hot run time: 31.68 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7c14a589ed15a1aa91953da377480a1df83c1e60 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

dataroaring
dataroaring previously approved these changes Jan 28, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 28, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@deardeng
Copy link
Contributor Author

run buildadd

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 29, 2024
@shuke987
Copy link
Collaborator

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37673 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06508f8bfc4fa781b2118a03ad88d9628139a4f0, data reload: false

------ Round 1 ----------------------------------
q1	17616	5123	4889	4889
q2	2037	144	134	134
q3	10604	925	925	925
q4	4646	752	696	696
q5	7684	2802	2799	2799
q6	187	122	122	122
q7	1171	749	716	716
q8	9322	1972	2018	1972
q9	7155	6351	6293	6293
q10	8122	2430	2433	2430
q11	412	207	195	195
q12	795	273	296	273
q13	18023	3296	3272	3272
q14	281	252	241	241
q15	535	498	492	492
q16	461	401	397	397
q17	950	534	484	484
q18	7061	6407	6289	6289
q19	1539	1363	1443	1363
q20	631	336	341	336
q21	7018	3157	3090	3090
q22	817	308	265	265
Total cold run time: 107067 ms
Total hot run time: 37673 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5162	4902	4899	4899
q2	360	242	228	228
q3	3062	2799	2634	2634
q4	1837	1562	1537	1537
q5	5323	5193	5274	5193
q6	208	115	118	115
q7	2192	1792	1733	1733
q8	3130	3185	3243	3185
q9	9079	8478	14250	8478
q10	5478	3419	3478	3419
q11	560	459	464	459
q12	779	572	603	572
q13	16962	3100	3089	3089
q14	297	257	280	257
q15	547	487	491	487
q16	521	465	465	465
q17	1840	1794	1733	1733
q18	8912	11716	8506	8506
q19	24078	1536	1555	1536
q20	4602	1933	1883	1883
q21	16096	4855	4820	4820
q22	1132	453	501	453
Total cold run time: 112157 ms
Total hot run time: 55681 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172940 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 06508f8bfc4fa781b2118a03ad88d9628139a4f0, data reload: false

query1	957	343	326	326
query2	6519	1909	1900	1900
query3	6696	199	199	199
query4	33749	21979	22053	21979
query5	4445	373	372	372
query6	248	157	155	155
query7	4600	270	260	260
query8	245	178	196	178
query9	8468	2279	2249	2249
query10	411	211	213	211
query11	17941	15654	15337	15337
query12	118	71	67	67
query13	1674	365	367	365
query14	9106	6518	6458	6458
query15	206	190	179	179
query16	5702	251	247	247
query17	958	470	468	468
query18	1803	254	254	254
query19	181	143	134	134
query20	68	73	71	71
query21	197	133	125	125
query22	4931	4723	4667	4667
query23	31445	30506	30449	30449
query24	12636	2741	2780	2741
query25	583	307	315	307
query26	1987	144	142	142
query27	3347	290	285	285
query28	7832	1834	1802	1802
query29	2229	622	609	609
query30	284	139	137	137
query31	889	723	714	714
query32	80	51	48	48
query33	706	209	215	209
query34	1257	459	464	459
query35	843	756	745	745
query36	1292	1119	1107	1107
query37	174	56	59	56
query38	3235	3170	3144	3144
query39	1306	1246	1235	1235
query40	336	86	82	82
query41	40	35	36	35
query42	87	85	79	79
query43	540	496	526	496
query44	1103	692	699	692
query45	194	183	174	174
query46	1044	652	637	637
query47	1611	1494	1531	1494
query48	378	303	295	295
query49	1222	286	289	286
query50	698	318	324	318
query51	5258	5257	5175	5175
query52	91	80	79	79
query53	326	257	273	257
query54	258	179	187	179
query55	74	78	79	78
query56	173	166	161	161
query57	1016	911	885	885
query58	187	174	163	163
query59	2649	2571	2600	2571
query60	202	177	176	176
query61	88	81	80	80
query62	634	381	351	351
query63	288	261	269	261
query64	5973	1739	1702	1702
query65	3287	3244	3203	3203
query66	1487	320	306	306
query67	14440	14277	13928	13928
query68	3597	499	492	492
query69	421	303	295	295
query70	1539	1465	1513	1465
query71	284	215	238	215
query72	3696	2822	2847	2822
query73	686	319	314	314
query74	6621	6206	6317	6206
query75	2851	2304	2303	2303
query76	2619	1032	1017	1017
query77	442	232	236	232
query78	9022	8785	8592	8592
query79	2465	488	492	488
query80	1443	325	310	310
query81	502	200	202	200
query82	711	80	79	79
query83	232	116	117	116
query84	277	74	72	72
query85	1769	346	327	327
query86	511	410	382	382
query87	3445	3261	3246	3246
query88	3815	2170	2172	2170
query89	397	354	371	354
query90	2165	189	189	189
query91	148	117	114	114
query92	51	44	45	44
query93	1937	438	417	417
query94	1507	165	159	159
query95	500	447	454	447
query96	614	324	313	313
query97	4283	4132	4162	4132
query98	205	195	187	187
query99	1024	708	715	708
Total cold run time: 282811 ms
Total hot run time: 172940 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 06508f8bfc4fa781b2118a03ad88d9628139a4f0, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.09
query5	0.52	0.51	0.52
query6	1.19	0.64	0.65
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.55	0.50	0.50
query10	0.54	0.57	0.54
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.60	0.62	0.60
query14	0.79	0.81	0.81
query15	0.79	0.78	0.77
query16	0.40	0.37	0.39
query17	1.00	1.01	1.02
query18	0.20	0.27	0.22
query19	1.88	1.72	1.78
query20	0.01	0.00	0.01
query21	15.40	0.59	0.61
query22	3.08	2.82	2.16
query23	17.44	0.86	0.84
query24	2.33	0.93	1.45
query25	0.32	0.13	0.31
query26	0.51	0.14	0.15
query27	0.05	0.05	0.05
query28	11.94	0.87	0.85
query29	12.52	3.25	3.17
query30	0.60	0.58	0.52
query31	2.79	0.35	0.35
query32	3.36	0.49	0.48
query33	3.18	3.23	3.23
query34	15.89	4.29	4.28
query35	4.25	4.24	4.25
query36	1.10	1.06	1.07
query37	0.07	0.05	0.06
query38	0.03	0.02	0.03
query39	0.02	0.02	0.02
query40	0.19	0.13	0.14
query41	0.07	0.01	0.02
query42	0.03	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.92 s
Total hot run time: 31.37 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 06508f8bfc4fa781b2118a03ad88d9628139a4f0 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

dataroaring
dataroaring approved these changes Jan 29, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37099 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06508f8bfc4fa781b2118a03ad88d9628139a4f0, data reload: false

------ Round 1 ----------------------------------
q1	17701	4774	4952	4774
q2	2049	141	131	131
q3	10711	929	902	902
q4	4723	673	669	669
q5	7679	2777	2723	2723
q6	184	117	118	117
q7	1125	711	697	697
q8	9406	1983	2024	1983
q9	7214	6333	6320	6320
q10	8098	2400	2427	2400
q11	413	193	197	193
q12	810	276	287	276
q13	18003	3263	3242	3242
q14	280	261	244	244
q15	531	513	482	482
q16	457	397	404	397
q17	928	538	542	538
q18	6839	6079	6006	6006
q19	1575	1348	1376	1348
q20	568	340	310	310
q21	6797	3061	3094	3061
q22	802	289	286	286
Total cold run time: 106893 ms
Total hot run time: 37099 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5154	4897	4984	4897
q2	339	238	247	238
q3	2981	2747	2814	2747
q4	1828	1512	1598	1512
q5	5294	5273	5247	5247
q6	198	120	118	118
q7	2156	1720	1773	1720
q8	3097	3233	3272	3233
q9	8621	8643	21187	8643
q10	5603	3500	3388	3388
q11	553	471	453	453
q12	763	621	568	568
q13	16936	3126	3048	3048
q14	284	248	272	248
q15	538	494	494	494
q16	527	455	484	455
q17	2020	1689	1728	1689
q18	8801	8565	8634	8565
q19	23849	1580	1556	1556
q20	4597	1892	1876	1876
q21	14791	4831	4869	4831
q22	1080	485	467	467
Total cold run time: 110010 ms
Total hot run time: 55993 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172723 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 06508f8bfc4fa781b2118a03ad88d9628139a4f0, data reload: false

query1	933	330	317	317
query2	6593	2042	2005	2005
query3	6708	196	194	194
query4	32273	22107	22071	22071
query5	4452	352	353	352
query6	251	175	154	154
query7	4620	261	258	258
query8	244	170	181	170
query9	8529	2274	2241	2241
query10	413	197	197	197
query11	17761	15386	15476	15386
query12	115	70	67	67
query13	1691	400	374	374
query14	9076	6404	6441	6404
query15	220	179	190	179
query16	5694	248	246	246
query17	962	472	458	458
query18	1794	256	249	249
query19	182	130	135	130
query20	71	72	66	66
query21	202	127	122	122
query22	4971	4589	4788	4589
query23	31536	30229	30226	30226
query24	7438	2769	2721	2721
query25	421	304	300	300
query26	785	133	142	133
query27	2763	281	285	281
query28	4035	1837	1821	1821
query29	916	596	615	596
query30	205	137	137	137
query31	890	695	729	695
query32	63	51	50	50
query33	422	215	201	201
query34	821	459	481	459
query35	810	766	742	742
query36	1313	1186	1265	1186
query37	90	54	59	54
query38	3256	3198	3168	3168
query39	1308	1249	1234	1234
query40	198	90	80	80
query41	38	39	34	34
query42	85	83	85	83
query43	551	490	514	490
query44	1056	694	688	688
query45	186	177	174	174
query46	1025	678	645	645
query47	1544	1531	1469	1469
query48	395	308	306	306
query49	626	287	278	278
query50	679	321	316	316
query51	5292	5130	5103	5103
query52	97	76	81	76
query53	313	263	261	261
query54	229	192	192	192
query55	78	77	73	73
query56	167	170	171	170
query57	938	908	904	904
query58	179	149	157	149
query59	2700	2694	2684	2684
query60	195	184	174	174
query61	82	81	78	78
query62	457	377	359	359
query63	282	258	277	258
query64	2647	1804	1704	1704
query65	3270	3238	3229	3229
query66	1137	319	321	319
query67	14488	14214	14293	14214
query68	3706	493	491	491
query69	445	300	294	294
query70	1533	1492	1395	1395
query71	273	219	204	204
query72	3706	2821	2813	2813
query73	682	312	311	311
query74	6634	6165	6255	6165
query75	2838	2311	2291	2291
query76	2641	1011	965	965
query77	363	232	227	227
query78	9092	8719	8532	8532
query79	2641	491	497	491
query80	1385	317	310	310
query81	483	196	197	196
query82	987	79	80	79
query83	226	114	112	112
query84	280	75	80	75
query85	1064	344	331	331
query86	436	378	373	373
query87	3441	3232	3289	3232
query88	3752	2164	2157	2157
query89	411	358	331	331
query90	2102	183	179	179
query91	142	115	114	114
query92	53	43	44	43
query93	1871	435	429	429
query94	1354	154	160	154
query95	500	463	457	457
query96	598	314	314	314
query97	4238	4093	4094	4093
query98	204	191	188	188
query99	1203	644	634	634
Total cold run time: 263045 ms
Total hot run time: 172723 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.64 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 06508f8bfc4fa781b2118a03ad88d9628139a4f0, data reload: false

query1	0.04	0.02	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.68	0.09	0.10
query5	0.53	0.52	0.51
query6	1.20	0.63	0.63
query7	0.02	0.02	0.01
query8	0.03	0.02	0.02
query9	0.54	0.50	0.49
query10	0.54	0.56	0.54
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.60	0.61	0.60
query14	0.78	0.78	0.79
query15	0.80	0.78	0.77
query16	0.39	0.38	0.39
query17	1.03	1.03	1.04
query18	0.23	0.22	0.26
query19	1.89	1.80	1.79
query20	0.01	0.01	0.01
query21	15.42	0.56	0.57
query22	2.47	2.53	1.86
query23	16.89	0.99	0.73
query24	2.33	1.45	0.63
query25	0.27	0.26	0.09
query26	0.54	0.14	0.15
query27	0.05	0.05	0.06
query28	12.11	0.83	0.83
query29	12.78	3.16	3.21
query30	0.64	0.49	0.54
query31	2.78	0.35	0.34
query32	3.37	0.49	0.48
query33	3.24	3.18	3.23
query34	15.71	4.34	4.27
query35	4.33	4.36	4.37
query36	1.12	1.06	1.07
query37	0.06	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.25 s
Total hot run time: 30.64 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 06508f8bfc4fa781b2118a03ad88d9628139a4f0 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@dataroaring dataroaring merged commit 3dade1c into apache:master Jan 29, 2024
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@deardeng @doris-robot @shuke987 @dataroaring @gavinchou