[fix](multi-table-load) fix multi table load can not finish #29957

sollhui · 2024-01-14T14:59:46Z

Proposed changes

Multi table load task could can not finish in this situation:

Thread A: submit and exec plan asynchronous
Thread B: exec plan finished
Thread A: set consumer done

If threadB faster than thread A, the task would not finish util abort after timeout, for consumer done would not set task finish status. Therefore, adding reference count check to consumer.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

sollhui · 2024-01-14T14:59:55Z

run buildall

sollhui · 2024-01-14T15:05:10Z

run buildall

github-actions · 2024-01-14T15:08:00Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-01-14T15:12:52Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-14T15:24:59Z

TPC-H: Total hot run time: 39943 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 45968bc955ad31e89f82063871a39c14c0b40e88, data reload: false

------ Round 1 ----------------------------------
q1	18139	6046	6285	6046
q2	2051	158	135	135
q3	10667	1152	1152	1152
q4	10279	786	822	786
q5	7823	3216	3176	3176
q6	203	125	125	125
q7	900	491	484	484
q8	9296	1963	2002	1963
q9	7278	6433	6431	6431
q10	8242	3048	3099	3048
q11	422	220	209	209
q12	369	191	194	191
q13	18110	3418	3423	3418
q14	245	230	219	219
q15	554	527	512	512
q16	444	412	398	398
q17	952	503	503	503
q18	7802	7233	6808	6808
q19	1637	1329	1560	1329
q20	569	319	297	297
q21	2926	2442	2397	2397
q22	365	333	316	316
Total cold run time: 109273 ms
Total hot run time: 39943 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5849	5672	6100	5672
q2	347	223	232	223
q3	3369	3122	3141	3122
q4	2216	2027	2090	2027
q5	6139	5985	5917	5917
q6	211	128	123	123
q7	2433	2023	1968	1968
q8	3579	3658	3733	3658
q9	9770	20972	10042	10042
q10	8615	3658	3865	3658
q11	593	473	462	462
q12	1191	628	682	628
q13	17080	3262	3236	3236
q14	322	266	257	257
q15	584	508	509	508
q16	550	487	485	485
q17	3689	1966	2069	1966
q18	9612	9642	10520	9642
q19	12931	1637	1623	1623
q20	3950	1953	1940	1940
q21	13194	5727	5731	5727
q22	1537	598	622	598
Total cold run time: 107761 ms
Total hot run time: 63482 ms

doris-robot · 2024-01-14T15:36:04Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.79 seconds
stream load tsv: 561 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.6 seconds inserted 10000000 Rows, about 462K ops/s
storage size: 17183973713 Bytes

doris-robot · 2024-01-14T15:36:47Z

TPC-DS: Total hot run time: 178647 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 45968bc955ad31e89f82063871a39c14c0b40e88, data reload: false

query1	939	337	328	328
query2	6736	1831	1868	1831
query3	6712	221	211	211
query4	34784	22172	22150	22150
query5	6953	527	585	527
query6	262	183	186	183
query7	4608	284	277	277
query8	233	197	201	197
query9	9164	2702	2781	2702
query10	481	243	238	238
query11	16556	15278	15193	15193
query12	130	71	70	70
query13	1690	384	381	381
query14	11259	7442	7415	7415
query15	216	184	186	184
query16	4618	260	255	255
query17	994	495	473	473
query18	1803	267	268	267
query19	180	149	147	147
query20	80	80	74	74
query21	189	122	124	122
query22	5106	4994	4799	4799
query23	31822	31845	31158	31158
query24	12709	2823	2780	2780
query25	569	328	328	328
query26	1854	152	155	152
query27	3623	288	275	275
query28	7485	1845	1840	1840
query29	2230	696	676	676
query30	283	138	145	138
query31	942	756	772	756
query32	75	63	61	61
query33	712	243	255	243
query34	1295	459	456	456
query35	899	771	757	757
query36	1394	1256	1260	1256
query37	186	67	71	67
query38	3431	3249	3244	3244
query39	1327	1273	1290	1273
query40	341	99	96	96
query41	39	35	35	35
query42	99	94	102	94
query43	558	508	531	508
query44	1109	689	697	689
query45	204	186	187	186
query46	1045	655	655	655
query47	1690	1587	1578	1578
query48	421	332	310	310
query49	1246	315	302	302
query50	703	318	309	309
query51	5312	5308	5215	5215
query52	106	86	99	86
query53	354	298	313	298
query54	897	449	451	449
query55	96	93	88	88
query56	209	196	206	196
query57	1022	969	934	934
query58	216	195	193	193
query59	2854	2574	2595	2574
query60	237	215	230	215
query61	86	95	89	89
query62	697	396	408	396
query63	299	291	296	291
query64	6250	1667	1652	1652
query65	3339	3274	3276	3274
query66	1422	335	332	332
query67	15413	15040	14931	14931
query68	12319	518	501	501
query69	657	335	338	335
query70	1702	1616	1536	1536
query71	584	259	233	233
query72	5021	2858	2902	2858
query73	3090	322	313	313
query74	7549	6407	6473	6407
query75	5680	2355	2310	2310
query76	6586	881	1094	881
query77	984	264	307	264
query78	9226	8709	8596	8596
query79	1257	503	509	503
query80	645	340	351	340
query81	464	210	208	208
query82	199	95	88	88
query83	209	122	121	121
query84	271	72	73	72
query85	1108	344	336	336
query86	383	397	384	384
query87	3605	3387	3372	3372
query88	3466	2291	2297	2291
query89	452	405	395	395
query90	2250	214	196	196
query91	153	138	135	135
query92	62	55	52	52
query93	1411	407	404	404
query94	1534	180	177	177
query95	512	490	472	472
query96	636	330	331	330
query97	4263	4168	4194	4168
query98	194	188	188	188
query99	1082	722	738	722
Total cold run time: 308347 ms
Total hot run time: 178647 ms

sollhui · 2024-01-14T15:37:28Z

run buildall

github-actions · 2024-01-14T15:45:10Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-14T15:57:21Z

TeamCity be ut coverage result:
Function Coverage: 36.70% (8649/23569)
Line Coverage: 28.71% (70445/245338)
Region Coverage: 27.63% (36411/131786)
Branch Coverage: 24.34% (18620/76486)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4561050a32cefa5359d52e86dd1781b0a2973274_4561050a32cefa5359d52e86dd1781b0a2973274/report/index.html

doris-robot · 2024-01-14T15:57:25Z

TPC-H: Total hot run time: 40100 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false

------ Round 1 ----------------------------------
q1	17947	6301	6020	6020
q2	2052	152	151	151
q3	10865	1112	1174	1112
q4	10522	809	762	762
q5	7816	3172	3164	3164
q6	206	124	124	124
q7	872	515	477	477
q8	9262	1989	2002	1989
q9	7299	6453	6418	6418
q10	8218	3043	3078	3043
q11	418	205	212	205
q12	363	193	191	191
q13	18100	3389	3435	3389
q14	250	221	210	210
q15	550	524	533	524
q16	432	393	384	384
q17	948	518	542	518
q18	7563	6988	7022	6988
q19	1618	1378	1402	1378
q20	564	307	280	280
q21	2892	2476	2460	2460
q22	395	330	313	313
Total cold run time: 109152 ms
Total hot run time: 40100 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5727	5582	5814	5582
q2	348	220	219	219
q3	3372	3161	3067	3067
q4	2222	2136	2031	2031
q5	6037	5933	5984	5933
q6	204	122	124	122
q7	2356	1942	2013	1942
q8	3657	3694	3790	3694
q9	9685	26064	9551	9551
q10	4145	3780	3841	3780
q11	595	449	486	449
q12	1168	650	640	640
q13	17107	3294	3247	3247
q14	310	267	275	267
q15	565	508	519	508
q16	544	475	472	472
q17	2515	2040	1994	1994
q18	9641	9475	12300	9475
q19	12792	1624	1634	1624
q20	3831	1960	1938	1938
q21	14733	5845	5602	5602
q22	980	635	594	594
Total cold run time: 102534 ms
Total hot run time: 62731 ms

doris-robot · 2024-01-14T16:08:23Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.66 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.1 seconds inserted 10000000 Rows, about 473K ops/s
storage size: 17183923051 Bytes

doris-robot · 2024-01-14T16:09:05Z

TPC-DS: Total hot run time: 179479 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false

query1	960	340	328	328
query2	6720	1912	1851	1851
query3	6708	220	212	212
query4	32297	22414	22099	22099
query5	7011	542	531	531
query6	269	197	196	196
query7	4635	282	276	276
query8	239	201	201	201
query9	9494	2757	2857	2757
query10	453	242	229	229
query11	16574	15348	15268	15268
query12	128	75	70	70
query13	1689	398	394	394
query14	11239	7387	7418	7387
query15	229	178	185	178
query16	4637	251	245	245
query17	999	500	479	479
query18	1808	269	265	265
query19	179	150	152	150
query20	81	80	75	75
query21	197	142	135	135
query22	4907	4811	4738	4738
query23	32270	31178	31084	31084
query24	12720	2914	2882	2882
query25	582	331	329	329
query26	1835	153	162	153
query27	3309	279	280	279
query28	7283	1867	1848	1848
query29	2223	692	684	684
query30	284	137	146	137
query31	941	751	762	751
query32	76	58	60	58
query33	717	241	261	241
query34	1275	470	467	467
query35	901	817	767	767
query36	1420	1308	1307	1307
query37	186	74	73	73
query38	3425	3291	3285	3285
query39	1337	1274	1274	1274
query40	345	102	99	99
query41	40	38	36	36
query42	101	102	91	91
query43	544	535	509	509
query44	1099	699	703	699
query45	205	190	181	181
query46	1072	634	652	634
query47	1623	1605	1538	1538
query48	401	315	332	315
query49	1253	307	307	307
query50	705	317	316	316
query51	5307	5244	5216	5216
query52	108	92	95	92
query53	355	292	295	292
query54	891	460	470	460
query55	95	90	90	90
query56	213	213	197	197
query57	1024	923	903	903
query58	218	198	200	198
query59	2991	2569	2691	2569
query60	234	216	221	216
query61	86	96	86	86
query62	682	423	406	406
query63	308	283	295	283
query64	6284	1680	1676	1676
query65	3359	3269	3257	3257
query66	1431	352	335	335
query67	15813	15343	15275	15275
query68	9651	550	534	534
query69	659	338	344	338
query70	1708	1604	1544	1544
query71	541	252	255	252
query72	4305	2867	2847	2847
query73	2053	323	332	323
query74	7229	6485	6466	6466
query75	4518	2344	2354	2344
query76	5782	1012	1068	1012
query77	857	279	271	271
query78	9082	8924	8621	8621
query79	1058	513	496	496
query80	737	350	361	350
query81	464	207	203	203
query82	307	102	91	91
query83	175	125	133	125
query84	279	80	73	73
query85	1138	341	329	329
query86	388	382	391	382
query87	3504	3381	3386	3381
query88	3564	2313	2294	2294
query89	464	385	397	385
query90	2069	210	202	202
query91	162	122	138	122
query92	64	54	55	54
query93	1618	412	426	412
query94	1282	182	182	182
query95	515	488	473	473
query96	634	331	331	331
query97	4287	4156	4262	4156
query98	206	183	183	183
query99	1027	756	744	744
Total cold run time: 299351 ms
Total hot run time: 179479 ms

doris-robot · 2024-01-14T16:14:31Z

ClickBench: Total hot run time: 30.94 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false

query1	0.05	0.06	0.05
query2	0.06	0.03	0.02
query3	0.26	0.12	0.11
query4	1.78	0.13	0.11
query5	0.54	0.53	0.53
query6	1.32	0.65	0.64
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.55	0.49	0.51
query10	0.56	0.58	0.57
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.60	0.60	0.60
query14	0.78	0.78	0.79
query15	0.80	0.78	0.77
query16	0.38	0.36	0.35
query17	1.01	0.98	1.04
query18	0.24	0.25	0.24
query19	1.89	1.81	1.78
query20	0.02	0.01	0.01
query21	15.41	0.59	0.56
query22	2.52	1.75	2.12
query23	17.29	0.77	0.87
query24	2.43	1.18	0.92
query25	1.98	0.16	0.15
query26	0.14	0.13	0.14
query27	0.14	0.15	0.15
query28	11.02	0.80	0.80
query29	12.56	3.17	3.16
query30	0.49	0.48	0.48
query31	2.78	0.35	0.35
query32	3.39	0.48	0.50
query33	3.22	3.25	3.25
query34	15.85	4.34	4.21
query35	4.25	4.23	4.23
query36	1.12	1.09	1.07
query37	0.06	0.05	0.05
query38	0.03	0.04	0.03
query39	0.02	0.01	0.02
query40	0.16	0.12	0.13
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.01	0.02
Total cold run time: 106.12 s
Total hot run time: 30.94 s

sollhui · 2024-01-15T03:28:56Z

run buildall

doris-robot · 2024-01-15T03:51:51Z

TeamCity be ut coverage result:
Function Coverage: 36.68% (8648/23577)
Line Coverage: 28.70% (70454/245455)
Region Coverage: 27.63% (36421/131818)
Branch Coverage: 24.35% (18628/76514)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4561050a32cefa5359d52e86dd1781b0a2973274_4561050a32cefa5359d52e86dd1781b0a2973274/report/index.html

doris-robot · 2024-01-15T03:59:21Z

TPC-H: Total hot run time: 40008 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4561050a32cefa5359d52e86dd1781b0a2973274, data reload: false

------ Round 1 ----------------------------------
q1	18053	6242	5932	5932
q2	2041	150	132	132
q3	10910	1193	1165	1165
q4	10544	809	802	802
q5	7829	3159	3178	3159
q6	204	124	127	124
q7	889	518	489	489
q8	9294	2009	1998	1998
q9	7270	6444	6388	6388
q10	8233	3090	3089	3089
q11	416	207	222	207
q12	364	192	192	192
q13	18099	3477	3423	3423
q14	257	220	218	218
q15	562	515	511	511
q16	461	395	377	377
q17	944	510	500	500
q18	7656	7012	6939	6939
q19	1612	1304	1355	1304
q20	538	307	293	293
q21	2818	2534	2450	2450
q22	389	328	316	316
Total cold run time: 109383 ms
Total hot run time: 40008 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6009	5806	5675	5675
q2	355	228	222	222
q3	3284	3122	3115	3115
q4	2251	2008	2089	2008
q5	6068	5944	6022	5944
q6	207	126	121	121
q7	2455	2050	1973	1973
q8	3705	3659	3874	3659
q9	9728	25503	9606	9606
q10	10511	3821	3901	3821
q11	721	461	482	461
q12	1498	658	673	658
q13	17356	3256	3235	3235
q14	394	256	274	256
q15	568	509	514	509
q16	564	477	473	473
q17	9324	1962	1960	1960
q18	9529	10413	9511	9511
q19	14488	1573	1617	1573
q20	4709	1956	1976	1956
q21	16303	5616	5709	5616
q22	3032	618	652	618
Total cold run time: 123059 ms
Total hot run time: 62970 ms

doris-robot · 2024-01-15T04:16:07Z

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.59 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.9 seconds inserted 10000000 Rows, about 456K ops/s
storage size: 17183914922 Bytes

github-actions · 2024-01-15T11:21:32Z

PR approved by anyone and no changes requested.

be/src/io/fs/multi_table_pipe.h

sollhui · 2024-01-16T10:11:26Z

run buildall

github-actions · 2024-01-16T10:19:10Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-16T10:33:26Z

TPC-H: Total hot run time: 39614 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e8b53fbf394389ba99924c201800c50b6abfec85, data reload: false

------ Round 1 ----------------------------------
q1	18544	5709	5864	5709
q2	2419	152	141	141
q3	11946	1131	1200	1131
q4	10844	840	834	834
q5	7972	3096	3190	3096
q6	220	127	131	127
q7	915	504	485	485
q8	9215	1942	2010	1942
q9	7310	6398	6364	6364
q10	8207	3026	3057	3026
q11	403	219	205	205
q12	363	196	198	196
q13	17998	3425	3386	3386
q14	252	218	214	214
q15	555	533	503	503
q16	445	409	387	387
q17	956	485	481	481
q18	7663	6984	6966	6966
q19	1593	1307	1438	1307
q20	541	320	345	320
q21	2926	2484	2532	2484
q22	386	311	310	310
Total cold run time: 111673 ms
Total hot run time: 39614 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5868	5804	5657	5657
q2	336	223	221	221
q3	3382	3320	3255	3255
q4	2165	2099	1995	1995
q5	6165	5957	6064	5957
q6	208	123	124	123
q7	2318	1977	1923	1923
q8	3320	3505	3511	3505
q9	9390	9011	8757	8757
q10	6371	3706	3918	3706
q11	568	457	446	446
q12	863	641	665	641
q13	16958	3238	3199	3199
q14	291	263	280	263
q15	550	511	511	511
q16	544	466	475	466
q17	2031	1979	1973	1973
q18	9641	9504	15925	9504
q19	23379	1699	1634	1634
q20	4602	1963	1939	1939
q21	13809	5520	5494	5494
q22	991	570	532	532
Total cold run time: 113750 ms
Total hot run time: 61701 ms

doris-robot · 2024-01-16T10:33:53Z

TeamCity be ut coverage result:
Function Coverage: 36.67% (8648/23581)
Line Coverage: 28.73% (70619/245773)
Region Coverage: 27.62% (36436/131925)
Branch Coverage: 24.34% (18640/76590)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e8b53fbf394389ba99924c201800c50b6abfec85_e8b53fbf394389ba99924c201800c50b6abfec85/report/index.html

doris-robot · 2024-01-16T10:44:58Z

TPC-DS: Total hot run time: 177369 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e8b53fbf394389ba99924c201800c50b6abfec85, data reload: false

query1	938	329	336	329
query2	6544	2074	1928	1928
query3	6711	199	197	197
query4	33718	22282	22172	22172
query5	6928	601	600	600
query6	264	177	174	174
query7	4607	265	263	263
query8	226	176	186	176
query9	8431	2507	2476	2476
query10	421	234	226	226
query11	16733	15341	15268	15268
query12	120	66	65	65
query13	1689	365	383	365
query14	10499	6828	6870	6828
query15	211	182	181	181
query16	4602	241	238	238
query17	980	484	477	477
query18	1789	264	259	259
query19	174	143	133	133
query20	71	70	66	66
query21	192	137	131	131
query22	5100	4859	4852	4852
query23	31517	30764	30677	30677
query24	11443	2783	2743	2743
query25	564	317	310	310
query26	1620	144	143	143
query27	3214	278	267	267
query28	7171	1851	1847	1847
query29	1395	653	684	653
query30	278	142	147	142
query31	926	746	763	746
query32	72	51	48	48
query33	691	225	215	215
query34	1115	464	462	462
query35	900	802	764	764
query36	1422	1262	1281	1262
query37	91	58	57	57
query38	3355	3224	3208	3208
query39	1310	1288	1255	1255
query40	206	92	85	85
query41	38	34	34	34
query42	89	82	89	82
query43	540	516	474	474
query44	1041	690	715	690
query45	200	181	176	176
query46	1071	657	655	655
query47	1703	1504	1553	1504
query48	400	315	312	312
query49	1132	294	286	286
query50	700	310	312	310
query51	5302	5205	5215	5205
query52	87	87	78	78
query53	338	276	281	276
query54	891	440	441	440
query55	86	88	86	86
query56	178	177	181	177
query57	999	916	938	916
query58	197	163	174	163
query59	2866	2734	2729	2729
query60	227	195	195	195
query61	84	83	89	83
query62	679	396	401	396
query63	307	265	264	264
query64	5015	1754	1735	1735
query65	3353	3251	3262	3251
query66	1244	331	320	320
query67	15927	15415	15310	15310
query68	10669	526	539	526
query69	623	303	300	300
query70	1782	1610	1603	1603
query71	548	213	225	213
query72	4834	2826	2834	2826
query73	2117	316	315	315
query74	7538	6454	6383	6383
query75	4728	2334	2277	2277
query76	6257	938	1001	938
query77	664	245	246	245
query78	9144	9040	8708	8708
query79	1031	499	495	495
query80	684	318	318	318
query81	466	204	212	204
query82	184	81	79	79
query83	146	119	118	118
query84	277	69	71	69
query85	1087	346	336	336
query86	384	397	381	381
query87	3468	3377	3345	3345
query88	3277	2190	2185	2185
query89	444	363	369	363
query90	1998	191	186	186
query91	149	124	129	124
query92	56	55	46	46
query93	3178	434	419	419
query94	1209	167	162	162
query95	498	457	454	454
query96	648	311	307	307
query97	4240	4148	4138	4138
query98	205	186	187	186
query99	989	685	685	685
Total cold run time: 296454 ms
Total hot run time: 177369 ms

doris-robot · 2024-01-16T10:50:23Z

ClickBench: Total hot run time: 30.81 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e8b53fbf394389ba99924c201800c50b6abfec85, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.02
query3	0.24	0.05	0.05
query4	1.69	0.06	0.06
query5	0.53	0.52	0.52
query6	1.25	0.66	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.53	0.48	0.51
query10	0.56	0.57	0.56
query11	0.13	0.09	0.09
query12	0.11	0.09	0.10
query13	0.60	0.60	0.59
query14	0.77	0.78	0.79
query15	0.80	0.78	0.78
query16	0.39	0.36	0.37
query17	1.01	1.00	1.01
query18	0.24	0.24	0.26
query19	1.88	1.71	1.76
query20	0.02	0.01	0.01
query21	15.40	0.56	0.56
query22	2.48	3.12	1.56
query23	17.22	1.02	1.01
query24	2.63	1.11	1.38
query25	0.35	0.28	0.15
query26	0.60	0.14	0.14
query27	0.06	0.06	0.05
query28	10.72	0.76	0.77
query29	12.52	3.14	3.23
query30	0.55	0.49	0.47
query31	2.78	0.34	0.35
query32	3.38	0.47	0.48
query33	3.23	3.24	3.21
query34	16.07	4.29	4.22
query35	4.26	4.28	4.24
query36	1.10	1.09	1.08
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.18	0.13	0.13
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.02
Total cold run time: 104.67 s
Total hot run time: 30.81 s

doris-robot · 2024-01-16T10:53:29Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e8b53fbf394389ba99924c201800c50b6abfec85 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       14.9 seconds inserted 10000000 Rows, about 671K ops/s

be/src/io/fs/multi_table_pipe.cpp

sollhui · 2024-01-16T13:28:36Z

run buildall

github-actions · 2024-01-16T13:36:24Z

clang-tidy review says "All clean, LGTM! 👍"

sollhui · 2024-01-16T13:57:09Z

run buildall

github-actions · 2024-01-16T14:04:58Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-01-16T14:28:33Z

TPC-H: Total hot run time: 38951 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380, data reload: false

------ Round 1 ----------------------------------
q1	17716	5530	5300	5300
q2	2036	142	137	137
q3	10733	1169	1159	1159
q4	10286	789	727	727
q5	7766	3191	3196	3191
q6	197	123	120	120
q7	846	487	499	487
q8	9223	1971	1980	1971
q9	7337	6413	6389	6389
q10	8235	3043	3079	3043
q11	433	212	193	193
q12	356	190	187	187
q13	18015	3378	3329	3329
q14	242	208	216	208
q15	539	503	510	503
q16	467	373	370	370
q17	955	515	476	476
q18	7405	6911	6826	6826
q19	1827	1354	1408	1354
q20	556	309	283	283
q21	2832	2400	2446	2400
q22	348	303	298	298
Total cold run time: 108350 ms
Total hot run time: 38951 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5443	5339	5203	5203
q2	327	213	218	213
q3	3347	3216	3193	3193
q4	2089	2057	2031	2031
q5	6017	5929	5798	5798
q6	197	116	113	113
q7	2315	1849	1875	1849
q8	3222	3360	3398	3360
q9	8947	8882	8837	8837
q10	3937	3789	3775	3775
q11	554	457	441	441
q12	794	652	634	634
q13	16900	3149	3136	3136
q14	298	256	260	256
q15	548	504	505	504
q16	548	480	480	480
q17	1855	1889	1875	1875
q18	9558	19567	9377	9377
q19	24733	1557	1510	1510
q20	4605	1933	1932	1932
q21	14402	5498	5391	5391
q22	973	547	544	544
Total cold run time: 111609 ms
Total hot run time: 60452 ms

doris-robot · 2024-01-16T14:40:02Z

TPC-DS: Total hot run time: 176822 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380, data reload: false

query1	937	340	328	328
query2	6564	1943	1842	1842
query3	6705	208	207	207
query4	31434	22287	22097	22097
query5	6905	597	594	594
query6	264	183	175	175
query7	4606	262	257	257
query8	227	178	177	177
query9	8374	2525	2516	2516
query10	413	239	220	220
query11	16666	15512	15593	15512
query12	117	70	74	70
query13	1706	377	370	370
query14	10490	6782	6814	6782
query15	206	172	176	172
query16	4611	241	227	227
query17	981	485	475	475
query18	1788	259	250	250
query19	168	138	130	130
query20	70	69	62	62
query21	187	127	125	125
query22	4904	4896	4809	4809
query23	32006	30812	30883	30812
query24	11484	2829	2817	2817
query25	536	312	310	310
query26	884	134	134	134
query27	2890	279	266	266
query28	6361	1802	1808	1802
query29	925	616	628	616
query30	284	136	144	136
query31	945	756	763	756
query32	69	47	48	47
query33	701	219	207	207
query34	1143	452	454	452
query35	858	762	735	735
query36	1338	1287	1290	1287
query37	91	60	58	58
query38	3428	3260	3222	3222
query39	1317	1274	1252	1252
query40	192	86	85	85
query41	38	35	35	35
query42	92	83	83	83
query43	525	500	508	500
query44	1051	677	693	677
query45	193	188	179	179
query46	1065	663	663	663
query47	1693	1560	1628	1560
query48	397	303	312	303
query49	1096	292	278	278
query50	689	304	303	303
query51	5312	5206	5172	5172
query52	83	84	75	75
query53	321	252	263	252
query54	887	428	443	428
query55	79	78	80	78
query56	170	162	172	162
query57	1011	933	954	933
query58	183	159	156	156
query59	2826	2692	2753	2692
query60	216	183	180	180
query61	84	81	93	81
query62	639	412	395	395
query63	272	259	252	252
query64	4545	1801	1766	1766
query65	3358	3279	3234	3234
query66	1233	315	306	306
query67	15810	15146	14911	14911
query68	12882	519	495	495
query69	628	300	298	298
query70	2066	1550	1488	1488
query71	554	210	202	202
query72	5422	2814	2847	2814
query73	3256	319	320	319
query74	7061	6485	6422	6422
query75	5225	2354	2297	2297
query76	6191	1018	1069	1018
query77	714	243	237	237
query78	9107	8614	8553	8553
query79	1615	481	487	481
query80	536	316	317	316
query81	468	206	206	206
query82	192	82	74	74
query83	168	120	118	118
query84	271	71	67	67
query85	1063	318	315	315
query86	388	410	382	382
query87	3545	3366	3376	3366
query88	3023	2261	2256	2256
query89	438	355	360	355
query90	2102	188	187	187
query91	156	123	123	123
query92	58	45	42	42
query93	1364	401	427	401
query94	1236	159	153	153
query95	491	454	434	434
query96	613	328	324	324
query97	4277	4133	4147	4133
query98	207	185	181	181
query99	1009	728	720	720
Total cold run time: 293969 ms
Total hot run time: 176822 ms

doris-robot · 2024-01-16T14:45:28Z

ClickBench: Total hot run time: 30.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.02
query3	0.22	0.06	0.05
query4	1.69	0.07	0.06
query5	0.54	0.52	0.53
query6	1.31	0.66	0.64
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.55	0.50	0.49
query10	0.56	0.56	0.57
query11	0.11	0.08	0.08
query12	0.10	0.09	0.09
query13	0.60	0.61	0.60
query14	0.76	0.80	0.79
query15	0.80	0.78	0.76
query16	0.38	0.38	0.37
query17	1.02	0.99	1.03
query18	0.25	0.24	0.25
query19	1.82	1.75	1.79
query20	0.01	0.01	0.02
query21	15.41	0.59	0.59
query22	2.81	2.38	1.84
query23	17.31	0.79	0.74
query24	2.35	1.65	0.90
query25	0.41	0.22	0.20
query26	0.63	0.14	0.13
query27	0.06	0.05	0.05
query28	11.27	0.75	0.77
query29	12.52	3.10	3.15
query30	0.62	0.55	0.50
query31	2.77	0.33	0.35
query32	3.39	0.48	0.48
query33	3.25	3.24	3.24
query34	15.76	4.27	4.26
query35	4.24	4.28	4.23
query36	1.12	1.06	1.07
query37	0.07	0.05	0.06
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.16	0.13	0.13
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.19 s
Total hot run time: 30.75 s

doris-robot · 2024-01-16T14:48:31Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 838d55e9adecd7024bcf2b7aec9f4ef6aeb30380 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

doris-robot · 2024-01-16T14:53:48Z

TeamCity be ut coverage result:
Function Coverage: 36.67% (8648/23586)
Line Coverage: 28.73% (70623/245834)
Region Coverage: 27.62% (36443/131943)
Branch Coverage: 24.34% (18644/76598)
Coverage Report: http://coverage.selectdb-in.cc/coverage/838d55e9adecd7024bcf2b7aec9f4ef6aeb30380_838d55e9adecd7024bcf2b7aec9f4ef6aeb30380/report/index.html

liaoxin01

LGTM

dataroaring

LGTM

github-actions · 2024-01-17T05:55:38Z

PR approved by at least one committer and no changes requested.

…9957)

…30936)

(apache#30936)

xiaokang added p0_b dev/2.0.5 labels Jan 15, 2024

kaijchen approved these changes Jan 15, 2024

View reviewed changes

github-actions bot added the reviewed label Jan 15, 2024

liaoxin01 reviewed Jan 16, 2024

View reviewed changes

be/src/io/fs/multi_table_pipe.h Outdated Show resolved Hide resolved

liaoxin01 reviewed Jan 16, 2024

View reviewed changes

be/src/io/fs/multi_table_pipe.cpp Outdated Show resolved Hide resolved

fix multi table load can not finish

838d55e

liaoxin01 approved these changes Jan 16, 2024

View reviewed changes

dataroaring approved these changes Jan 17, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 17, 2024

dataroaring merged commit 140bfd1 into apache:master Jan 17, 2024
27 checks passed

yiguolei pushed a commit that referenced this pull request Jan 18, 2024

[fix](multi-table-load) fix multi table load can not finish (#29957)

6ef9ed0

hello-stephen pushed a commit to hello-stephen/doris that referenced this pull request Jan 19, 2024

[fix](multi-table-load) fix multi table load can not finish (apache#2…

465abad

…9957)

sollhui mentioned this pull request Feb 7, 2024

[branch-2.0](multi-table-load) fix multi table load can not finish #30936

Merged

xiaokang pushed a commit that referenced this pull request Feb 7, 2024

[fix](multi-table-load) fix multi table load can not finish #29957 (#…

e2f1ba4

…30936)

xiaokang added dev/2.0.5-merged and removed dev/2.0.5 labels Feb 7, 2024

dataroaring added the dev/3.0.0-merged label Aug 5, 2024

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024

[fix](multi-table-load) fix multi table load can not finish apache#29957

2628453

(apache#30936)

[fix](multi-table-load) fix multi table load can not finish #29957

[fix](multi-table-load) fix multi table load can not finish #29957

Conversation

sollhui commented Jan 14, 2024 • edited Loading

Proposed changes

Further comments

sollhui commented Jan 14, 2024

sollhui commented Jan 14, 2024

github-actions bot commented Jan 14, 2024

github-actions bot commented Jan 14, 2024

doris-robot commented Jan 14, 2024

doris-robot commented Jan 14, 2024

doris-robot commented Jan 14, 2024

sollhui commented Jan 14, 2024

github-actions bot commented Jan 14, 2024

doris-robot commented Jan 14, 2024

doris-robot commented Jan 14, 2024

doris-robot commented Jan 14, 2024

doris-robot commented Jan 14, 2024

doris-robot commented Jan 14, 2024

sollhui commented Jan 15, 2024

doris-robot commented Jan 15, 2024

doris-robot commented Jan 15, 2024

doris-robot commented Jan 15, 2024

github-actions bot commented Jan 15, 2024

sollhui commented Jan 16, 2024

github-actions bot commented Jan 16, 2024

doris-robot commented Jan 16, 2024

doris-robot commented Jan 16, 2024

doris-robot commented Jan 16, 2024

doris-robot commented Jan 16, 2024

doris-robot commented Jan 16, 2024

sollhui commented Jan 16, 2024

github-actions bot commented Jan 16, 2024

sollhui commented Jan 16, 2024

github-actions bot commented Jan 16, 2024

doris-robot commented Jan 16, 2024

doris-robot commented Jan 16, 2024

doris-robot commented Jan 16, 2024

doris-robot commented Jan 16, 2024

doris-robot commented Jan 16, 2024

liaoxin01 left a comment

Choose a reason for hiding this comment

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 17, 2024

sollhui commented Jan 14, 2024 •

edited

Loading