Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](decommission) decommission backend skip leaky tablets #42401

Merged
merged 9 commits into from
Nov 19, 2024

Conversation

yujun777
Copy link
Collaborator

When decommission a backend, it will get this backend's tablet meta list from TabletInvertIndex. Only after all its tablets had migrated or moved to recyle bin, then can drop this backend.

But sometimes, TabletInvertIndex may had leaky because deletting a partition forget to delete its tablet meta. After that, decommission will be blocked. So let decommission skip the leaky tablet metas, if a tablet meta couldn't found its partition, and not in catalog recyle bin, that just skip it. But for safy reason, let drop after leaky had exceed 5 hours(Config.decommission_skip_leaky_tablet_second).

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yujun777 yujun777 marked this pull request as ready for review October 24, 2024 09:00
@yujun777
Copy link
Collaborator Author

run buildall

@yujun777
Copy link
Collaborator Author

run buildall

@yujun777
Copy link
Collaborator Author

run p0

@doris-robot
Copy link

TPC-H: Total hot run time: 41074 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6f3c533b71d636dd4555ace52c2dcbfa5bef4bfb, data reload: false

------ Round 1 ----------------------------------
q1	17565	7371	7301	7301
q2	2023	278	266	266
q3	12199	1044	1228	1044
q4	10576	879	842	842
q5	7757	3085	3065	3065
q6	238	155	149	149
q7	1023	606	598	598
q8	9346	1933	1959	1933
q9	6576	6448	6430	6430
q10	7042	2410	2436	2410
q11	441	250	250	250
q12	410	219	220	219
q13	17768	2997	3017	2997
q14	232	208	206	206
q15	572	515	511	511
q16	638	590	582	582
q17	954	559	470	470
q18	7212	6810	6737	6737
q19	1352	935	974	935
q20	470	186	179	179
q21	3983	2933	2981	2933
q22	1122	1023	1017	1017
Total cold run time: 109499 ms
Total hot run time: 41074 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7331	7250	7340	7250
q2	329	231	230	230
q3	3011	3009	2940	2940
q4	2094	1896	1774	1774
q5	5728	5818	5787	5787
q6	241	154	149	149
q7	2232	1884	1799	1799
q8	3377	3552	3432	3432
q9	8957	8931	8853	8853
q10	3547	3582	3563	3563
q11	593	485	499	485
q12	833	620	636	620
q13	8681	3141	3179	3141
q14	317	292	276	276
q15	563	508	546	508
q16	702	674	626	626
q17	1864	1643	1610	1610
q18	8237	7920	7524	7524
q19	1729	1651	1515	1515
q20	2131	1880	1901	1880
q21	5679	5411	5465	5411
q22	1132	1082	1059	1059
Total cold run time: 69308 ms
Total hot run time: 60432 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192670 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6f3c533b71d636dd4555ace52c2dcbfa5bef4bfb, data reload: false

query1	907	387	410	387
query2	6265	2124	2088	2088
query3	8688	194	208	194
query4	35203	23575	23651	23575
query5	4826	488	456	456
query6	264	168	162	162
query7	4189	320	300	300
query8	298	234	237	234
query9	9198	2712	2713	2712
query10	489	276	268	268
query11	18246	15372	15342	15342
query12	153	103	97	97
query13	1558	426	411	411
query14	9486	7571	7319	7319
query15	273	172	183	172
query16	7456	483	534	483
query17	1565	590	574	574
query18	2038	317	306	306
query19	367	162	168	162
query20	120	117	117	117
query21	210	114	114	114
query22	4844	4702	4491	4491
query23	34993	34193	34311	34193
query24	10958	2780	2752	2752
query25	614	409	395	395
query26	1291	161	170	161
query27	2365	285	286	285
query28	7512	2440	2418	2418
query29	810	423	424	423
query30	265	162	172	162
query31	1055	793	790	790
query32	98	55	58	55
query33	771	285	311	285
query34	928	498	510	498
query35	1032	901	887	887
query36	1078	949	962	949
query37	152	89	86	86
query38	4430	4242	4417	4242
query39	1625	1451	1406	1406
query40	260	101	100	100
query41	48	47	46	46
query42	130	106	105	105
query43	520	490	498	490
query44	1288	792	820	792
query45	198	165	166	165
query46	1144	689	713	689
query47	1955	1832	1835	1832
query48	426	315	324	315
query49	939	433	434	433
query50	818	376	393	376
query51	7151	6977	6956	6956
query52	101	90	93	90
query53	262	182	179	179
query54	1310	423	422	422
query55	77	73	79	73
query56	257	246	248	246
query57	1299	1156	1174	1156
query58	229	231	228	228
query59	3270	3007	2976	2976
query60	290	260	262	260
query61	99	99	104	99
query62	833	675	673	673
query63	223	193	183	183
query64	5062	631	602	602
query65	3317	3237	3213	3213
query66	1418	296	302	296
query67	15864	15760	15756	15756
query68	4270	562	552	552
query69	533	295	291	291
query70	1206	1140	1139	1139
query71	332	279	285	279
query72	7216	3868	3910	3868
query73	771	356	362	356
query74	10003	9044	8917	8917
query75	3447	2681	2719	2681
query76	2915	886	873	873
query77	498	302	297	297
query78	10501	9647	9536	9536
query79	3021	617	605	605
query80	2142	433	453	433
query81	598	239	234	234
query82	739	135	138	135
query83	284	135	138	135
query84	273	67	71	67
query85	1180	300	279	279
query86	336	310	304	304
query87	4705	4663	4671	4663
query88	4158	2191	2198	2191
query89	417	292	280	280
query90	1978	186	188	186
query91	135	101	102	101
query92	67	48	48	48
query93	1482	529	543	529
query94	828	293	286	286
query95	343	244	240	240
query96	610	280	285	280
query97	2857	2719	2692	2692
query98	225	195	195	195
query99	1683	1318	1296	1296
Total cold run time: 305161 ms
Total hot run time: 192670 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.62 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6f3c533b71d636dd4555ace52c2dcbfa5bef4bfb, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.22	0.06	0.06
query4	1.65	0.10	0.10
query5	0.42	0.40	0.41
query6	1.15	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.54	0.52	0.49
query10	0.55	0.55	0.55
query11	0.14	0.10	0.11
query12	0.14	0.10	0.10
query13	0.60	0.60	0.62
query14	2.72	2.81	2.77
query15	0.90	0.81	0.82
query16	0.39	0.39	0.37
query17	1.04	1.08	1.00
query18	0.21	0.19	0.20
query19	1.97	1.87	1.93
query20	0.01	0.01	0.01
query21	15.35	0.58	0.58
query22	2.87	2.03	2.00
query23	16.84	0.97	0.86
query24	3.08	1.77	0.69
query25	0.22	0.23	0.05
query26	0.58	0.14	0.14
query27	0.04	0.03	0.04
query28	10.52	1.08	1.06
query29	12.63	3.29	3.28
query30	0.24	0.07	0.06
query31	2.87	0.39	0.38
query32	3.25	0.46	0.46
query33	3.00	3.14	3.07
query34	16.95	4.51	4.55
query35	4.52	4.56	4.51
query36	0.66	0.47	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.03	0.02
query40	0.15	0.13	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.89 s
Total hot run time: 32.62 s

@yujun777
Copy link
Collaborator Author

run p0

@yujun777 yujun777 changed the title [improvement](decommission) decommission skip leaky tablets [feature](decommission) decommission skip leaky tablets Oct 25, 2024
@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41142 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f5da66be5f1ae16b66f242d686f59ce4a94b4aa, data reload: false

------ Round 1 ----------------------------------
q1	17563	7382	7255	7255
q2	2030	274	278	274
q3	12175	1074	1141	1074
q4	10564	872	776	776
q5	7726	3060	3050	3050
q6	232	151	146	146
q7	1012	605	599	599
q8	9366	1941	1997	1941
q9	7421	6421	6403	6403
q10	7020	2409	2394	2394
q11	445	251	248	248
q12	405	216	216	216
q13	17810	3053	3019	3019
q14	240	217	212	212
q15	567	518	504	504
q16	660	589	589	589
q17	959	517	583	517
q18	7263	6664	6673	6664
q19	1334	1088	993	993
q20	479	182	185	182
q21	4043	3097	3351	3097
q22	1089	989	1001	989
Total cold run time: 110403 ms
Total hot run time: 41142 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7213	7228	7213	7213
q2	323	229	233	229
q3	3103	2938	2923	2923
q4	2150	1829	1849	1829
q5	5728	5755	5790	5755
q6	220	141	141	141
q7	2260	1805	1791	1791
q8	3363	3549	3453	3453
q9	8940	8902	8890	8890
q10	3581	3539	3539	3539
q11	584	494	505	494
q12	842	649	624	624
q13	9036	3220	3221	3220
q14	302	278	274	274
q15	569	520	515	515
q16	695	635	637	635
q17	1834	1638	1578	1578
q18	8313	7754	7634	7634
q19	1704	1584	1626	1584
q20	2133	1826	1871	1826
q21	5616	5451	5364	5364
q22	1141	1064	1055	1055
Total cold run time: 69650 ms
Total hot run time: 60566 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192040 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4f5da66be5f1ae16b66f242d686f59ce4a94b4aa, data reload: false

query1	843	391	418	391
query2	6220	2122	2041	2041
query3	8685	196	215	196
query4	34822	23731	23616	23616
query5	4929	459	444	444
query6	295	181	189	181
query7	4215	307	304	304
query8	305	247	241	241
query9	9505	2737	2740	2737
query10	480	251	253	251
query11	18212	15280	15120	15120
query12	158	103	104	103
query13	1565	429	424	424
query14	9861	6851	7319	6851
query15	243	165	187	165
query16	7449	464	509	464
query17	1450	581	565	565
query18	2050	301	297	297
query19	366	155	156	155
query20	117	112	112	112
query21	203	108	101	101
query22	4806	4486	4544	4486
query23	34691	34314	34056	34056
query24	11035	2719	2755	2719
query25	611	384	381	381
query26	1077	155	159	155
query27	2287	283	284	283
query28	7604	2435	2416	2416
query29	786	410	399	399
query30	253	153	156	153
query31	1030	795	796	795
query32	95	52	54	52
query33	750	260	265	260
query34	936	506	505	505
query35	1028	879	860	860
query36	1116	949	932	932
query37	121	76	73	73
query38	4623	4255	4200	4200
query39	1484	1462	1423	1423
query40	200	95	96	95
query41	50	47	46	46
query42	111	101	100	100
query43	545	494	500	494
query44	1337	803	798	798
query45	179	159	163	159
query46	1124	680	684	680
query47	1933	1842	1841	1841
query48	416	325	327	325
query49	899	420	411	411
query50	805	408	392	392
query51	7312	7045	7023	7023
query52	99	92	93	92
query53	258	183	182	182
query54	1094	408	414	408
query55	81	80	82	80
query56	273	260	259	259
query57	1319	1171	1159	1159
query58	240	225	219	219
query59	3249	3131	3324	3131
query60	290	263	269	263
query61	126	122	123	122
query62	873	671	686	671
query63	226	195	191	191
query64	4334	726	712	712
query65	3287	3181	3211	3181
query66	831	315	321	315
query67	16016	15702	15631	15631
query68	4827	578	570	570
query69	465	276	273	273
query70	1245	1140	1183	1140
query71	420	265	269	265
query72	6380	3958	4054	3958
query73	786	367	366	366
query74	10213	9079	8994	8994
query75	3402	2606	2635	2606
query76	2918	983	959	959
query77	418	272	286	272
query78	11121	9900	9564	9564
query79	1682	603	625	603
query80	920	433	445	433
query81	567	239	241	239
query82	646	119	118	118
query83	244	138	136	136
query84	245	73	73	73
query85	845	301	278	278
query86	453	300	278	278
query87	4815	4694	4637	4637
query88	3415	2256	2192	2192
query89	402	291	296	291
query90	2055	187	186	186
query91	134	103	103	103
query92	76	50	53	50
query93	2201	542	556	542
query94	849	283	295	283
query95	342	255	259	255
query96	622	281	293	281
query97	2864	2723	2701	2701
query98	210	197	202	197
query99	1802	1317	1310	1310
Total cold run time: 301397 ms
Total hot run time: 192040 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.65 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4f5da66be5f1ae16b66f242d686f59ce4a94b4aa, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.64	0.10	0.10
query5	0.40	0.39	0.40
query6	1.14	0.67	0.64
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.49	0.48
query10	0.55	0.54	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.59	0.59
query14	2.75	2.73	2.82
query15	0.91	0.83	0.85
query16	0.38	0.37	0.39
query17	1.01	0.98	0.98
query18	0.20	0.20	0.21
query19	1.96	1.86	1.92
query20	0.01	0.01	0.02
query21	15.68	0.60	0.57
query22	2.56	2.33	2.72
query23	17.27	1.01	0.70
query24	3.22	1.81	0.75
query25	0.33	0.10	0.08
query26	0.55	0.13	0.13
query27	0.06	0.03	0.04
query28	10.26	1.09	1.06
query29	12.55	3.30	3.32
query30	0.24	0.06	0.06
query31	2.88	0.38	0.38
query32	3.28	0.46	0.44
query33	3.00	3.05	3.02
query34	16.94	4.48	4.43
query35	4.45	4.47	4.47
query36	0.68	0.48	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.25 s
Total hot run time: 32.65 s

@yujun777
Copy link
Collaborator Author

run buildall

@yujun777
Copy link
Collaborator Author

run buildall

@yujun777
Copy link
Collaborator Author

run buildall

@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41269 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dec14aed452bf050f9a2d114cecad7ea343a78a2, data reload: false

------ Round 1 ----------------------------------
q1	18001	8773	7418	7418
q2	2053	298	167	167
q3	10531	1141	1159	1141
q4	10341	839	815	815
q5	7771	3123	3053	3053
q6	237	149	144	144
q7	1024	611	592	592
q8	9358	1933	1996	1933
q9	6684	6483	6466	6466
q10	7080	2413	2462	2413
q11	449	250	244	244
q12	405	215	215	215
q13	17792	2997	3008	2997
q14	243	221	213	213
q15	562	510	510	510
q16	648	579	586	579
q17	971	610	529	529
q18	7408	6830	6702	6702
q19	1336	994	940	940
q20	483	191	179	179
q21	3987	3177	3000	3000
q22	1150	1039	1019	1019
Total cold run time: 108514 ms
Total hot run time: 41269 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7313	7304	7334	7304
q2	325	234	235	234
q3	3085	2994	2956	2956
q4	2136	1863	1885	1863
q5	5716	5752	5787	5752
q6	221	141	138	138
q7	2265	1831	1804	1804
q8	3412	3536	3463	3463
q9	8900	8903	8905	8903
q10	3615	3561	3579	3561
q11	605	490	496	490
q12	845	633	653	633
q13	7503	3163	3224	3163
q14	299	274	294	274
q15	566	534	530	530
q16	688	664	646	646
q17	1846	1619	1603	1603
q18	8527	7882	7542	7542
q19	1715	1589	1498	1498
q20	2141	1901	1849	1849
q21	5673	5449	5607	5449
q22	1183	1088	1057	1057
Total cold run time: 68579 ms
Total hot run time: 60712 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191766 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dec14aed452bf050f9a2d114cecad7ea343a78a2, data reload: false

query1	856	390	433	390
query2	6250	2058	2034	2034
query3	8677	193	207	193
query4	35493	23588	23709	23588
query5	4799	456	447	447
query6	304	179	196	179
query7	4198	299	293	293
query8	303	238	239	238
query9	9413	2680	2667	2667
query10	495	252	246	246
query11	18368	15331	15284	15284
query12	149	106	98	98
query13	1593	447	404	404
query14	9580	6694	7445	6694
query15	288	170	178	170
query16	7394	464	470	464
query17	1551	589	585	585
query18	2001	301	315	301
query19	382	153	163	153
query20	134	114	113	113
query21	207	106	107	106
query22	4911	4683	4438	4438
query23	34972	33893	34181	33893
query24	11008	2755	2719	2719
query25	600	379	377	377
query26	920	153	165	153
query27	2255	278	280	278
query28	7451	2415	2413	2413
query29	733	401	399	399
query30	254	157	159	157
query31	989	819	801	801
query32	93	61	56	56
query33	754	267	261	261
query34	927	516	515	515
query35	1063	875	888	875
query36	1112	953	952	952
query37	119	72	72	72
query38	4539	4188	4185	4185
query39	1466	1432	1406	1406
query40	197	97	98	97
query41	48	48	46	46
query42	112	99	95	95
query43	527	477	511	477
query44	1232	821	850	821
query45	179	166	163	163
query46	1137	702	692	692
query47	1976	1860	1862	1860
query48	448	318	316	316
query49	885	407	416	407
query50	799	383	386	383
query51	7201	7052	7006	7006
query52	105	90	91	90
query53	249	190	193	190
query54	1230	410	406	406
query55	83	75	75	75
query56	275	256	252	252
query57	1302	1191	1179	1179
query58	228	212	211	211
query59	3182	3041	2922	2922
query60	293	253	260	253
query61	147	126	122	122
query62	864	673	669	669
query63	233	191	184	184
query64	4156	717	680	680
query65	3296	3239	3199	3199
query66	870	332	314	314
query67	16096	15896	15794	15794
query68	4371	567	563	563
query69	435	272	272	272
query70	1252	1104	1166	1104
query71	351	272	262	262
query72	6306	3959	4022	3959
query73	745	356	366	356
query74	10295	9106	9160	9106
query75	3407	2645	2668	2645
query76	2629	915	945	915
query77	442	286	286	286
query78	10655	9619	9645	9619
query79	2546	594	628	594
query80	1068	418	436	418
query81	567	241	244	241
query82	628	116	116	116
query83	232	141	131	131
query84	246	72	68	68
query85	1575	306	289	289
query86	451	297	305	297
query87	4965	4740	4720	4720
query88	4006	2190	2179	2179
query89	401	299	288	288
query90	2057	187	185	185
query91	137	110	102	102
query92	67	48	49	48
query93	2296	536	524	524
query94	941	267	295	267
query95	351	245	245	245
query96	603	274	283	274
query97	2866	2751	2690	2690
query98	220	202	204	202
query99	1528	1321	1301	1301
Total cold run time: 302590 ms
Total hot run time: 191766 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dec14aed452bf050f9a2d114cecad7ea343a78a2, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.22	0.07	0.07
query4	1.65	0.10	0.10
query5	0.41	0.41	0.39
query6	1.16	0.66	0.65
query7	0.02	0.01	0.02
query8	0.03	0.03	0.03
query9	0.57	0.50	0.50
query10	0.54	0.55	0.54
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.63	0.60
query14	2.69	2.75	2.76
query15	0.90	0.83	0.84
query16	0.39	0.38	0.38
query17	0.97	1.09	1.06
query18	0.20	0.19	0.20
query19	1.94	1.79	1.94
query20	0.01	0.01	0.01
query21	15.35	0.59	0.57
query22	2.48	1.75	2.17
query23	17.12	0.83	0.84
query24	3.32	1.59	0.67
query25	0.18	0.20	0.14
query26	0.56	0.14	0.14
query27	0.04	0.04	0.04
query28	10.52	1.09	1.07
query29	12.55	3.32	3.34
query30	0.25	0.06	0.06
query31	2.88	0.38	0.38
query32	3.28	0.46	0.45
query33	2.97	2.99	3.01
query34	17.53	4.49	4.50
query35	4.50	4.47	4.51
query36	0.69	0.48	0.50
query37	0.08	0.05	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 107.38 s
Total hot run time: 32.3 s

@yujun777
Copy link
Collaborator Author

run p0

2 similar comments
@yujun777
Copy link
Collaborator Author

run p0

@yujun777
Copy link
Collaborator Author

run p0

@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41867 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f293ec3e4ac8212ae4650974b3bf7310c9a07cb, data reload: false

------ Round 1 ----------------------------------
q1	17733	7593	7368	7368
q2	2060	167	178	167
q3	10698	1088	1231	1088
q4	10420	837	854	837
q5	7759	3084	3107	3084
q6	238	146	150	146
q7	1021	607	623	607
q8	9354	1977	2056	1977
q9	6654	6500	6494	6494
q10	7038	2445	2416	2416
q11	451	255	248	248
q12	414	220	212	212
q13	17793	3042	3056	3042
q14	242	211	207	207
q15	579	529	526	526
q16	660	602	605	602
q17	988	563	580	563
q18	7592	6824	6772	6772
q19	1348	1096	1111	1096
q20	484	182	185	182
q21	4127	3231	3223	3223
q22	1119	1010	1014	1010
Total cold run time: 108772 ms
Total hot run time: 41867 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7390	7332	7296	7296
q2	326	231	219	219
q3	3111	2981	3032	2981
q4	2105	1859	1808	1808
q5	5760	5784	5834	5784
q6	226	140	139	139
q7	2260	1819	1776	1776
q8	3495	3511	3498	3498
q9	9062	8998	8905	8905
q10	3621	3577	3572	3572
q11	588	491	511	491
q12	837	647	585	585
q13	8510	3232	3222	3222
q14	307	273	288	273
q15	589	529	510	510
q16	707	629	667	629
q17	1873	1622	1623	1622
q18	8340	7898	7668	7668
q19	1746	1642	1669	1642
q20	2158	1874	1873	1873
q21	5622	5552	5503	5503
q22	1097	1099	1067	1067
Total cold run time: 69730 ms
Total hot run time: 61063 ms

@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41174 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fec95b17ecccfca985b4dc46b92d240837e8457a, data reload: false

------ Round 1 ----------------------------------
q1	17578	7388	7291	7291
q2	2060	162	164	162
q3	10585	1093	1236	1093
q4	10585	874	873	873
q5	7726	3077	3061	3061
q6	233	144	140	140
q7	1032	608	582	582
q8	9337	1981	1931	1931
q9	6591	6401	6497	6401
q10	7135	2419	2428	2419
q11	445	250	249	249
q12	404	212	210	210
q13	17769	2970	2994	2970
q14	242	206	207	206
q15	570	528	503	503
q16	669	592	594	592
q17	977	603	555	555
q18	7203	6880	6728	6728
q19	1318	912	977	912
q20	452	185	177	177
q21	4043	3122	3267	3122
q22	1113	1004	997	997
Total cold run time: 108067 ms
Total hot run time: 41174 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7247	7239	7281	7239
q2	320	223	222	222
q3	2964	2935	2974	2935
q4	2076	1848	1812	1812
q5	5717	5734	5775	5734
q6	230	140	143	140
q7	2230	1822	1830	1822
q8	3393	3528	3464	3464
q9	8942	8931	8906	8906
q10	3582	3520	3528	3520
q11	589	503	510	503
q12	841	625	589	589
q13	9755	3166	3246	3166
q14	300	278	298	278
q15	577	518	509	509
q16	677	659	665	659
q17	1843	1626	1642	1626
q18	8305	7705	7631	7631
q19	1700	1532	1607	1532
q20	2153	1852	1847	1847
q21	5618	5413	5420	5413
q22	1136	1055	1033	1033
Total cold run time: 70195 ms
Total hot run time: 60580 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196246 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fec95b17ecccfca985b4dc46b92d240837e8457a, data reload: false

query1	1242	998	973	973
query2	6234	2058	2014	2014
query3	11436	4561	4746	4561
query4	37893	23601	23577	23577
query5	4587	440	433	433
query6	276	184	181	181
query7	3985	290	291	290
query8	288	231	223	223
query9	9429	2625	2608	2608
query10	456	261	236	236
query11	16660	15326	15347	15326
query12	154	100	100	100
query13	1592	443	421	421
query14	8605	7122	6981	6981
query15	257	181	179	179
query16	7999	474	495	474
query17	1359	590	563	563
query18	2120	289	285	285
query19	229	147	158	147
query20	121	105	112	105
query21	199	101	98	98
query22	4653	4579	4467	4467
query23	34814	34362	34039	34039
query24	11036	2754	2791	2754
query25	566	372	359	359
query26	1232	157	153	153
query27	2715	277	267	267
query28	8001	2411	2395	2395
query29	702	408	388	388
query30	262	160	164	160
query31	1012	799	826	799
query32	89	50	54	50
query33	716	263	263	263
query34	934	515	519	515
query35	1003	886	864	864
query36	1106	937	944	937
query37	113	66	72	66
query38	4461	4242	4212	4212
query39	1473	1418	1413	1413
query40	255	96	110	96
query41	44	41	42	41
query42	102	93	102	93
query43	522	476	483	476
query44	1255	784	792	784
query45	181	160	159	159
query46	1125	681	704	681
query47	1955	1822	1834	1822
query48	406	306	309	306
query49	997	366	388	366
query50	805	390	388	388
query51	7099	6995	6949	6949
query52	97	88	91	88
query53	247	174	176	174
query54	1063	380	391	380
query55	84	76	73	73
query56	239	233	234	233
query57	1282	1166	1144	1144
query58	206	200	197	197
query59	3223	2973	3108	2973
query60	260	235	236	235
query61	99	99	99	99
query62	848	695	669	669
query63	213	185	183	183
query64	4862	712	688	688
query65	3268	3212	3221	3212
query66	1118	327	316	316
query67	16103	15628	15608	15608
query68	4418	565	547	547
query69	430	270	261	261
query70	1206	1146	1161	1146
query71	314	263	250	250
query72	6189	4176	4040	4040
query73	755	359	356	356
query74	10214	9016	8849	8849
query75	3438	2656	2717	2656
query76	2263	1113	986	986
query77	377	278	293	278
query78	10604	9511	9543	9511
query79	1252	591	620	591
query80	1099	443	451	443
query81	581	243	240	240
query82	1122	116	107	107
query83	241	135	137	135
query84	242	67	66	66
query85	1484	294	286	286
query86	439	295	292	292
query87	4789	4610	4705	4610
query88	3358	2195	2156	2156
query89	423	290	281	281
query90	1961	181	181	181
query91	125	99	96	96
query92	60	47	48	47
query93	1342	546	534	534
query94	923	273	287	273
query95	347	240	236	236
query96	616	277	289	277
query97	2896	2728	2682	2682
query98	209	206	202	202
query99	1520	1322	1349	1322
Total cold run time: 303204 ms
Total hot run time: 196246 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.61 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fec95b17ecccfca985b4dc46b92d240837e8457a, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.03
query3	0.23	0.06	0.06
query4	1.65	0.11	0.10
query5	0.44	0.40	0.39
query6	1.16	0.67	0.65
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.51	0.50
query10	0.55	0.55	0.57
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.60	0.60	0.60
query14	2.82	2.84	2.70
query15	0.92	0.83	0.83
query16	0.39	0.38	0.39
query17	1.05	1.05	1.04
query18	0.20	0.20	0.20
query19	1.83	1.88	1.99
query20	0.01	0.00	0.01
query21	15.36	0.58	0.57
query22	2.87	1.86	2.19
query23	17.22	0.96	0.73
query24	3.25	1.30	1.03
query25	0.27	0.17	0.11
query26	0.43	0.14	0.13
query27	0.03	0.05	0.04
query28	10.41	1.10	1.07
query29	12.53	3.26	3.26
query30	0.25	0.06	0.08
query31	2.84	0.38	0.39
query32	3.27	0.46	0.46
query33	2.98	2.99	2.98
query34	17.09	4.40	4.48
query35	4.46	4.53	4.55
query36	0.67	0.49	0.49
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.15	0.13	0.13
query41	0.07	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.24 s
Total hot run time: 32.61 s

Copy link
Contributor

@deardeng deardeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yujun777
Copy link
Collaborator Author

run buildall

Copy link
Contributor

PR approved by anyone and no changes requested.

@yujun777
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41158 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0247b2ee429281ef7c2332fe2ea9fb4418c2afb7, data reload: false

------ Round 1 ----------------------------------
q1	17923	7576	7282	7282
q2	2060	168	159	159
q3	10846	1087	1175	1087
q4	10388	833	840	833
q5	7775	3102	3077	3077
q6	238	143	143	143
q7	1013	609	623	609
q8	9348	1965	2013	1965
q9	6618	6450	6489	6450
q10	7066	2479	2420	2420
q11	456	253	251	251
q12	409	221	219	219
q13	17786	2972	2975	2972
q14	243	213	211	211
q15	576	515	515	515
q16	636	588	570	570
q17	982	514	555	514
q18	7365	6599	6751	6599
q19	1439	1012	1006	1006
q20	477	184	182	182
q21	4014	3111	3125	3111
q22	1108	983	1003	983
Total cold run time: 108766 ms
Total hot run time: 41158 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7301	7289	7247	7247
q2	321	226	232	226
q3	2984	2938	2994	2938
q4	2065	1892	1811	1811
q5	5717	5761	5794	5761
q6	229	140	143	140
q7	2485	1818	1810	1810
q8	3366	3542	3418	3418
q9	8888	8947	8908	8908
q10	3588	3596	3596	3596
q11	589	494	481	481
q12	818	682	615	615
q13	9338	3204	3212	3204
q14	322	270	285	270
q15	579	540	532	532
q16	682	662	651	651
q17	1862	1637	1593	1593
q18	8368	7833	7708	7708
q19	1724	1482	1614	1482
q20	2105	1888	1881	1881
q21	5519	5442	5412	5412
q22	1164	1098	1026	1026
Total cold run time: 70014 ms
Total hot run time: 60710 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195373 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0247b2ee429281ef7c2332fe2ea9fb4418c2afb7, data reload: false

query1	1219	892	886	886
query2	6246	2076	2077	2076
query3	10795	3960	3887	3887
query4	68220	29388	23666	23666
query5	4936	449	433	433
query6	408	186	175	175
query7	5616	285	288	285
query8	314	222	226	222
query9	9134	2745	2722	2722
query10	461	257	243	243
query11	17770	15395	16067	15395
query12	157	102	103	102
query13	1593	442	430	430
query14	10432	6545	7038	6545
query15	215	179	189	179
query16	6951	456	442	442
query17	995	530	527	527
query18	1752	295	281	281
query19	193	149	149	149
query20	115	106	107	106
query21	197	98	96	96
query22	4611	4229	4452	4229
query23	34228	34881	33980	33980
query24	5942	2729	2734	2729
query25	477	364	377	364
query26	637	155	151	151
query27	1688	280	288	280
query28	4200	2479	2472	2472
query29	677	432	420	420
query30	230	151	159	151
query31	976	787	833	787
query32	65	57	57	57
query33	413	266	283	266
query34	917	499	508	499
query35	857	749	739	739
query36	1086	942	944	942
query37	120	71	71	71
query38	4283	4362	4313	4313
query39	1480	1418	1434	1418
query40	202	96	99	96
query41	47	44	50	44
query42	109	103	96	96
query43	542	481	495	481
query44	1153	808	809	808
query45	180	166	166	166
query46	1126	687	708	687
query47	1997	1896	1849	1849
query48	403	319	333	319
query49	732	402	398	398
query50	806	394	410	394
query51	7300	7270	7127	7127
query52	98	92	90	90
query53	249	177	184	177
query54	506	407	402	402
query55	73	75	79	75
query56	258	235	251	235
query57	1301	1220	1200	1200
query58	224	225	203	203
query59	3260	3089	2907	2907
query60	278	244	258	244
query61	121	122	119	119
query62	790	671	690	671
query63	218	189	186	186
query64	1429	725	673	673
query65	3258	3187	3191	3187
query66	715	305	300	300
query67	15888	15666	15584	15584
query68	3904	584	585	584
query69	411	248	245	245
query70	1154	1158	1142	1142
query71	349	255	250	250
query72	6060	3988	3913	3913
query73	752	353	349	349
query74	10204	8934	9142	8934
query75	3382	2646	2657	2646
query76	1816	1099	1050	1050
query77	492	272	268	268
query78	10348	9439	9418	9418
query79	2020	590	594	590
query80	1327	422	422	422
query81	545	237	242	237
query82	701	113	110	110
query83	158	141	137	137
query84	288	69	69	69
query85	1044	294	282	282
query86	471	302	298	298
query87	4934	4756	4727	4727
query88	3875	2192	2141	2141
query89	416	294	286	286
query90	2037	181	177	177
query91	127	99	100	99
query92	69	47	48	47
query93	2779	553	548	548
query94	904	297	295	295
query95	345	243	247	243
query96	630	280	274	274
query97	2908	2719	2669	2669
query98	215	198	203	198
query99	1608	1322	1292	1292
Total cold run time: 320142 ms
Total hot run time: 195373 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.59 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0247b2ee429281ef7c2332fe2ea9fb4418c2afb7, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.02
query3	0.23	0.06	0.06
query4	1.63	0.10	0.10
query5	0.41	0.40	0.41
query6	1.17	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.58	0.50	0.48
query10	0.56	0.54	0.55
query11	0.14	0.11	0.11
query12	0.16	0.12	0.11
query13	0.61	0.60	0.60
query14	2.73	2.80	2.81
query15	0.90	0.83	0.81
query16	0.38	0.39	0.39
query17	1.04	0.97	0.96
query18	0.20	0.20	0.20
query19	1.94	1.85	1.93
query20	0.02	0.01	0.01
query21	15.36	0.60	0.56
query22	2.71	2.19	1.53
query23	16.99	1.08	0.90
query24	2.96	1.24	2.23
query25	0.14	0.23	0.08
query26	0.59	0.14	0.14
query27	0.05	0.04	0.04
query28	9.63	1.10	1.08
query29	12.56	3.26	3.27
query30	0.24	0.06	0.06
query31	2.88	0.39	0.37
query32	3.26	0.46	0.46
query33	2.93	3.01	3.06
query34	17.06	4.47	4.50
query35	4.48	4.53	4.46
query36	0.66	0.47	0.48
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.8 s
Total hot run time: 32.59 s

@yujun777 yujun777 changed the title [feature](decommission) decommission skip leaky tablets [feature](decommission) decommission backend skip leaky tablets Oct 31, 2024
@yujun777
Copy link
Collaborator Author

yujun777 commented Nov 4, 2024

run buildall

@yujun777
Copy link
Collaborator Author

yujun777 commented Nov 4, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41755 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e827b0023442d836ee7414708415c7a333f50b56, data reload: false

------ Round 1 ----------------------------------
q1	17578	7763	7316	7316
q2	2069	173	156	156
q3	10572	1098	1204	1098
q4	10566	901	893	893
q5	7748	3106	3104	3104
q6	230	142	147	142
q7	1017	606	604	604
q8	9348	2004	2057	2004
q9	6574	6482	6493	6482
q10	7094	2451	2466	2451
q11	472	271	260	260
q12	414	215	218	215
q13	17774	3054	3000	3000
q14	246	210	218	210
q15	584	512	522	512
q16	657	578	588	578
q17	988	538	467	467
q18	7326	6832	6802	6802
q19	1340	1054	973	973
q20	461	179	184	179
q21	4012	3374	3304	3304
q22	1128	1054	1005	1005
Total cold run time: 108198 ms
Total hot run time: 41755 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7346	7311	7299	7299
q2	323	229	222	222
q3	3060	2942	2991	2942
q4	2192	1856	1827	1827
q5	5745	5795	5810	5795
q6	224	141	140	140
q7	2240	1790	1800	1790
q8	3432	3567	3478	3478
q9	8986	8961	8915	8915
q10	3576	3598	3586	3586
q11	603	498	531	498
q12	813	624	620	620
q13	9735	3165	3161	3161
q14	311	276	285	276
q15	595	531	535	531
q16	687	647	636	636
q17	1855	1638	1647	1638
q18	8354	8008	7647	7647
q19	1750	1540	1535	1535
q20	2124	1826	1881	1826
q21	5588	5426	5365	5365
q22	1173	1047	1073	1047
Total cold run time: 70712 ms
Total hot run time: 60774 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 196390 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e827b0023442d836ee7414708415c7a333f50b56, data reload: false

query1	1208	916	892	892
query2	6232	2084	2149	2084
query3	10955	4041	4077	4041
query4	67912	29062	23588	23588
query5	4907	452	439	439
query6	396	174	164	164
query7	5553	290	289	289
query8	320	222	220	220
query9	8837	2681	2682	2681
query10	427	261	265	261
query11	17334	15308	16003	15308
query12	154	108	106	106
query13	1483	443	429	429
query14	10721	7179	6830	6830
query15	215	198	198	198
query16	7133	491	492	491
query17	1039	561	578	561
query18	1820	312	295	295
query19	206	150	150	150
query20	118	116	113	113
query21	195	101	100	100
query22	4614	4436	4241	4241
query23	34578	34131	34203	34131
query24	6037	2792	2760	2760
query25	515	409	407	407
query26	649	162	160	160
query27	1647	288	291	288
query28	4346	2432	2420	2420
query29	696	428	428	428
query30	226	156	156	156
query31	992	814	827	814
query32	65	55	55	55
query33	469	275	275	275
query34	904	507	522	507
query35	843	747	733	733
query36	1063	970	959	959
query37	122	72	81	72
query38	4481	4341	4252	4252
query39	1457	1490	1469	1469
query40	201	100	96	96
query41	47	44	46	44
query42	108	97	96	96
query43	534	504	500	500
query44	1167	815	812	812
query45	189	168	167	167
query46	1124	723	716	716
query47	1971	1875	1877	1875
query48	417	319	320	319
query49	732	405	406	405
query50	832	401	396	396
query51	7164	7260	7151	7151
query52	101	89	89	89
query53	258	180	179	179
query54	532	396	419	396
query55	79	72	74	72
query56	243	233	244	233
query57	1265	1222	1175	1175
query58	216	207	204	204
query59	3159	3080	2918	2918
query60	266	247	246	246
query61	107	104	106	104
query62	777	663	683	663
query63	213	187	184	184
query64	1338	646	613	613
query65	3282	3214	3246	3214
query66	697	317	311	311
query67	15842	15902	15687	15687
query68	3884	569	568	568
query69	408	255	253	253
query70	1191	1164	1130	1130
query71	325	256	262	256
query72	6357	4016	4007	4007
query73	767	360	364	360
query74	10095	9154	8988	8988
query75	3408	2701	2687	2687
query76	2117	1191	1058	1058
query77	473	271	277	271
query78	10340	9430	9396	9396
query79	1270	576	588	576
query80	846	408	434	408
query81	505	231	239	231
query82	1279	119	112	112
query83	154	147	137	137
query84	277	71	67	67
query85	857	291	303	291
query86	340	301	293	293
query87	4924	4794	4761	4761
query88	3514	2225	2171	2171
query89	405	293	291	291
query90	2014	186	180	180
query91	136	103	104	103
query92	66	49	53	49
query93	1633	539	533	533
query94	820	297	294	294
query95	352	243	252	243
query96	615	287	285	285
query97	2888	2753	2686	2686
query98	214	200	191	191
query99	1579	1282	1315	1282
Total cold run time: 317783 ms
Total hot run time: 196390 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.66 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e827b0023442d836ee7414708415c7a333f50b56, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.04	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.09
query5	0.42	0.39	0.40
query6	1.14	0.67	0.66
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.55	0.51	0.50
query10	0.54	0.55	0.56
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.61	0.60	0.58
query14	2.68	2.79	2.71
query15	0.89	0.82	0.82
query16	0.38	0.38	0.39
query17	1.05	1.08	1.02
query18	0.20	0.20	0.20
query19	1.95	1.87	1.97
query20	0.02	0.01	0.01
query21	15.36	0.60	0.56
query22	2.68	2.33	1.57
query23	17.13	0.85	0.78
query24	2.60	1.26	1.58
query25	0.20	0.10	0.10
query26	0.61	0.13	0.14
query27	0.05	0.04	0.04
query28	10.34	1.09	1.07
query29	12.56	3.28	3.27
query30	0.24	0.06	0.05
query31	2.88	0.38	0.37
query32	3.32	0.46	0.46
query33	3.00	3.00	3.04
query34	16.85	4.59	4.53
query35	4.58	4.53	4.54
query36	0.66	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.13	0.12
query41	0.09	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 106.28 s
Total hot run time: 32.66 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 19, 2024
@dataroaring dataroaring merged commit 956da4c into apache:master Nov 19, 2024
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants