Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement](cloud) improve the retry policy of cloud mode #49067

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

luwei16
Copy link
Contributor

@luwei16 luwei16 commented Mar 14, 2025

No description provided.

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@luwei16
Copy link
Contributor Author

luwei16 commented Mar 14, 2025

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.76% (1075/1299)
Line Coverage: 65.63% (17696/26965)
Region Coverage: 65.17% (8702/13352)
Branch Coverage: 55.18% (4695/8508)
Coverage Report: http://coverage.selectdb-in.cc/coverage/44a532ca57e211184a1d4327fbd453bf2296a970_44a532ca57e211184a1d4327fbd453bf2296a970_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 32512 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 44a532ca57e211184a1d4327fbd453bf2296a970, data reload: false

------ Round 1 ----------------------------------
q1	17600	5165	5023	5023
q2	2050	283	170	170
q3	10502	1287	730	730
q4	10219	1014	540	540
q5	7620	2409	2370	2370
q6	193	163	132	132
q7	917	745	619	619
q8	9313	1295	1119	1119
q9	4952	4790	4807	4790
q10	6810	2312	1900	1900
q11	485	279	250	250
q12	353	357	215	215
q13	17752	3701	3069	3069
q14	242	233	205	205
q15	541	479	471	471
q16	623	629	591	591
q17	581	878	352	352
q18	6939	6550	6326	6326
q19	2022	952	550	550
q20	318	339	194	194
q21	2716	2154	1913	1913
q22	1059	1017	983	983
Total cold run time: 103807 ms
Total hot run time: 32512 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5199	5098	5094	5094
q2	241	328	235	235
q3	2156	2647	2282	2282
q4	1425	1821	1434	1434
q5	4228	4129	4173	4129
q6	207	165	122	122
q7	1868	1918	1752	1752
q8	2616	2643	2580	2580
q9	7323	7149	7203	7149
q10	3009	3244	2796	2796
q11	583	506	501	501
q12	710	754	599	599
q13	3350	3974	3242	3242
q14	273	285	293	285
q15	536	488	473	473
q16	659	715	659	659
q17	1172	1598	1356	1356
q18	7927	7757	7508	7508
q19	803	819	949	819
q20	2058	2029	1941	1941
q21	5463	5043	4690	4690
q22	1083	1088	1019	1019
Total cold run time: 52889 ms
Total hot run time: 50665 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192688 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 44a532ca57e211184a1d4327fbd453bf2296a970, data reload: false

query1	1447	1020	994	994
query2	6331	2002	1969	1969
query3	10950	4668	4594	4594
query4	53493	24742	23488	23488
query5	5245	567	476	476
query6	353	203	188	188
query7	4983	502	285	285
query8	329	232	214	214
query9	6059	2625	2586	2586
query10	417	306	258	258
query11	15197	15035	15025	15025
query12	162	107	102	102
query13	1114	500	410	410
query14	10512	6616	6388	6388
query15	207	198	175	175
query16	7036	645	497	497
query17	1056	700	563	563
query18	1518	416	340	340
query19	196	190	163	163
query20	129	129	117	117
query21	211	124	104	104
query22	4536	4671	4316	4316
query23	34034	33278	33360	33278
query24	5637	2418	2386	2386
query25	468	471	389	389
query26	678	273	154	154
query27	1898	485	329	329
query28	2804	2451	2446	2446
query29	573	576	453	453
query30	271	222	206	206
query31	872	904	775	775
query32	79	70	94	70
query33	471	362	296	296
query34	764	848	501	501
query35	819	837	761	761
query36	970	1007	934	934
query37	125	99	78	78
query38	4177	4189	4122	4122
query39	1542	1452	1436	1436
query40	210	117	104	104
query41	52	52	52	52
query42	126	105	109	105
query43	536	555	493	493
query44	1302	797	794	794
query45	184	171	161	161
query46	851	1044	641	641
query47	1857	1862	1768	1768
query48	388	448	320	320
query49	750	514	450	450
query50	688	738	411	411
query51	4282	4321	4245	4245
query52	107	109	97	97
query53	236	259	191	191
query54	485	502	417	417
query55	84	80	82	80
query56	268	281	274	274
query57	1137	1147	1103	1103
query58	240	242	233	233
query59	2813	2897	2930	2897
query60	297	270	265	265
query61	129	145	152	145
query62	761	741	684	684
query63	229	198	187	187
query64	1597	1139	807	807
query65	4504	4445	4406	4406
query66	791	394	288	288
query67	15937	15558	15320	15320
query68	7010	876	491	491
query69	542	301	251	251
query70	1158	1118	1114	1114
query71	492	292	266	266
query72	5540	3838	3795	3795
query73	1427	725	351	351
query74	8986	9113	8811	8811
query75	3878	3126	2704	2704
query76	4257	1190	749	749
query77	699	359	268	268
query78	9961	10285	9302	9302
query79	2259	820	574	574
query80	778	519	495	495
query81	465	260	217	217
query82	542	128	93	93
query83	292	171	151	151
query84	276	92	70	70
query85	785	361	312	312
query86	341	303	307	303
query87	4493	4473	4488	4473
query88	3234	2240	2237	2237
query89	410	307	275	275
query90	1939	210	211	210
query91	139	141	112	112
query92	74	68	58	58
query93	1175	1038	586	586
query94	696	405	301	301
query95	348	270	255	255
query96	493	550	270	270
query97	3375	3389	3272	3272
query98	227	203	201	201
query99	1426	1397	1317	1317
Total cold run time: 296221 ms
Total hot run time: 192688 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.38 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 44a532ca57e211184a1d4327fbd453bf2296a970, data reload: false

query1	0.04	0.03	0.03
query2	0.10	0.05	0.05
query3	0.28	0.05	0.05
query4	1.59	0.09	0.09
query5	0.55	0.53	0.54
query6	1.19	0.71	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.62	0.51	0.53
query10	0.58	0.59	0.58
query11	0.25	0.12	0.12
query12	0.24	0.12	0.13
query13	0.64	0.62	0.62
query14	2.69	2.81	2.67
query15	0.99	0.87	0.87
query16	0.38	0.37	0.38
query17	1.00	1.06	1.03
query18	0.18	0.18	0.17
query19	1.93	1.92	1.96
query20	0.01	0.01	0.02
query21	15.37	0.95	0.66
query22	0.92	0.97	0.78
query23	14.73	1.56	0.74
query24	5.10	0.63	0.31
query25	0.17	0.09	0.08
query26	0.55	0.21	0.18
query27	0.08	0.08	0.08
query28	11.04	1.13	0.56
query29	12.55	4.02	3.37
query30	0.28	0.08	0.06
query31	2.80	0.62	0.44
query32	3.23	0.59	0.50
query33	2.99	3.11	3.09
query34	16.63	5.11	4.44
query35	4.43	4.46	4.51
query36	0.63	0.52	0.48
query37	0.21	0.17	0.18
query38	0.17	0.14	0.15
query39	0.05	0.03	0.04
query40	0.19	0.16	0.16
query41	0.11	0.05	0.06
query42	0.07	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 105.68 s
Total hot run time: 31.38 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 47.97% (12825/26738)
Line Coverage 37.35% (109427/292983)
Region Coverage 36.48% (56082/153722)
Branch Coverage 31.77% (28258/88956)

int get_dynamic_retry_count() {
int64_t busyness_percent = get_fdb_client_thread_busyness_percent();
if (busyness_percent > config::retry_disable_busyness_threshold) {
return 0;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add bvar to record reduce and disable count
consider latency recorder

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@@ -1402,6 +1402,8 @@ enum MetaServiceCode {

SCHEMA_DICT_NOT_FOUND = 11001;

KV_TXN_CONFLICT_BUSY = 12001;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comment

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@luwei16
Copy link
Contributor Author

luwei16 commented Mar 19, 2025

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.96% (1081/1303)
Line Coverage: 65.80% (17957/27291)
Region Coverage: 65.38% (8834/13512)
Branch Coverage: 55.24% (4755/8608)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f121af4670e959a57a016ee6bcd694200d60f21e_f121af4670e959a57a016ee6bcd694200d60f21e_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 32545 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f121af4670e959a57a016ee6bcd694200d60f21e, data reload: false

------ Round 1 ----------------------------------
q1	24247	5065	5061	5061
q2	2048	322	180	180
q3	10382	1270	677	677
q4	10231	1022	539	539
q5	7583	2480	2337	2337
q6	202	167	131	131
q7	928	727	622	622
q8	9297	1300	1111	1111
q9	4931	4769	4888	4769
q10	6815	2312	1916	1916
q11	464	267	256	256
q12	344	355	220	220
q13	17778	3689	3077	3077
q14	223	224	219	219
q15	540	479	482	479
q16	609	624	580	580
q17	594	880	354	354
q18	7217	6587	6374	6374
q19	1867	1127	576	576
q20	316	341	195	195
q21	2811	2423	1918	1918
q22	1018	1010	954	954
Total cold run time: 110445 ms
Total hot run time: 32545 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5214	5160	5163	5160
q2	244	330	229	229
q3	2143	2675	2282	2282
q4	1430	1808	1384	1384
q5	4292	4254	4665	4254
q6	212	167	128	128
q7	2067	1929	1824	1824
q8	2599	2673	2838	2673
q9	7223	7222	7017	7017
q10	2956	3231	2785	2785
q11	581	512	484	484
q12	668	736	606	606
q13	3589	3825	3203	3203
q14	282	293	273	273
q15	524	471	471	471
q16	678	664	672	664
q17	1138	1592	1352	1352
q18	7809	7511	7561	7511
q19	835	882	1095	882
q20	1954	1997	1874	1874
q21	5426	4970	4589	4589
q22	1059	989	987	987
Total cold run time: 52923 ms
Total hot run time: 50632 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184679 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f121af4670e959a57a016ee6bcd694200d60f21e, data reload: false

query1	1034	511	475	475
query2	6547	1883	1846	1846
query3	6808	228	220	220
query4	26275	23576	23395	23395
query5	4361	671	478	478
query6	306	204	213	204
query7	4599	500	298	298
query8	302	251	246	246
query9	8644	2573	2579	2573
query10	504	302	259	259
query11	15437	15106	14917	14917
query12	158	111	105	105
query13	1655	525	404	404
query14	9810	6020	6169	6020
query15	200	189	168	168
query16	7394	643	488	488
query17	1200	702	568	568
query18	1966	405	310	310
query19	201	193	160	160
query20	125	120	117	117
query21	213	121	105	105
query22	4197	4305	4129	4129
query23	33857	33127	33235	33127
query24	7747	2385	2334	2334
query25	576	437	388	388
query26	1238	269	149	149
query27	2616	460	328	328
query28	4377	2404	2416	2404
query29	758	586	430	430
query30	283	222	198	198
query31	978	900	750	750
query32	71	64	69	64
query33	555	354	308	308
query34	790	830	491	491
query35	797	819	747	747
query36	956	982	894	894
query37	117	100	73	73
query38	4287	4312	4106	4106
query39	1437	1411	1403	1403
query40	216	116	105	105
query41	79	53	51	51
query42	118	107	99	99
query43	485	512	447	447
query44	1282	785	782	782
query45	173	169	164	164
query46	837	1021	610	610
query47	1760	1808	1752	1752
query48	388	415	306	306
query49	782	516	412	412
query50	679	732	400	400
query51	4212	4218	4113	4113
query52	105	106	95	95
query53	229	253	185	185
query54	489	491	413	413
query55	87	82	91	82
query56	266	275	233	233
query57	1126	1170	1037	1037
query58	247	274	232	232
query59	2658	2649	2636	2636
query60	281	279	254	254
query61	136	121	117	117
query62	797	733	677	677
query63	226	184	186	184
query64	4386	996	672	672
query65	4442	4286	4302	4286
query66	1130	409	297	297
query67	15832	15428	15272	15272
query68	8307	867	491	491
query69	458	317	260	260
query70	1205	1096	1108	1096
query71	486	301	256	256
query72	5521	3497	3629	3497
query73	787	720	346	346
query74	9024	8907	8687	8687
query75	3860	3136	2688	2688
query76	3727	1191	749	749
query77	790	368	286	286
query78	10199	10075	9219	9219
query79	2581	840	588	588
query80	594	517	457	457
query81	523	252	223	223
query82	731	132	95	95
query83	175	167	160	160
query84	242	101	72	72
query85	797	363	306	306
query86	386	303	293	293
query87	4560	4453	4323	4323
query88	3663	2214	2235	2214
query89	405	322	275	275
query90	1975	216	207	207
query91	138	143	113	113
query92	77	63	57	57
query93	1623	1065	571	571
query94	673	413	274	274
query95	363	268	257	257
query96	489	557	277	277
query97	3336	3388	3292	3292
query98	229	209	200	200
query99	1372	1439	1258	1258
Total cold run time: 276120 ms
Total hot run time: 184679 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.69 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f121af4670e959a57a016ee6bcd694200d60f21e, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.10	0.11
query3	0.25	0.19	0.20
query4	1.59	0.20	0.11
query5	0.56	0.57	0.55
query6	1.21	0.72	0.71
query7	0.02	0.02	0.02
query8	0.05	0.03	0.04
query9	0.57	0.53	0.52
query10	0.61	0.59	0.58
query11	0.15	0.11	0.11
query12	0.16	0.12	0.12
query13	0.62	0.60	0.59
query14	2.65	2.83	2.72
query15	0.91	0.85	0.85
query16	0.38	0.39	0.37
query17	1.05	1.02	0.99
query18	0.21	0.20	0.19
query19	2.01	1.87	1.77
query20	0.02	0.02	0.01
query21	15.35	0.89	0.53
query22	0.76	1.19	0.67
query23	14.97	1.42	0.61
query24	7.29	1.92	0.50
query25	0.51	0.24	0.07
query26	0.59	0.16	0.13
query27	0.05	0.05	0.04
query28	9.16	0.85	0.42
query29	12.54	4.15	3.37
query30	0.25	0.09	0.06
query31	2.83	0.59	0.40
query32	3.22	0.55	0.47
query33	2.94	3.04	3.02
query34	15.74	5.14	4.53
query35	4.50	4.55	4.50
query36	0.65	0.50	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.03	0.03
query40	0.17	0.15	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.04 s
Total hot run time: 30.69 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 48.79% (13069/26784)
Line Coverage 38.37% (112700/293716)
Region Coverage 37.17% (57304/154149)
Branch Coverage 32.27% (28804/89250)

@luwei16
Copy link
Contributor Author

luwei16 commented Mar 20, 2025

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.96% (1081/1303)
Line Coverage: 65.84% (17970/27293)
Region Coverage: 65.42% (8840/13512)
Branch Coverage: 55.22% (4753/8608)
Coverage Report: http://coverage.selectdb-in.cc/coverage/80ffee0bcd95a7b504ef6d4184e4a5110c611dbc_80ffee0bcd95a7b504ef6d4184e4a5110c611dbc_cloud/report/index.html

@luwei16
Copy link
Contributor Author

luwei16 commented Mar 20, 2025

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.96% (1081/1303)
Line Coverage: 65.83% (17967/27293)
Region Coverage: 65.42% (8840/13512)
Branch Coverage: 55.24% (4755/8608)
Coverage Report: http://coverage.selectdb-in.cc/coverage/28de80ce54565c7169c9315920e6e8edbe76d201_28de80ce54565c7169c9315920e6e8edbe76d201_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 33416 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 28de80ce54565c7169c9315920e6e8edbe76d201, data reload: false

------ Round 1 ----------------------------------
q1	24256	5116	5143	5116
q2	1880	286	161	161
q3	5014	1294	713	713
q4	1905	1065	572	572
q5	960	1221	856	856
q6	188	178	146	146
q7	764	810	494	494
q8	752	1099	832	832
q9	5165	4836	4748	4748
q10	4223	4613	4196	4196
q11	426	297	279	279
q12	336	416	242	242
q13	3414	3757	3129	3129
q14	399	408	380	380
q15	569	505	503	503
q16	669	647	613	613
q17	574	891	355	355
q18	6570	6592	6272	6272
q19	565	988	562	562
q20	475	475	346	346
q21	3313	2905	2594	2594
q22	385	332	307	307
Total cold run time: 62802 ms
Total hot run time: 33416 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5164	5169	5180	5169
q2	537	636	536	536
q3	2186	2774	2315	2315
q4	1461	1848	1394	1394
q5	4344	4256	4250	4250
q6	216	172	132	132
q7	8171	7853	7805	7805
q8	2974	2909	2879	2879
q9	25529	25633	25516	25516
q10	4509	4865	4477	4477
q11	1372	1250	1246	1246
q12	665	837	639	639
q13	3497	4025	3394	3394
q14	402	424	383	383
q15	546	515	517	515
q16	774	809	735	735
q17	2386	2725	2518	2518
q18	7967	7618	7560	7560
q19	792	921	1007	921
q20	2011	2004	1876	1876
q21	12847	11903	12295	11903
q22	1006	596	576	576
Total cold run time: 89356 ms
Total hot run time: 86739 ms

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 48.80% (13070/26781)
Line Coverage 38.40% (112758/293675)
Region Coverage 37.17% (57296/154138)
Branch Coverage 32.28% (28810/89242)

@luwei16
Copy link
Contributor Author

luwei16 commented Mar 20, 2025

run performance

@luwei16
Copy link
Contributor Author

luwei16 commented Mar 20, 2025

run cloud_p0

@doris-robot
Copy link

TPC-H: Total hot run time: 34717 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 28de80ce54565c7169c9315920e6e8edbe76d201, data reload: false

------ Round 1 ----------------------------------
q1	24495	5363	5086	5086
q2	2059	356	175	175
q3	10435	1290	732	732
q4	10244	1038	545	545
q5	7604	2469	2426	2426
q6	198	161	131	131
q7	943	745	606	606
q8	9317	1350	1213	1213
q9	6750	5175	5200	5175
q10	6880	2323	1903	1903
q11	490	288	270	270
q12	363	364	222	222
q13	17769	3767	3173	3173
q14	238	231	207	207
q15	549	493	490	490
q16	621	608	593	593
q17	601	878	361	361
q18	7798	7361	7067	7067
q19	1807	962	581	581
q20	327	337	198	198
q21	4305	3491	2580	2580
q22	1097	1011	983	983
Total cold run time: 114890 ms
Total hot run time: 34717 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5261	5223	5172	5172
q2	245	332	234	234
q3	2145	2650	2287	2287
q4	1516	1903	1506	1506
q5	4555	4430	4404	4404
q6	223	170	132	132
q7	2016	1951	1790	1790
q8	2639	2596	2608	2596
q9	7244	7173	6999	6999
q10	3045	3236	2783	2783
q11	602	524	505	505
q12	712	801	627	627
q13	3581	3975	3293	3293
q14	277	300	291	291
q15	565	505	497	497
q16	660	675	650	650
q17	1191	1641	1407	1407
q18	8026	7653	7424	7424
q19	890	862	1036	862
q20	2041	2039	1852	1852
q21	5342	5048	4849	4849
q22	1090	1064	1028	1028
Total cold run time: 53866 ms
Total hot run time: 51188 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192064 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 28de80ce54565c7169c9315920e6e8edbe76d201, data reload: false

query1	1405	1095	1051	1051
query2	6277	1938	1892	1892
query3	11021	4455	4571	4455
query4	56019	26029	23327	23327
query5	5117	513	500	500
query6	356	195	178	178
query7	4951	508	275	275
query8	316	248	253	248
query9	6323	2609	2628	2609
query10	422	310	262	262
query11	15342	15059	15078	15059
query12	162	110	101	101
query13	1086	510	402	402
query14	11683	8138	6756	6756
query15	221	198	182	182
query16	7212	649	516	516
query17	1090	709	576	576
query18	1653	394	305	305
query19	205	197	166	166
query20	130	130	131	130
query21	210	137	119	119
query22	4569	4600	4271	4271
query23	34278	33462	33428	33428
query24	5818	2512	2497	2497
query25	498	524	444	444
query26	705	293	161	161
query27	1754	525	334	334
query28	2842	2482	2486	2482
query29	622	616	469	469
query30	288	235	194	194
query31	899	883	800	800
query32	78	68	70	68
query33	492	381	318	318
query34	793	894	525	525
query35	871	846	777	777
query36	980	1024	922	922
query37	157	104	79	79
query38	4211	4214	4120	4120
query39	1709	1478	1452	1452
query40	211	119	106	106
query41	53	56	51	51
query42	125	102	106	102
query43	506	532	501	501
query44	1405	839	828	828
query45	181	182	171	171
query46	911	1081	675	675
query47	1859	1818	1772	1772
query48	407	449	324	324
query49	702	533	435	435
query50	783	813	443	443
query51	4357	4274	4300	4274
query52	109	116	113	113
query53	233	269	188	188
query54	512	527	441	441
query55	78	81	82	81
query56	288	297	300	297
query57	1205	1234	1151	1151
query58	266	245	257	245
query59	2647	2972	2489	2489
query60	293	289	264	264
query61	128	122	127	122
query62	757	743	673	673
query63	232	190	188	188
query64	1792	1070	715	715
query65	4557	4448	4463	4448
query66	726	400	296	296
query67	15742	15565	15093	15093
query68	7744	892	505	505
query69	550	312	271	271
query70	1219	1137	1111	1111
query71	497	313	267	267
query72	5621	2740	5533	2740
query73	1373	773	353	353
query74	9100	9132	8990	8990
query75	3757	3228	2735	2735
query76	4371	1207	782	782
query77	643	384	367	367
query78	10105	10022	9275	9275
query79	2786	835	575	575
query80	745	564	429	429
query81	489	256	223	223
query82	721	131	97	97
query83	315	177	155	155
query84	287	98	75	75
query85	822	349	321	321
query86	394	318	303	303
query87	4387	4535	4283	4283
query88	3421	2271	2267	2267
query89	424	317	282	282
query90	1935	225	220	220
query91	148	144	106	106
query92	73	61	57	57
query93	1259	1078	597	597
query94	752	418	303	303
query95	369	283	266	266
query96	495	571	284	284
query97	3355	3457	3367	3367
query98	233	216	203	203
query99	1435	1388	1319	1319
Total cold run time: 303655 ms
Total hot run time: 192064 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.03 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 28de80ce54565c7169c9315920e6e8edbe76d201, data reload: false

query1	0.03	0.03	0.03
query2	0.14	0.11	0.11
query3	0.36	0.21	0.20
query4	1.59	0.20	0.19
query5	0.63	0.63	0.64
query6	1.17	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.63	0.53	0.53
query10	0.58	0.60	0.58
query11	0.25	0.12	0.12
query12	0.25	0.13	0.14
query13	0.63	0.62	0.61
query14	2.82	2.70	2.70
query15	1.01	0.89	0.88
query16	0.38	0.37	0.38
query17	1.04	1.06	1.05
query18	0.18	0.19	0.18
query19	1.95	2.02	1.88
query20	0.02	0.01	0.01
query21	15.35	0.96	0.66
query22	0.94	1.08	0.80
query23	14.68	1.53	0.75
query24	5.46	0.56	0.29
query25	0.17	0.08	0.09
query26	0.55	0.22	0.18
query27	0.08	0.08	0.09
query28	10.94	1.15	0.58
query29	12.54	4.13	3.50
query30	0.29	0.09	0.06
query31	2.82	0.62	0.42
query32	3.24	0.59	0.50
query33	3.10	3.07	3.21
query34	16.21	5.14	4.51
query35	4.47	4.59	4.44
query36	0.64	0.50	0.51
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.18	0.16	0.15
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 106 s
Total hot run time: 32.03 s

@luwei16
Copy link
Contributor Author

luwei16 commented Mar 20, 2025

run buildall

@doris-robot
Copy link

TeamCity cloud ut coverage result:
Function Coverage: 82.96% (1081/1303)
Line Coverage: 65.80% (17958/27293)
Region Coverage: 65.39% (8835/13512)
Branch Coverage: 55.20% (4752/8608)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d4424783e502cdbb242f2f7d3f97ccf4a6c65422_d4424783e502cdbb242f2f7d3f97ccf4a6c65422_cloud/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 34186 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d4424783e502cdbb242f2f7d3f97ccf4a6c65422, data reload: false

------ Round 1 ----------------------------------
q1	24267	5089	5049	5049
q2	2049	303	186	186
q3	10749	1236	687	687
q4	10314	998	540	540
q5	9565	2420	2324	2324
q6	269	164	130	130
q7	943	762	626	626
q8	9338	1333	1111	1111
q9	6826	5116	5088	5088
q10	6858	2313	1918	1918
q11	479	270	251	251
q12	346	351	226	226
q13	17785	3712	3074	3074
q14	229	230	214	214
q15	529	480	489	480
q16	627	623	589	589
q17	594	841	351	351
q18	7567	7292	7150	7150
q19	1770	967	556	556
q20	334	343	202	202
q21	3930	2686	2463	2463
q22	1065	1001	971	971
Total cold run time: 116433 ms
Total hot run time: 34186 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5273	5126	5151	5126
q2	245	331	227	227
q3	2142	2654	2303	2303
q4	1444	1832	1475	1475
q5	4462	4447	4406	4406
q6	215	171	126	126
q7	2022	1900	1759	1759
q8	2656	2596	2533	2533
q9	7237	7139	6878	6878
q10	3067	3210	2755	2755
q11	597	525	495	495
q12	727	757	650	650
q13	3540	3872	3344	3344
q14	285	316	299	299
q15	552	479	469	469
q16	642	679	639	639
q17	1154	1597	1356	1356
q18	7841	7684	7535	7535
q19	824	784	897	784
q20	1906	1954	1814	1814
q21	5248	5064	4886	4886
q22	1096	1059	1045	1045
Total cold run time: 53175 ms
Total hot run time: 50904 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193589 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d4424783e502cdbb242f2f7d3f97ccf4a6c65422, data reload: false

query1	1397	1071	1022	1022
query2	6212	1923	1911	1911
query3	11061	4389	4638	4389
query4	55550	25214	22994	22994
query5	5006	570	481	481
query6	343	188	175	175
query7	4914	497	271	271
query8	324	238	237	237
query9	6020	2588	2610	2588
query10	415	312	261	261
query11	15117	15178	14833	14833
query12	160	107	102	102
query13	1072	508	383	383
query14	11982	6742	7236	6742
query15	202	212	181	181
query16	7018	667	505	505
query17	1075	710	560	560
query18	1527	405	316	316
query19	203	207	162	162
query20	135	119	124	119
query21	211	119	105	105
query22	4665	4616	4346	4346
query23	33947	33407	33264	33264
query24	5739	2421	2464	2421
query25	488	496	416	416
query26	665	276	146	146
query27	1917	517	328	328
query28	2747	2470	2468	2468
query29	573	593	463	463
query30	283	237	196	196
query31	877	924	797	797
query32	77	69	64	64
query33	462	409	331	331
query34	776	866	522	522
query35	802	852	775	775
query36	990	1020	928	928
query37	128	107	80	80
query38	4227	4378	4210	4210
query39	1549	1420	1424	1420
query40	229	124	107	107
query41	58	57	54	54
query42	129	106	111	106
query43	488	510	504	504
query44	1394	842	846	842
query45	187	175	171	171
query46	870	1053	695	695
query47	1792	1890	1779	1779
query48	387	412	309	309
query49	729	518	479	479
query50	695	751	436	436
query51	4353	4368	4313	4313
query52	107	105	101	101
query53	223	279	186	186
query54	510	498	411	411
query55	83	81	78	78
query56	272	269	259	259
query57	1175	1186	1113	1113
query58	256	244	263	244
query59	2653	2944	2692	2692
query60	298	279	279	279
query61	136	139	127	127
query62	756	742	688	688
query63	226	188	185	185
query64	1474	1099	706	706
query65	4630	4409	4337	4337
query66	705	385	291	291
query67	15768	15604	15523	15523
query68	7450	876	501	501
query69	542	295	258	258
query70	1228	1129	1085	1085
query71	493	287	270	270
query72	6084	5239	5463	5239
query73	1418	675	359	359
query74	8932	9058	8737	8737
query75	3958	3259	2769	2769
query76	4383	1193	744	744
query77	609	379	269	269
query78	9999	10125	9342	9342
query79	2208	861	564	564
query80	645	510	427	427
query81	475	252	216	216
query82	677	126	94	94
query83	176	170	154	154
query84	290	95	80	80
query85	816	406	302	302
query86	376	289	273	273
query87	4462	4434	4341	4341
query88	3570	2262	2285	2262
query89	405	306	273	273
query90	1933	208	207	207
query91	141	137	110	110
query92	74	62	54	54
query93	1504	1066	580	580
query94	689	409	266	266
query95	351	272	270	270
query96	484	564	273	273
query97	3318	3420	3267	3267
query98	226	200	197	197
query99	1413	1406	1279	1279
Total cold run time: 300768 ms
Total hot run time: 193589 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.13 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d4424783e502cdbb242f2f7d3f97ccf4a6c65422, data reload: false

query1	0.04	0.03	0.03
query2	0.15	0.12	0.10
query3	0.34	0.19	0.20
query4	1.59	0.20	0.20
query5	0.61	0.63	0.61
query6	1.20	0.74	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.05
query9	0.63	0.52	0.52
query10	0.59	0.59	0.57
query11	0.25	0.13	0.13
query12	0.24	0.13	0.13
query13	0.66	0.63	0.62
query14	2.80	2.81	2.82
query15	1.02	0.87	0.87
query16	0.38	0.39	0.38
query17	1.06	1.05	1.04
query18	0.20	0.20	0.21
query19	1.97	1.98	1.87
query20	0.02	0.01	0.01
query21	15.36	0.96	0.66
query22	0.92	1.06	0.78
query23	14.72	1.49	0.74
query24	5.27	0.64	0.30
query25	0.17	0.10	0.09
query26	0.57	0.22	0.18
query27	0.09	0.09	0.08
query28	11.10	1.18	0.58
query29	12.52	4.29	3.53
query30	0.28	0.09	0.06
query31	2.82	0.66	0.43
query32	3.25	0.60	0.51
query33	3.10	3.13	3.09
query34	16.16	5.19	4.41
query35	4.46	4.48	4.54
query36	0.65	0.51	0.51
query37	0.19	0.17	0.16
query38	0.17	0.15	0.15
query39	0.05	0.04	0.04
query40	0.18	0.16	0.15
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 106.07 s
Total hot run time: 32.13 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 0.00% (0/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 49.96% (13367/26754)
Line Coverage 39.40% (115596/293382)
Region Coverage 38.11% (58679/153992)
Branch Coverage 33.22% (29600/89096)

@@ -264,4 +264,8 @@ CONF_Bool(enable_loopback_address_for_ms, "false");
// Which vaults should be recycled. If empty, recycle all vaults.
// Comma seprated list: recycler_storage_vault_white_list="aaa,bbb,ccc"
CONF_Strings(recycler_storage_vault_white_list, "");

CONF_mInt32(retry_reduce_busyness_threshold, "80");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comments

int get_dynamic_retry_count() {
int64_t busyness_percent = get_fdb_client_thread_busyness_percent();
if (busyness_percent > config::retry_disable_busyness_threshold) {
g_bvar_busynesss_disable_counter << 1;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bvar adder should do
promethues can process qps

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants