Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](memory) Fix LRU cache deleter and memory tracking #32080

Merged
merged 7 commits into from
Mar 13, 2024
Merged

[fix](memory) Fix LRU cache deleter and memory tracking #32080

merged 7 commits into from
Mar 13, 2024

Conversation

xinyiZzz
Copy link
Contributor

Proposed changes

In order to add common code to the value deleter of LRU cache, let all lru cache values inherit from LRUCacheValueBase class and tracking memory in destructor.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@xinyiZzz
Copy link
Contributor Author

run buildall

@xinyiZzz xinyiZzz force-pushed the 20240310_fix_lrucache_delter branch from 4236f41 to 5bd29b5 Compare March 12, 2024 07:05
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 36643 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5bd29b5b7bef14d59926a7d60e8076df42469885, data reload: false

------ Round 1 ----------------------------------
q1	17628	4221	4098	4098
q2	2027	163	145	145
q3	10562	910	880	880
q4	4648	660	720	660
q5	7623	2813	2791	2791
q6	181	124	123	123
q7	1194	835	799	799
q8	9489	2081	2020	2020
q9	7376	6494	6460	6460
q10	8126	2464	2493	2464
q11	417	218	210	210
q12	762	300	289	289
q13	17983	3017	3016	3016
q14	280	259	253	253
q15	493	457	449	449
q16	484	396	391	391
q17	958	551	537	537
q18	6706	5927	5874	5874
q19	1577	1451	1466	1451
q20	568	267	286	267
q21	7324	3174	3244	3174
q22	793	297	292	292
Total cold run time: 107199 ms
Total hot run time: 36643 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4076	4049	4059	4049
q2	325	225	226	225
q3	2978	2821	2836	2821
q4	1889	1544	1529	1529
q5	5215	5264	5268	5264
q6	191	115	115	115
q7	2249	1845	1872	1845
q8	3186	3295	3345	3295
q9	8580	8595	8569	8569
q10	6112	3651	3628	3628
q11	533	455	433	433
q12	735	563	548	548
q13	12415	2885	2848	2848
q14	274	246	250	246
q15	490	451	445	445
q16	439	425	398	398
q17	1739	1491	1462	1462
q18	7714	7198	7047	7047
q19	6827	1513	1538	1513
q20	1908	1695	1704	1695
q21	4846	4560	4706	4560
q22	533	460	441	441
Total cold run time: 73254 ms
Total hot run time: 52976 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186741 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5bd29b5b7bef14d59926a7d60e8076df42469885, data reload: false

query1	914	353	347	347
query2	7567	1960	1969	1960
query3	6706	221	216	216
query4	31093	20583	20637	20583
query5	4353	541	456	456
query6	261	179	184	179
query7	4620	295	282	282
query8	226	167	181	167
query9	9824	2218	2206	2206
query10	453	239	242	239
query11	17305	14062	14123	14062
query12	130	91	84	84
query13	1635	417	413	413
query14	12634	10535	11390	10535
query15	256	221	199	199
query16	8069	249	248	248
query17	1963	546	530	530
query18	2086	267	269	267
query19	224	172	159	159
query20	87	84	83	83
query21	204	126	120	120
query22	4659	4464	4429	4429
query23	31526	31189	31247	31189
query24	10847	2834	2821	2821
query25	622	377	376	376
query26	1275	150	150	150
query27	3087	344	341	341
query28	8104	1866	1807	1807
query29	863	651	619	619
query30	302	149	150	149
query31	965	725	724	724
query32	97	57	57	57
query33	764	276	258	258
query34	1046	460	472	460
query35	852	608	599	599
query36	1008	869	851	851
query37	121	76	86	76
query38	3244	3100	3142	3100
query39	1413	1372	1407	1372
query40	213	111	112	111
query41	51	46	47	46
query42	101	92	100	92
query43	507	433	447	433
query44	1293	692	706	692
query45	275	255	261	255
query46	1076	688	676	676
query47	1653	1586	1609	1586
query48	438	359	343	343
query49	1129	332	335	332
query50	752	363	372	363
query51	6854	6739	6782	6739
query52	106	91	92	91
query53	343	271	271	271
query54	297	241	235	235
query55	93	84	81	81
query56	257	227	232	227
query57	1101	1009	1008	1008
query58	226	198	213	198
query59	2710	2630	2646	2630
query60	262	254	257	254
query61	115	112	121	112
query62	593	382	398	382
query63	295	271	271	271
query64	5899	3891	3920	3891
query65	3131	3013	3037	3013
query66	1422	332	335	332
query67	15038	14228	14297	14228
query68	9238	542	530	530
query69	673	374	372	372
query70	1378	1154	1110	1110
query71	534	275	278	275
query72	6860	2652	2516	2516
query73	1651	313	308	308
query74	7102	6724	6508	6508
query75	9818	7987	8162	7987
query76	5673	948	999	948
query77	650	258	254	254
query78	10301	9653	9606	9606
query79	10280	506	524	506
query80	2209	398	393	393
query81	501	213	213	213
query82	270	198	206	198
query83	248	139	150	139
query84	283	81	74	74
query85	1260	332	316	316
query86	342	319	287	287
query87	3483	3224	3205	3205
query88	4790	2265	2272	2265
query89	477	352	351	351
query90	2392	177	178	177
query91	167	130	129	129
query92	62	45	45	45
query93	5620	503	481	481
query94	1550	173	174	173
query95	432	324	323	323
query96	613	264	262	262
query97	4397	4283	4246	4246
query98	218	211	209	209
query99	976	764	730	730
Total cold run time: 322083 ms
Total hot run time: 186741 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5bd29b5b7bef14d59926a7d60e8076df42469885, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.02
query3	0.23	0.06	0.06
query4	1.69	0.10	0.10
query5	0.49	0.51	0.49
query6	1.19	0.62	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.54	0.54	0.52
query10	0.53	0.55	0.54
query11	0.12	0.09	0.10
query12	0.12	0.09	0.09
query13	0.56	0.57	0.58
query14	0.73	0.75	0.74
query15	0.79	0.78	0.78
query16	0.40	0.38	0.40
query17	1.01	0.95	0.94
query18	0.21	0.25	0.21
query19	1.87	1.69	1.78
query20	0.01	0.01	0.01
query21	15.40	0.57	0.56
query22	2.72	2.14	1.81
query23	17.44	0.74	0.78
query24	3.90	1.33	1.48
query25	0.15	0.15	0.03
query26	0.74	0.12	0.12
query27	0.03	0.04	0.03
query28	10.16	0.90	0.85
query29	12.50	3.21	3.11
query30	0.70	0.61	0.56
query31	2.79	0.33	0.32
query32	3.42	0.44	0.45
query33	2.84	2.82	2.78
query34	16.11	4.38	4.33
query35	4.30	4.31	4.41
query36	1.09	1.01	1.02
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.02
query40	0.17	0.15	0.14
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 105.37 s
Total hot run time: 30.53 s

yiguolei
yiguolei reviewed Mar 12, 2024
View reviewed changes
@xinyiZzz xinyiZzz force-pushed the 20240310_fix_lrucache_delter branch from 14b61f4 to 76725bf Compare March 12, 2024 12:01
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 36815 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 76725bfcbb01c33248624c89cd45ccb5ed7b1411, data reload: false

------ Round 1 ----------------------------------
q1	17642	4383	4153	4153
q2	2020	152	138	138
q3	10584	928	894	894
q4	4661	704	715	704
q5	7601	2808	2865	2808
q6	186	125	129	125
q7	1239	859	824	824
q8	9505	2081	2020	2020
q9	7328	6492	6477	6477
q10	8097	2490	2499	2490
q11	422	213	217	213
q12	756	290	285	285
q13	17964	3018	3001	3001
q14	277	243	247	243
q15	492	456	449	449
q16	488	390	402	390
q17	946	556	576	556
q18	6658	5876	5929	5876
q19	1594	1409	1460	1409
q20	562	290	271	271
q21	7397	3209	3234	3209
q22	835	280	311	280
Total cold run time: 107254 ms
Total hot run time: 36815 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4159	4077	4100	4077
q2	325	223	225	223
q3	2971	2817	2819	2817
q4	1823	1584	1579	1579
q5	5232	5286	5249	5249
q6	196	116	117	116
q7	2255	1848	1811	1811
q8	3178	3296	3317	3296
q9	8613	8602	8567	8567
q10	6188	3671	3652	3652
q11	531	435	457	435
q12	716	581	552	552
q13	12636	2854	2927	2854
q14	280	250	247	247
q15	486	444	442	442
q16	464	422	410	410
q17	1731	1480	1461	1461
q18	7739	7197	7010	7010
q19	3815	1510	1499	1499
q20	1909	1733	1683	1683
q21	4993	4655	4631	4631
q22	522	434	440	434
Total cold run time: 70762 ms
Total hot run time: 53045 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186747 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 76725bfcbb01c33248624c89cd45ccb5ed7b1411, data reload: false

query1	916	358	351	351
query2	7499	2203	2040	2040
query3	6719	218	207	207
query4	31285	20618	20562	20562
query5	4406	488	410	410
query6	267	170	171	170
query7	4625	298	281	281
query8	229	168	170	168
query9	9896	2215	2212	2212
query10	453	240	237	237
query11	14712	14079	13995	13995
query12	135	81	82	81
query13	1616	418	439	418
query14	10691	10678	10861	10678
query15	250	200	189	189
query16	7958	268	266	266
query17	1921	578	556	556
query18	2077	294	280	280
query19	227	157	150	150
query20	90	84	82	82
query21	202	124	129	124
query22	4610	4415	4416	4415
query23	31836	31016	31059	31016
query24	10833	2818	2828	2818
query25	613	391	381	381
query26	1294	154	154	154
query27	2620	350	350	350
query28	7699	1858	1826	1826
query29	912	652	627	627
query30	310	145	153	145
query31	969	712	704	704
query32	96	63	61	61
query33	771	255	246	246
query34	1085	472	490	472
query35	842	602	597	597
query36	1007	888	836	836
query37	131	78	74	74
query38	3320	3163	3172	3163
query39	1433	1398	1385	1385
query40	210	111	111	111
query41	51	48	48	48
query42	103	100	99	99
query43	503	466	480	466
query44	1265	740	702	702
query45	284	259	255	255
query46	1103	688	675	675
query47	1653	1597	1581	1581
query48	466	360	386	360
query49	1143	346	348	346
query50	752	371	383	371
query51	6671	6547	6751	6547
query52	105	90	93	90
query53	343	275	281	275
query54	304	249	240	240
query55	85	84	85	84
query56	252	228	229	228
query57	1073	1028	1016	1016
query58	243	206	208	206
query59	3004	2816	2629	2629
query60	262	272	257	257
query61	112	110	114	110
query62	602	408	405	405
query63	306	280	281	280
query64	5693	4027	3929	3929
query65	3038	3005	3008	3005
query66	859	333	317	317
query67	15121	14502	14157	14157
query68	8890	537	536	536
query69	633	381	395	381
query70	1282	1161	1208	1161
query71	515	273	288	273
query72	6785	2676	2520	2520
query73	1166	318	319	318
query74	7474	6540	6607	6540
query75	9471	7975	8041	7975
query76	5361	812	984	812
query77	670	268	266	266
query78	10421	9784	9552	9552
query79	10146	517	508	508
query80	1830	441	365	365
query81	501	209	214	209
query82	810	203	207	203
query83	214	143	140	140
query84	281	79	75	75
query85	1451	330	313	313
query86	406	292	301	292
query87	3423	3249	3242	3242
query88	4732	2369	2357	2357
query89	516	355	362	355
query90	1984	176	176	176
query91	156	126	130	126
query92	60	47	48	47
query93	7038	539	489	489
query94	1213	175	176	175
query95	429	342	325	325
query96	611	268	265	265
query97	4398	4268	4269	4268
query98	216	215	213	213
query99	1119	757	754	754
Total cold run time: 316363 ms
Total hot run time: 186747 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 76725bfcbb01c33248624c89cd45ccb5ed7b1411, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.68	0.10	0.10
query5	0.52	0.53	0.51
query6	1.18	0.62	0.63
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.56	0.48	0.48
query10	0.55	0.52	0.53
query11	0.12	0.08	0.08
query12	0.12	0.10	0.09
query13	0.57	0.56	0.56
query14	0.72	0.75	0.76
query15	0.81	0.79	0.77
query16	0.40	0.38	0.38
query17	0.95	1.01	0.99
query18	0.19	0.28	0.22
query19	1.82	1.75	1.67
query20	0.01	0.01	0.00
query21	15.40	0.56	0.56
query22	2.76	2.02	2.04
query23	17.28	0.73	0.89
query24	3.27	2.03	1.28
query25	0.29	0.12	0.12
query26	0.61	0.13	0.12
query27	0.04	0.03	0.03
query28	10.34	0.89	0.84
query29	12.65	3.16	3.27
query30	0.63	0.60	0.56
query31	2.80	0.33	0.35
query32	3.38	0.44	0.45
query33	2.76	2.80	2.79
query34	16.07	4.29	4.34
query35	4.32	4.32	4.34
query36	1.07	1.01	1.00
query37	0.07	0.06	0.05
query38	0.04	0.02	0.02
query39	0.03	0.01	0.01
query40	0.17	0.13	0.14
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.7 s
Total hot run time: 30.76 s

yiguolei
yiguolei previously approved these changes Mar 12, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@xinyiZzz
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 12, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 37059 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a2817d24ca6e90b39f32e2e87d0dd16ace82fe0f, data reload: false

------ Round 1 ----------------------------------
q1	17847	4383	4255	4255
q2	2431	157	152	152
q3	11767	982	937	937
q4	5086	711	725	711
q5	8073	2912	2911	2911
q6	190	133	131	131
q7	1220	849	837	837
q8	9701	2081	2030	2030
q9	7347	6585	6520	6520
q10	8178	2478	2557	2478
q11	427	228	217	217
q12	797	311	289	289
q13	17954	3025	3001	3001
q14	276	260	248	248
q15	487	490	453	453
q16	477	399	388	388
q17	939	602	534	534
q18	6629	5943	5784	5784
q19	1554	1449	1488	1449
q20	553	285	275	275
q21	7424	3164	3265	3164
q22	789	295	316	295
Total cold run time: 110146 ms
Total hot run time: 37059 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4166	4076	4053	4053
q2	319	226	229	226
q3	2966	2826	2826	2826
q4	1864	1587	1568	1568
q5	5211	5247	5251	5247
q6	193	113	114	113
q7	2252	1899	1866	1866
q8	3173	3297	3292	3292
q9	8627	8532	8563	8532
q10	6216	3734	3647	3647
q11	531	448	437	437
q12	727	565	555	555
q13	7820	2876	2875	2875
q14	275	262	255	255
q15	478	456	450	450
q16	450	404	409	404
q17	1749	1511	1484	1484
q18	7682	7247	7136	7136
q19	7010	1533	1595	1533
q20	1930	1724	1724	1724
q21	4734	4804	4708	4708
q22	556	437	447	437
Total cold run time: 68929 ms
Total hot run time: 53368 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187306 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a2817d24ca6e90b39f32e2e87d0dd16ace82fe0f, data reload: false

query1	910	395	358	358
query2	7424	2074	1996	1996
query3	6709	220	215	215
query4	31652	20759	20530	20530
query5	4360	409	395	395
query6	261	177	178	177
query7	4621	295	295	295
query8	243	191	174	174
query9	9948	2290	2259	2259
query10	448	241	249	241
query11	17046	13972	14092	13972
query12	146	90	79	79
query13	1629	418	423	418
query14	10961	10849	10819	10819
query15	260	220	188	188
query16	8070	263	264	263
query17	1925	569	536	536
query18	2083	285	281	281
query19	258	158	167	158
query20	94	82	90	82
query21	199	128	127	127
query22	4690	4493	4419	4419
query23	31425	31134	31372	31134
query24	10735	2884	2833	2833
query25	635	383	378	378
query26	1397	156	157	156
query27	3049	363	350	350
query28	8056	1908	1890	1890
query29	892	639	630	630
query30	342	144	146	144
query31	968	720	715	715
query32	98	60	56	56
query33	773	257	271	257
query34	1072	465	474	465
query35	846	601	600	600
query36	998	866	881	866
query37	126	74	76	74
query38	3348	3173	3140	3140
query39	1400	1382	1394	1382
query40	204	122	109	109
query41	50	49	47	47
query42	101	100	97	97
query43	487	456	468	456
query44	1302	717	696	696
query45	248	272	236	236
query46	1100	687	678	678
query47	1700	1591	1571	1571
query48	432	366	360	360
query49	1141	342	343	342
query50	753	379	373	373
query51	6732	6724	6632	6632
query52	100	90	96	90
query53	351	277	319	277
query54	300	234	229	229
query55	84	85	78	78
query56	233	228	235	228
query57	1079	994	995	994
query58	229	200	205	200
query59	3006	2757	2611	2611
query60	256	254	261	254
query61	113	110	129	110
query62	577	401	394	394
query63	295	279	276	276
query64	5633	3928	3900	3900
query65	3047	3044	2992	2992
query66	920	330	312	312
query67	15334	14721	14355	14355
query68	9248	525	541	525
query69	671	369	381	369
query70	1379	1117	1173	1117
query71	519	282	271	271
query72	7183	2631	2486	2486
query73	1604	324	320	320
query74	8487	6598	6593	6593
query75	9794	8006	7990	7990
query76	5789	985	929	929
query77	665	262	263	262
query78	10354	9722	9625	9625
query79	11729	521	532	521
query80	2459	449	366	366
query81	508	205	209	205
query82	350	203	198	198
query83	246	145	146	145
query84	290	76	75	75
query85	1233	321	310	310
query86	358	275	287	275
query87	3431	3258	3387	3258
query88	5531	2283	2292	2283
query89	482	363	365	363
query90	2064	175	177	175
query91	174	131	131	131
query92	61	50	47	47
query93	6402	506	507	506
query94	1347	174	175	174
query95	439	334	330	330
query96	596	269	273	269
query97	4409	4298	4323	4298
query98	233	209	211	209
query99	1010	735	732	732
Total cold run time: 324949 ms
Total hot run time: 187306 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.02% (8574/24482)
Line Coverage: 26.81% (69528/259318)
Region Coverage: 26.07% (36115/138512)
Branch Coverage: 23.03% (18454/80120)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a2817d24ca6e90b39f32e2e87d0dd16ace82fe0f_a2817d24ca6e90b39f32e2e87d0dd16ace82fe0f/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 29.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a2817d24ca6e90b39f32e2e87d0dd16ace82fe0f, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.66	0.10	0.10
query5	0.52	0.48	0.50
query6	1.19	0.63	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.50	0.51
query10	0.53	0.55	0.55
query11	0.12	0.09	0.10
query12	0.12	0.09	0.09
query13	0.58	0.56	0.55
query14	0.73	0.75	0.75
query15	0.78	0.79	0.78
query16	0.38	0.40	0.40
query17	1.05	1.05	0.94
query18	0.19	0.26	0.25
query19	1.72	1.70	1.69
query20	0.02	0.01	0.01
query21	15.46	0.57	0.57
query22	2.94	2.77	1.11
query23	17.18	0.79	0.70
query24	3.95	1.44	0.80
query25	0.07	0.04	0.03
query26	0.86	0.12	0.11
query27	0.04	0.03	0.04
query28	11.03	0.87	0.86
query29	12.59	3.23	3.26
query30	0.65	0.58	0.58
query31	2.79	0.32	0.33
query32	3.41	0.44	0.45
query33	2.84	2.82	2.81
query34	16.13	4.32	4.33
query35	4.34	4.37	4.28
query36	1.05	1.00	1.02
query37	0.07	0.06	0.05
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 106.33 s
Total hot run time: 29.45 s

@xinyiZzz xinyiZzz force-pushed the 20240310_fix_lrucache_delter branch from a2817d2 to cdb4d46 Compare March 12, 2024 18:48
@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.03% (8575/24482)
Line Coverage: 26.81% (69531/259321)
Region Coverage: 26.07% (36111/138512)
Branch Coverage: 23.03% (18455/80120)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cdb4d46ca86d35b729a84ad74443060c2443ec56_cdb4d46ca86d35b729a84ad74443060c2443ec56/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 36344 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cdb4d46ca86d35b729a84ad74443060c2443ec56, data reload: false

------ Round 1 ----------------------------------
q1	17665	4230	4070	4070
q2	2021	155	141	141
q3	10573	906	914	906
q4	4653	695	701	695
q5	7597	2708	2753	2708
q6	186	127	125	125
q7	1184	829	796	796
q8	9486	2044	2005	2005
q9	7348	6466	6416	6416
q10	8155	2498	2480	2480
q11	427	219	217	217
q12	777	278	287	278
q13	17953	2995	3000	2995
q14	281	245	258	245
q15	508	444	439	439
q16	488	390	407	390
q17	944	548	557	548
q18	6676	5893	5761	5761
q19	1567	1389	1442	1389
q20	545	278	267	267
q21	7330	3255	3184	3184
q22	806	289	308	289
Total cold run time: 107170 ms
Total hot run time: 36344 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4080	4055	4054	4054
q2	328	225	232	225
q3	2984	2857	2785	2785
q4	1841	1590	1578	1578
q5	5165	5257	5222	5222
q6	192	115	118	115
q7	2234	1825	1846	1825
q8	3149	3271	3291	3271
q9	8541	8537	8538	8537
q10	6156	3676	3667	3667
q11	529	457	441	441
q12	737	553	534	534
q13	12917	2930	2850	2850
q14	281	244	246	244
q15	490	449	440	440
q16	463	426	408	408
q17	1707	1467	1452	1452
q18	7514	7201	7067	7067
q19	5336	1512	1491	1491
q20	1945	1688	1715	1688
q21	4848	4610	4688	4610
q22	525	435	456	435
Total cold run time: 71962 ms
Total hot run time: 52939 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186172 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cdb4d46ca86d35b729a84ad74443060c2443ec56, data reload: false

query1	904	351	365	351
query2	7418	2062	1938	1938
query3	6702	222	212	212
query4	31211	20623	20751	20623
query5	4367	474	473	473
query6	252	180	179	179
query7	4619	307	289	289
query8	228	165	173	165
query9	9842	2276	2235	2235
query10	448	233	253	233
query11	17210	14053	14116	14053
query12	135	89	83	83
query13	1628	422	422	422
query14	11417	10884	10158	10158
query15	259	208	189	189
query16	8001	261	258	258
query17	1937	570	541	541
query18	2051	296	279	279
query19	219	155	160	155
query20	94	95	81	81
query21	196	134	124	124
query22	4617	4444	4421	4421
query23	31840	31056	31099	31056
query24	7942	2798	2868	2798
query25	583	375	383	375
query26	722	159	152	152
query27	2356	342	348	342
query28	6156	1893	1885	1885
query29	869	635	622	622
query30	306	150	158	150
query31	967	719	713	713
query32	92	57	55	55
query33	523	266	252	252
query34	871	476	480	476
query35	725	619	608	608
query36	990	854	873	854
query37	107	76	74	74
query38	3217	3120	3085	3085
query39	1406	1364	1405	1364
query40	207	112	110	110
query41	51	48	48	48
query42	105	97	103	97
query43	501	470	467	467
query44	1040	722	705	705
query45	265	234	246	234
query46	1084	689	689	689
query47	1688	1572	1582	1572
query48	418	354	370	354
query49	1101	339	348	339
query50	757	376	386	376
query51	6887	6510	6609	6510
query52	104	97	87	87
query53	340	281	280	280
query54	264	241	235	235
query55	86	77	81	77
query56	262	235	231	231
query57	1087	998	993	993
query58	232	198	208	198
query59	2870	2606	2573	2573
query60	305	250	253	250
query61	111	110	110	110
query62	567	408	402	402
query63	301	276	279	276
query64	5346	3907	3862	3862
query65	3027	3009	3039	3009
query66	839	318	331	318
query67	14762	14263	14460	14263
query68	7660	525	534	525
query69	629	376	371	371
query70	1288	1119	1206	1119
query71	461	273	278	273
query72	6818	2696	2492	2492
query73	713	309	325	309
query74	6842	6571	6534	6534
query75	9767	7913	8023	7913
query76	4468	929	935	929
query77	657	256	259	256
query78	10365	9555	9549	9549
query79	8780	526	531	526
query80	1523	390	383	383
query81	522	210	219	210
query82	854	208	213	208
query83	225	153	147	147
query84	282	79	81	79
query85	1298	331	334	331
query86	432	301	303	301
query87	3357	3179	3227	3179
query88	5167	2315	2318	2315
query89	505	383	386	383
query90	1953	177	182	177
query91	174	139	131	131
query92	61	47	47	47
query93	6528	520	497	497
query94	1180	175	178	175
query95	436	330	331	330
query96	615	272	270	270
query97	4422	4253	4284	4253
query98	228	223	219	219
query99	1140	753	781	753
Total cold run time: 307384 ms
Total hot run time: 186172 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cdb4d46ca86d35b729a84ad74443060c2443ec56, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.69	0.10	0.10
query5	0.56	0.49	0.52
query6	1.22	0.63	0.63
query7	0.03	0.02	0.02
query8	0.04	0.02	0.02
query9	0.57	0.48	0.47
query10	0.53	0.55	0.54
query11	0.11	0.08	0.09
query12	0.11	0.10	0.10
query13	0.56	0.56	0.56
query14	0.74	0.75	0.76
query15	0.80	0.78	0.77
query16	0.38	0.40	0.39
query17	0.96	0.96	0.94
query18	0.19	0.27	0.22
query19	1.80	1.72	1.69
query20	0.01	0.00	0.00
query21	15.40	0.57	0.57
query22	2.50	1.90	1.86
query23	17.33	0.77	0.68
query24	3.16	2.87	1.49
query25	0.16	0.24	0.04
query26	0.76	0.13	0.12
query27	0.04	0.03	0.03
query28	9.95	0.87	0.86
query29	12.56	3.22	3.23
query30	0.66	0.65	0.57
query31	2.80	0.35	0.34
query32	3.39	0.43	0.44
query33	2.76	2.87	2.78
query34	16.12	4.39	4.34
query35	4.34	4.29	4.31
query36	1.06	0.98	1.00
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.14	0.13
query41	0.07	0.02	0.01
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 104.03 s
Total hot run time: 30.76 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit cdb4d46ca86d35b729a84ad74443060c2443ec56 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       17.4 seconds inserted 10000000 Rows, about 574K ops/s

@xinyiZzz
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.02% (8574/24482)
Line Coverage: 26.81% (69527/259323)
Region Coverage: 26.07% (36117/138512)
Branch Coverage: 23.03% (18455/80120)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b744327a68d241dd325793c6aa7375cd3acf109d_b744327a68d241dd325793c6aa7375cd3acf109d/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 36495 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b744327a68d241dd325793c6aa7375cd3acf109d, data reload: false

------ Round 1 ----------------------------------
q1	17628	4156	4097	4097
q2	2033	154	145	145
q3	10564	897	895	895
q4	4645	670	687	670
q5	7597	2783	2788	2783
q6	185	124	126	124
q7	1173	804	812	804
q8	9479	2013	1999	1999
q9	7234	6447	6415	6415
q10	8129	2549	2445	2445
q11	424	227	214	214
q12	754	287	283	283
q13	17969	2997	2981	2981
q14	284	253	257	253
q15	497	463	458	458
q16	494	401	392	392
q17	943	588	524	524
q18	6600	5868	5895	5868
q19	1560	1370	1450	1370
q20	532	293	278	278
q21	7405	3235	3212	3212
q22	828	285	297	285
Total cold run time: 106957 ms
Total hot run time: 36495 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4089	4064	4075	4064
q2	322	226	231	226
q3	2950	2825	2836	2825
q4	1806	1523	1541	1523
q5	5191	5255	5244	5244
q6	195	116	116	116
q7	2228	1822	1829	1822
q8	3116	3269	3286	3269
q9	8519	8492	8556	8492
q10	6120	3677	3722	3677
q11	531	439	440	439
q12	719	525	579	525
q13	11351	2892	2860	2860
q14	275	242	267	242
q15	469	453	446	446
q16	438	415	423	415
q17	1717	1478	1443	1443
q18	7546	7283	7179	7179
q19	5823	1473	1509	1473
q20	1928	1707	1742	1707
q21	4805	4684	4645	4645
q22	541	460	442	442
Total cold run time: 70679 ms
Total hot run time: 53074 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186627 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b744327a68d241dd325793c6aa7375cd3acf109d, data reload: false

query1	903	365	340	340
query2	7459	2119	2100	2100
query3	6702	223	220	220
query4	31431	20559	20642	20559
query5	4318	408	391	391
query6	272	174	182	174
query7	4632	305	289	289
query8	222	170	174	170
query9	9927	2273	2260	2260
query10	471	241	263	241
query11	14684	14014	14268	14014
query12	133	88	87	87
query13	1652	417	427	417
query14	10923	10915	10504	10504
query15	264	228	200	200
query16	8025	260	258	258
query17	1939	573	540	540
query18	2110	295	286	286
query19	213	164	156	156
query20	88	86	85	85
query21	203	138	130	130
query22	4684	4462	4467	4462
query23	31995	31199	31095	31095
query24	10745	2875	2821	2821
query25	610	388	377	377
query26	907	156	161	156
query27	2347	351	346	346
query28	6688	1887	1866	1866
query29	896	662	631	631
query30	307	149	153	149
query31	956	707	720	707
query32	100	58	57	57
query33	779	265	254	254
query34	1084	491	473	473
query35	836	612	597	597
query36	1023	899	845	845
query37	116	77	77	77
query38	3325	3165	3186	3165
query39	1391	1408	1372	1372
query40	211	112	113	112
query41	50	51	49	49
query42	104	99	102	99
query43	492	467	449	449
query44	1164	707	720	707
query45	270	244	265	244
query46	1092	693	687	687
query47	1672	1577	1589	1577
query48	481	369	356	356
query49	1109	355	346	346
query50	767	373	375	373
query51	6717	6655	6572	6572
query52	108	90	119	90
query53	352	272	279	272
query54	311	231	243	231
query55	95	83	82	82
query56	255	233	239	233
query57	1067	1010	1011	1010
query58	230	200	225	200
query59	2897	2624	2645	2624
query60	265	254	262	254
query61	124	110	111	110
query62	587	418	401	401
query63	298	277	277	277
query64	5549	4004	3847	3847
query65	3064	2991	2979	2979
query66	828	322	320	320
query67	14837	14693	14156	14156
query68	8735	561	538	538
query69	626	392	387	387
query70	1270	1097	1183	1097
query71	520	281	276	276
query72	6552	2697	2527	2527
query73	726	317	316	316
query74	6973	6530	6601	6530
query75	9689	8040	7835	7835
query76	5229	900	989	900
query77	688	261	265	261
query78	10438	9655	9648	9648
query79	11786	536	526	526
query80	1944	449	372	372
query81	516	205	222	205
query82	866	197	205	197
query83	211	147	142	142
query84	288	76	83	76
query85	1374	368	320	320
query86	424	312	285	285
query87	3426	3163	3242	3163
query88	5192	2309	2307	2307
query89	509	366	363	363
query90	2015	177	187	177
query91	164	130	133	130
query92	55	46	46	46
query93	7168	510	492	492
query94	1193	173	172	172
query95	431	324	330	324
query96	589	272	266	266
query97	4393	4265	4276	4265
query98	241	215	215	215
query99	1097	753	741	741
Total cold run time: 315684 ms
Total hot run time: 186627 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

ClickBench: Total hot run time: 30.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b744327a68d241dd325793c6aa7375cd3acf109d, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.03
query3	0.27	0.07	0.07
query4	2.49	0.10	0.10
query5	0.54	0.52	0.56
query6	1.19	0.63	0.62
query7	0.02	0.02	0.02
query8	0.04	0.02	0.02
query9	0.57	0.52	0.47
query10	0.53	0.53	0.54
query11	0.12	0.08	0.09
query12	0.11	0.09	0.09
query13	0.57	0.57	0.57
query14	0.74	0.75	0.76
query15	0.79	0.80	0.76
query16	0.37	0.38	0.36
query17	0.95	0.97	0.94
query18	0.19	0.26	0.24
query19	1.80	1.71	1.75
query20	0.01	0.01	0.02
query21	15.41	0.57	0.56
query22	2.12	1.99	1.08
query23	17.24	0.71	0.71
query24	3.95	2.18	2.30
query25	0.19	0.17	0.24
query26	0.69	0.12	0.12
query27	0.03	0.02	0.02
query28	9.27	0.87	0.86
query29	12.53	3.16	3.12
query30	0.64	0.56	0.55
query31	2.81	0.32	0.33
query32	3.39	0.45	0.44
query33	2.76	2.78	2.83
query34	16.07	4.28	4.30
query35	4.37	4.27	4.33
query36	1.08	0.99	1.00
query37	0.06	0.05	0.06
query38	0.04	0.03	0.04
query39	0.02	0.01	0.02
query40	0.16	0.15	0.14
query41	0.06	0.02	0.02
query42	0.02	0.01	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.34 s
Total hot run time: 30.66 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b744327a68d241dd325793c6aa7375cd3acf109d with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       17.1 seconds inserted 10000000 Rows, about 584K ops/s

wangbo
wangbo approved these changes Mar 13, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xinyiZzz xinyiZzz merged commit b1cae7c into apache:master Mar 13, 2024
25 of 29 checks passed
yiguolei pushed a commit that referenced this pull request Mar 15, 2024
@xinyiZzz @Doris-Extras
[fix](memory) Fix LRU cache deleter and memory tracking (#32080)
7b74b19 
In order to add common code to the value deleter of LRU cache, let all lru cache values inherit from LRUCacheValueBase class and tracking memory in destructor.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@wangbo wangbo wangbo approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@xinyiZzz @doris-robot @yiguolei @wangbo