Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](inverted index) fix inverted index compound reader memory leak #36146

Merged
merged 2 commits into from
Jun 17, 2024
Merged

[Fix](inverted index) fix inverted index compound reader memory leak #36146

merged 2 commits into from
Jun 17, 2024

Conversation

airborne12
Copy link
Member

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

qidaye
qidaye previously approved these changes Jun 12, 2024
View reviewed changes
Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

zzzxl1993
zzzxl1993 approved these changes Jun 12, 2024
View reviewed changes
Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@qidaye
Copy link
Contributor

qidaye commented Jun 12, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40100 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 13357639e59285b498aecc20e54372150375b511, data reload: false

------ Round 1 ----------------------------------
q1	18062	4598	4384	4384
q2	3055	189	202	189
q3	11194	1105	1182	1105
q4	10758	788	830	788
q5	8874	2710	2644	2644
q6	222	136	135	135
q7	961	605	591	591
q8	9228	2072	2093	2072
q9	8860	6517	6494	6494
q10	9057	3729	3730	3729
q11	448	236	239	236
q12	438	231	223	223
q13	17758	2965	2970	2965
q14	260	228	217	217
q15	519	462	479	462
q16	509	392	371	371
q17	997	668	609	609
q18	8163	7474	7460	7460
q19	5698	1499	1505	1499
q20	666	315	333	315
q21	4965	3305	3285	3285
q22	387	327	331	327
Total cold run time: 121079 ms
Total hot run time: 40100 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4372	4253	4263	4253
q2	371	266	262	262
q3	3000	2739	2741	2739
q4	1877	1609	1578	1578
q5	5219	5262	5259	5259
q6	218	127	130	127
q7	2125	1733	1740	1733
q8	3170	3342	3337	3337
q9	8382	8362	8323	8323
q10	3910	3688	3668	3668
q11	581	487	514	487
q12	767	607	583	583
q13	17372	2987	2967	2967
q14	289	255	262	255
q15	527	478	474	474
q16	476	414	405	405
q17	1817	1487	1464	1464
q18	7738	7489	7460	7460
q19	1680	1643	1655	1643
q20	2031	1802	1775	1775
q21	6115	4705	4817	4705
q22	593	519	552	519
Total cold run time: 72630 ms
Total hot run time: 54016 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171464 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 13357639e59285b498aecc20e54372150375b511, data reload: false

query1	922	373	380	373
query2	6478	2452	2303	2303
query3	6647	212	213	212
query4	18764	17210	17259	17210
query5	4212	486	452	452
query6	246	163	161	161
query7	4604	292	298	292
query8	351	310	300	300
query9	8535	2377	2367	2367
query10	607	291	276	276
query11	10568	9958	9908	9908
query12	138	87	81	81
query13	1641	356	362	356
query14	10027	7614	6277	6277
query15	245	183	183	183
query16	8113	263	254	254
query17	1913	514	522	514
query18	2012	283	278	278
query19	192	158	157	157
query20	92	81	84	81
query21	212	126	125	125
query22	4232	4267	3951	3951
query23	33562	32977	32890	32890
query24	10958	2912	2836	2836
query25	667	348	350	348
query26	1366	145	147	145
query27	2884	320	315	315
query28	7686	2032	2034	2032
query29	882	591	600	591
query30	278	145	149	145
query31	947	717	739	717
query32	99	56	58	56
query33	778	285	266	266
query34	977	473	457	457
query35	777	599	616	599
query36	1066	951	959	951
query37	149	66	70	66
query38	2844	2730	2742	2730
query39	857	793	781	781
query40	206	122	122	122
query41	52	51	49	49
query42	119	95	99	95
query43	568	558	557	557
query44	1288	731	741	731
query45	189	161	161	161
query46	1100	715	739	715
query47	1840	1785	1772	1772
query48	377	303	306	303
query49	1093	399	394	394
query50	772	376	385	376
query51	6700	6654	6670	6654
query52	101	93	91	91
query53	357	283	280	280
query54	964	439	433	433
query55	73	73	71	71
query56	266	287	248	248
query57	1142	1058	1063	1058
query58	251	244	254	244
query59	3570	3327	3147	3147
query60	285	263	254	254
query61	86	83	89	83
query62	663	444	448	444
query63	324	280	283	280
query64	9994	2226	1694	1694
query65	3169	3133	3085	3085
query66	1376	325	328	325
query67	15483	15073	15085	15073
query68	4621	552	534	534
query69	515	426	392	392
query70	1111	1123	1127	1123
query71	388	271	271	271
query72	7205	5375	5702	5375
query73	754	320	321	320
query74	5860	5474	5439	5439
query75	3344	2654	2671	2654
query76	2835	964	924	924
query77	637	291	290	290
query78	10313	10041	9806	9806
query79	1227	508	517	508
query80	1226	459	453	453
query81	590	216	221	216
query82	817	99	100	99
query83	257	165	167	165
query84	257	82	81	81
query85	1540	268	253	253
query86	488	321	284	284
query87	3265	3054	3060	3054
query88	4060	2433	2411	2411
query89	454	374	374	374
query90	1763	186	188	186
query91	129	96	96	96
query92	60	48	47	47
query93	2067	498	491	491
query94	1124	186	179	179
query95	403	306	316	306
query96	595	271	270	270
query97	3167	3051	3046	3046
query98	216	192	246	192
query99	1143	853	850	850
Total cold run time: 272367 ms
Total hot run time: 171464 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 13357639e59285b498aecc20e54372150375b511, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.09	0.10
query5	0.50	0.52	0.50
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.55	0.50	0.50
query10	0.54	0.56	0.55
query11	0.16	0.11	0.11
query12	0.14	0.11	0.11
query13	0.59	0.59	0.59
query14	0.77	0.76	0.77
query15	0.85	0.82	0.81
query16	0.37	0.36	0.37
query17	0.96	1.03	0.96
query18	0.23	0.26	0.26
query19	1.88	1.72	1.72
query20	0.02	0.01	0.01
query21	15.41	0.66	0.66
query22	4.25	7.86	1.66
query23	18.27	1.45	1.23
query24	2.13	0.22	0.22
query25	0.16	0.09	0.09
query26	0.26	0.17	0.17
query27	0.09	0.07	0.08
query28	13.24	1.01	1.00
query29	12.58	3.27	3.26
query30	0.26	0.07	0.06
query31	2.85	0.39	0.37
query32	3.30	0.47	0.46
query33	2.90	2.90	2.86
query34	17.19	4.40	4.43
query35	4.54	4.45	4.52
query36	0.66	0.48	0.47
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.04	0.03	0.04
query40	0.17	0.14	0.15
query41	0.09	0.04	0.05
query42	0.06	0.05	0.04
query43	0.05	0.03	0.03
Total cold run time: 109.59 s
Total hot run time: 30.22 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.40% (8984/24680)
Line Coverage: 27.95% (73456/262828)
Region Coverage: 27.36% (38085/139179)
Branch Coverage: 24.00% (19342/80586)
Coverage Report: http://coverage.selectdb-in.cc/coverage/13357639e59285b498aecc20e54372150375b511_13357639e59285b498aecc20e54372150375b511/report/index.html

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 17, 2024
@airborne12
Copy link
Member Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40218 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b458353efb226cad3043966bea8a966a6d71f0b3, data reload: false

------ Round 1 ----------------------------------
q1	17612	4500	4244	4244
q2	2013	194	197	194
q3	10461	1195	1025	1025
q4	10218	791	654	654
q5	7498	2694	2676	2676
q6	219	136	139	136
q7	956	618	600	600
q8	9219	2093	2075	2075
q9	9097	6517	6496	6496
q10	8893	3751	3725	3725
q11	451	244	241	241
q12	558	233	229	229
q13	17778	2978	2975	2975
q14	258	214	234	214
q15	524	471	472	471
q16	523	372	372	372
q17	982	647	696	647
q18	7991	7625	7403	7403
q19	5528	1381	1423	1381
q20	639	313	306	306
q21	5184	3813	3929	3813
q22	409	341	345	341
Total cold run time: 117011 ms
Total hot run time: 40218 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4388	4248	4200	4200
q2	376	274	259	259
q3	2997	2986	2876	2876
q4	2056	1712	1743	1712
q5	5558	5477	5463	5463
q6	233	134	133	133
q7	2263	1881	1853	1853
q8	3302	3459	3444	3444
q9	8714	8728	8810	8728
q10	4127	3737	3832	3737
q11	595	495	502	495
q12	812	652	651	651
q13	16153	3145	3148	3145
q14	293	285	270	270
q15	534	472	477	472
q16	497	438	415	415
q17	1826	1536	1506	1506
q18	8259	7941	7781	7781
q19	1847	1691	1746	1691
q20	3037	1907	1885	1885
q21	5177	5048	4776	4776
q22	625	559	561	559
Total cold run time: 73669 ms
Total hot run time: 56051 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.47% (8999/24678)
Line Coverage: 28.01% (73714/263217)
Region Coverage: 27.48% (38296/139344)
Branch Coverage: 24.17% (19517/80734)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b458353efb226cad3043966bea8a966a6d71f0b3_b458353efb226cad3043966bea8a966a6d71f0b3/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 175126 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b458353efb226cad3043966bea8a966a6d71f0b3, data reload: false

query1	926	400	369	369
query2	6461	2475	2382	2382
query3	6637	209	202	202
query4	19295	17472	17459	17459
query5	3625	472	470	470
query6	245	158	179	158
query7	4589	302	292	292
query8	319	278	279	278
query9	8539	2370	2362	2362
query10	578	304	277	277
query11	10558	10179	10184	10179
query12	115	88	83	83
query13	1630	365	363	363
query14	8351	7799	7615	7615
query15	238	196	187	187
query16	7695	272	269	269
query17	1903	530	522	522
query18	1952	274	269	269
query19	204	161	156	156
query20	89	83	78	78
query21	210	125	125	125
query22	4265	4174	4042	4042
query23	33705	33648	33634	33634
query24	10952	2858	2908	2858
query25	591	364	382	364
query26	745	160	163	160
query27	2362	330	339	330
query28	6143	2153	2130	2130
query29	899	646	629	629
query30	248	151	151	151
query31	983	767	786	767
query32	91	54	56	54
query33	753	289	275	275
query34	1074	466	469	466
query35	734	648	629	629
query36	1122	944	944	944
query37	157	70	83	70
query38	2944	2868	2877	2868
query39	934	834	812	812
query40	214	131	137	131
query41	59	58	54	54
query42	107	105	108	105
query43	607	562	557	557
query44	1262	720	739	720
query45	195	164	167	164
query46	1076	724	706	706
query47	1855	1772	1785	1772
query48	384	293	303	293
query49	852	415	421	415
query50	752	393	388	388
query51	6909	6569	6696	6569
query52	115	87	95	87
query53	361	292	284	284
query54	898	455	457	455
query55	75	74	74	74
query56	303	281	286	281
query57	1199	1054	1039	1039
query58	264	265	283	265
query59	3467	3127	3139	3127
query60	326	291	292	291
query61	117	115	109	109
query62	608	450	442	442
query63	323	292	292	292
query64	8576	2342	1836	1836
query65	3222	3128	3118	3118
query66	751	335	336	335
query67	15641	14903	14897	14897
query68	8176	554	545	545
query69	690	484	429	429
query70	1218	1122	1129	1122
query71	490	271	278	271
query72	8884	5649	5596	5596
query73	1139	330	320	320
query74	5964	5490	5492	5490
query75	4731	2671	2699	2671
query76	4305	1014	902	902
query77	785	304	303	303
query78	10503	9809	9871	9809
query79	9205	511	509	509
query80	1039	494	470	470
query81	541	226	229	226
query82	717	102	107	102
query83	316	167	168	167
query84	271	81	88	81
query85	1304	281	262	262
query86	405	323	277	277
query87	3372	3101	3112	3101
query88	4623	2361	2355	2355
query89	527	396	392	392
query90	1903	192	185	185
query91	128	99	101	99
query92	63	51	52	51
query93	6621	507	503	503
query94	1173	194	183	183
query95	403	316	322	316
query96	607	273	264	264
query97	3215	3044	3036	3036
query98	226	198	196	196
query99	1174	838	809	809
Total cold run time: 288097 ms
Total hot run time: 175126 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b458353efb226cad3043966bea8a966a6d71f0b3, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.09	0.08
query5	0.50	0.48	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.50	0.49
query10	0.56	0.55	0.53
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.60	0.61
query14	0.78	0.80	0.78
query15	0.83	0.82	0.81
query16	0.36	0.37	0.37
query17	1.01	1.00	1.02
query18	0.23	0.24	0.23
query19	1.79	1.82	1.74
query20	0.01	0.01	0.02
query21	15.41	0.64	0.63
query22	3.59	7.61	2.10
query23	18.25	1.36	1.27
query24	2.07	0.23	0.23
query25	0.14	0.09	0.09
query26	0.27	0.17	0.17
query27	0.09	0.08	0.08
query28	13.26	1.02	1.00
query29	12.63	3.34	3.28
query30	0.26	0.06	0.08
query31	2.86	0.40	0.38
query32	3.27	0.48	0.49
query33	2.92	2.91	2.91
query34	17.30	4.45	4.39
query35	4.50	4.53	4.47
query36	0.66	0.48	0.46
query37	0.19	0.16	0.15
query38	0.15	0.15	0.14
query39	0.05	0.03	0.03
query40	0.17	0.14	0.16
query41	0.10	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 108.97 s
Total hot run time: 30.81 s

@qidaye qidaye merged commit f9a4340 into apache:master Jun 17, 2024
25 of 28 checks passed
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 17, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

airborne12 added a commit to airborne12/apache-doris that referenced this pull request Jun 17, 2024
@airborne12
[Fix](inverted index) fix inverted index compound reader memory leak (a…
fc7120f 
…pache#36146)
@airborne12 airborne12 mentioned this pull request Jun 17, 2024
Merged
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Jun 17, 2024
@airborne12
[Fix](inverted index) fix inverted index compound reader memory leak (a…
a679adb 
…pache#36146)
@airborne12 airborne12 mentioned this pull request Jun 17, 2024
Merged
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Jun 18, 2024
@airborne12
[Fix](inverted index) fix inverted index compound reader memory leak (a…
13bb285 
…pache#36146)
airborne12 added a commit to airborne12/apache-doris that referenced this pull request Jun 18, 2024
@airborne12
[Fix](inverted index) fix inverted index compound reader memory leak (a…
e9aa27c 
…pache#36146)
airborne12 added a commit that referenced this pull request Jun 18, 2024
@airborne12
[Pick 2.0](inverted index) fix inverted index compound reader memory …
f4dbf83 
…leak (#36384)

## Proposed changes

Issue Number: close #xxx

Pick from #36146
airborne12 added a commit that referenced this pull request Jun 18, 2024
@airborne12
[Pick 2.1](inverted index) fix inverted index compound reader memory …
33540ec 
…leak (#36387)

## Proposed changes

Issue Number: close #xxx

Pick from #36146 #36420
@airborne12 airborne12 removed the p0_b label Jun 19, 2024
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
@airborne12 @dataroaring
[Fix](inverted index) fix inverted index compound reader memory leak (#…
04e70d1 
…36146)
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@airborne12
[Pick 2.0](inverted index) fix inverted index compound reader memory …
b60bf7f 
…leak (apache#36384)

## Proposed changes

Issue Number: close #xxx

Pick from apache#36146
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xiaokang xiaokang xiaokang left review comments

@qidaye qidaye qidaye approved these changes

@zzzxl1993 zzzxl1993 zzzxl1993 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.12-merged dev/2.1.4-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@airborne12 @doris-robot @qidaye @xiaokang @zzzxl1993 @dataroaring