Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](filecache) ttl cache runs wild after enable LRU eviction #39814

Merged
merged 1 commit into from
Aug 23, 2024
Merged

[fix](filecache) ttl cache runs wild after enable LRU eviction #39814

merged 1 commit into from
Aug 23, 2024

Conversation

freemandealer
Copy link
Contributor

@freemandealer freemandealer commented Aug 22, 2024 •
edited
Loading

Scenarios where issues occur:

  • TTL with LRU eviction enabled (config::enable_ttl_cache_evict_using_lru = true)
  • TTL's try_reserve_for_ttl_without_lru encounters a limitation set by config::max_ttl_cache_ratio = 90%
  • LRU begins the eviction process. However, the amount of eviction is determined solely by the condition !is_overflow(), which does not consider the 90% limitation. This leads to a premature return of a successful reserve, resulting in the overall TTL exceeding the 90% limit.

Modification:
For the TTL and LRU eviction quantities, in addition to checking for !is_overflow, the condition must also satisfy the restriction set by config::max_ttl_cache_ratio.

Proposed changes

Issue Number: close #xxx

@freemandealer
[fix](filecache) ttl cache runs wild after enable LRU eviction
d2815d6 
Scenarios where issues occur:
- TTL with LRU eviction enabled (config::enable_ttl_cache_evict_using_lru = true)
- TTL's try_reserve_for_ttl_without_lru encounters a limitation set by config::max_ttl_cache_ratio = 90%
- LRU begins the eviction process. However, the amount of eviction is determined solely by the condition !is_overflow(), which does not consider the 90% limitation. This leads to a premature return of a successful reserve, resulting in the overall TTL exceeding the 90% limit.

Modification method:
For the TTL and LRU eviction quantities, in addition to checking for !is_overflow, the condition must also satisfy the restriction set by config::max_ttl_cache_ratio.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@freemandealer
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37966 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d2815d6047c382ce9202d2dfb38251471f5c56da, data reload: false

------ Round 1 ----------------------------------
q1	17618	5286	4300	4300
q2	2016	183	172	172
q3	11855	959	1074	959
q4	10515	806	672	672
q5	7763	2845	2806	2806
q6	222	141	135	135
q7	985	609	622	609
q8	9350	2083	2108	2083
q9	7288	6556	6601	6556
q10	7008	2221	2191	2191
q11	471	241	249	241
q12	396	227	235	227
q13	18668	3018	3032	3018
q14	294	233	234	233
q15	522	476	495	476
q16	492	408	393	393
q17	995	680	756	680
q18	7401	6813	6794	6794
q19	1391	1082	1020	1020
q20	677	344	344	344
q21	4001	3019	3104	3019
q22	1119	1038	1048	1038
Total cold run time: 111047 ms
Total hot run time: 37966 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4339	4299	4249	4249
q2	389	274	264	264
q3	2898	2674	2669	2669
q4	1950	1668	1650	1650
q5	5656	5714	5809	5714
q6	236	134	135	134
q7	2244	1870	1848	1848
q8	3278	3445	3496	3445
q9	8878	8845	8846	8845
q10	3622	3358	3337	3337
q11	596	503	491	491
q12	826	677	648	648
q13	13483	3149	3188	3149
q14	326	295	297	295
q15	543	500	476	476
q16	496	456	433	433
q17	1874	1550	1519	1519
q18	8132	7882	7958	7882
q19	1720	1690	1580	1580
q20	2140	1922	1896	1896
q21	5655	5631	5504	5504
q22	1169	1016	1068	1016
Total cold run time: 70450 ms
Total hot run time: 57044 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191158 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d2815d6047c382ce9202d2dfb38251471f5c56da, data reload: false

query1	1245	885	856	856
query2	6290	1882	1838	1838
query3	10837	4171	4255	4171
query4	59717	23813	23177	23177
query5	5369	494	492	492
query6	408	162	163	162
query7	5752	289	288	288
query8	281	202	206	202
query9	9025	2456	2444	2444
query10	490	275	278	275
query11	17971	14896	15086	14896
query12	165	108	109	108
query13	1597	391	381	381
query14	10077	7345	7022	7022
query15	227	189	169	169
query16	7594	504	502	502
query17	1138	571	573	571
query18	1823	314	308	308
query19	279	154	149	149
query20	119	112	114	112
query21	202	110	106	106
query22	4450	4442	4435	4435
query23	34255	34023	33332	33332
query24	5900	2895	2830	2830
query25	525	376	377	376
query26	682	151	158	151
query27	1801	282	276	276
query28	3848	2050	2027	2027
query29	669	406	401	401
query30	235	154	152	152
query31	922	794	731	731
query32	81	52	64	52
query33	426	278	272	272
query34	852	478	478	478
query35	820	701	694	694
query36	1048	941	923	923
query37	139	88	82	82
query38	4074	3771	3888	3771
query39	1469	1383	1406	1383
query40	203	116	113	113
query41	47	47	44	44
query42	116	96	97	96
query43	507	474	462	462
query44	1080	740	738	738
query45	198	163	164	163
query46	1094	801	752	752
query47	1901	1814	1839	1814
query48	373	303	307	303
query49	749	414	431	414
query50	822	429	420	420
query51	7261	7105	6973	6973
query52	93	84	87	84
query53	250	179	176	176
query54	548	450	458	450
query55	75	77	74	74
query56	274	256	282	256
query57	1189	1055	1040	1040
query58	214	226	219	219
query59	3056	2822	2726	2726
query60	284	273	268	268
query61	101	95	102	95
query62	757	654	665	654
query63	209	181	180	180
query64	3371	1762	1780	1762
query65	3267	3232	3180	3180
query66	632	339	341	339
query67	15413	15097	15246	15097
query68	2969	584	588	584
query69	408	290	285	285
query70	1199	1176	1119	1119
query71	364	281	282	281
query72	2550	2036	2088	2036
query73	705	323	325	323
query74	9071	8722	8928	8722
query75	3357	2689	2698	2689
query76	1398	996	1055	996
query77	545	311	310	310
query78	9780	9120	9139	9120
query79	1026	535	522	522
query80	683	517	500	500
query81	457	229	230	229
query82	291	142	140	140
query83	171	150	144	144
query84	260	78	78	78
query85	676	288	343	288
query86	308	305	301	301
query87	4288	4349	4191	4191
query88	3165	2394	2358	2358
query89	382	287	285	285
query90	1913	190	193	190
query91	119	108	103	103
query92	62	51	50	50
query93	1058	537	535	535
query94	682	250	309	250
query95	323	264	264	264
query96	590	263	270	263
query97	3233	3049	3081	3049
query98	214	203	198	198
query99	1580	1282	1268	1268
Total cold run time: 300713 ms
Total hot run time: 191158 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d2815d6047c382ce9202d2dfb38251471f5c56da, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.66	0.08	0.08
query5	0.49	0.49	0.50
query6	1.14	0.73	0.74
query7	0.02	0.02	0.01
query8	0.06	0.05	0.04
query9	0.54	0.49	0.49
query10	0.55	0.54	0.53
query11	0.17	0.12	0.12
query12	0.15	0.12	0.13
query13	0.62	0.58	0.59
query14	0.77	0.79	0.78
query15	0.84	0.82	0.82
query16	0.36	0.37	0.37
query17	1.05	1.05	1.01
query18	0.20	0.21	0.21
query19	1.90	1.75	1.75
query20	0.01	0.00	0.01
query21	15.41	0.66	0.65
query22	4.53	7.13	1.76
query23	18.31	1.41	1.31
query24	2.14	0.23	0.23
query25	0.15	0.08	0.08
query26	0.26	0.17	0.18
query27	0.09	0.07	0.07
query28	13.18	1.03	1.01
query29	12.60	3.30	3.28
query30	0.24	0.06	0.05
query31	2.85	0.40	0.40
query32	3.25	0.49	0.48
query33	2.95	2.96	2.98
query34	16.95	4.42	4.41
query35	4.53	4.49	4.50
query36	0.64	0.47	0.47
query37	0.20	0.16	0.16
query38	0.17	0.16	0.15
query39	0.05	0.04	0.04
query40	0.16	0.13	0.12
query41	0.09	0.06	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.72 s
Total hot run time: 30.67 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Aug 23, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gavinchou gavinchou merged commit 4eb090c into apache:master Aug 23, 2024
29 of 32 checks passed
dataroaring pushed a commit that referenced this pull request Aug 23, 2024
@freemandealer @dataroaring
[fix](filecache) ttl cache runs wild after enable LRU eviction (#39814)
b0db304 
Scenarios where issues occur:
- TTL with LRU eviction enabled
(config::enable_ttl_cache_evict_using_lru = true)
- TTL's try_reserve_for_ttl_without_lru encounters a limitation set by
config::max_ttl_cache_ratio = 90%
- LRU begins the eviction process. However, the amount of eviction is
determined solely by the condition !is_overflow(), which does not
consider the 90% limitation. This leads to a premature return of a
successful reserve, resulting in the overall TTL exceeding the 90%
limit.

Modification:
For the TTL and LRU eviction quantities, in addition to checking for
!is_overflow, the condition must also satisfy the restriction set by
config::max_ttl_cache_ratio.

Signed-off-by: freemandealer <freeman.zhang1992@gmail.com>
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@freemandealer @doris-robot @gavinchou @dataroaring