[enhancement](trash) support skip trash, update trash default expire time #37170

zhannngchen · 2024-07-02T12:52:05Z

Proposed changes

Issue Number: close #xxx

if user set trash_file_expire_time_sec to 0, skip trash directly
update trash_file_expire_time_sec default value from 3 days to 1 day

docs PR: apache/doris-website#817

doris-robot · 2024-07-02T12:52:10Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

dataroaring

LGTM

dataroaring · 2024-07-02T12:54:52Z

run buildall

github-actions · 2024-07-02T12:55:32Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-02T12:55:34Z

PR approved by anyone and no changes requested.

github-actions · 2024-07-02T12:57:46Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-02T13:16:15Z

TPC-H: Total hot run time: 39874 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b65f6e95998bf2c9b3678953d2049585aa8871c, data reload: false

------ Round 1 ----------------------------------
q1	17639	4349	4346	4346
q2	2017	186	190	186
q3	10488	1152	1071	1071
q4	10175	880	770	770
q5	7802	2738	2661	2661
q6	228	145	138	138
q7	971	610	631	610
q8	9539	2103	2097	2097
q9	8681	6502	6500	6500
q10	8983	3752	3752	3752
q11	454	240	235	235
q12	461	243	234	234
q13	17763	2984	2961	2961
q14	261	235	218	218
q15	530	467	492	467
q16	517	379	373	373
q17	967	610	678	610
q18	8083	7468	7387	7387
q19	2659	1410	1545	1410
q20	653	331	337	331
q21	5024	3175	3924	3175
q22	417	342	342	342
Total cold run time: 114312 ms
Total hot run time: 39874 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4337	4259	4259
q2	367	270	261	261
q3	2937	2728	2757	2728
q4	1912	1606	1639	1606
q5	5266	5281	5297	5281
q6	216	133	130	130
q7	2142	1698	1748	1698
q8	3244	3352	3353	3352
q9	8392	8356	8396	8356
q10	3861	3663	3693	3663
q11	561	477	489	477
q12	797	624	617	617
q13	17538	2983	2948	2948
q14	298	264	269	264
q15	519	477	485	477
q16	465	406	413	406
q17	1756	1484	1456	1456
q18	7609	7596	7467	7467
q19	3453	1659	1576	1576
q20	2015	1784	1777	1777
q21	5045	4754	4800	4754
q22	662	531	579	531
Total cold run time: 73412 ms
Total hot run time: 54084 ms

doris-robot · 2024-07-02T13:27:21Z

TPC-DS: Total hot run time: 174064 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b65f6e95998bf2c9b3678953d2049585aa8871c, data reload: false

query1	920	392	382	382
query2	6474	2469	2355	2355
query3	6671	203	207	203
query4	21775	17397	17332	17332
query5	4172	496	484	484
query6	269	175	191	175
query7	4597	296	301	296
query8	330	310	294	294
query9	8580	2391	2378	2378
query10	647	303	276	276
query11	10474	9966	10029	9966
query12	130	89	83	83
query13	1631	370	367	367
query14	10319	7682	7609	7609
query15	249	189	190	189
query16	7963	270	265	265
query17	1894	523	553	523
query18	1868	272	273	272
query19	195	187	158	158
query20	101	82	84	82
query21	215	131	124	124
query22	4240	4064	3986	3986
query23	33500	33024	33057	33024
query24	11398	2763	2756	2756
query25	595	359	359	359
query26	1621	154	158	154
query27	2934	317	317	317
query28	7499	2103	2094	2094
query29	970	638	633	633
query30	287	156	159	156
query31	986	723	774	723
query32	100	54	61	54
query33	796	299	309	299
query34	921	465	471	465
query35	762	668	672	668
query36	1125	928	935	928
query37	147	75	73	73
query38	2920	2744	2735	2735
query39	842	819	784	784
query40	283	125	122	122
query41	56	52	53	52
query42	118	102	107	102
query43	622	532	518	518
query44	1267	736	738	736
query45	188	162	158	158
query46	1082	716	694	694
query47	1906	1783	1795	1783
query48	376	300	308	300
query49	1150	411	421	411
query50	769	385	381	381
query51	6912	6818	6876	6818
query52	104	91	94	91
query53	356	293	299	293
query54	898	447	449	447
query55	75	73	72	72
query56	280	264	267	264
query57	1161	1056	1075	1056
query58	261	260	312	260
query59	3501	3181	3239	3181
query60	303	277	273	273
query61	94	93	88	88
query62	646	444	458	444
query63	321	290	289	289
query64	9874	2232	1789	1789
query65	3164	3091	3098	3091
query66	1093	341	326	326
query67	15406	14984	14967	14967
query68	5403	538	545	538
query69	617	394	328	328
query70	1183	1089	1159	1089
query71	447	279	282	279
query72	7353	5617	5593	5593
query73	772	329	328	328
query74	5937	5510	5426	5426
query75	3696	2662	2691	2662
query76	3593	979	983	979
query77	658	315	311	311
query78	10327	9978	9819	9819
query79	1507	516	507	507
query80	1706	473	466	466
query81	550	224	226	224
query82	516	103	104	103
query83	297	163	166	163
query84	261	89	84	84
query85	1501	277	323	277
query86	458	339	313	313
query87	3288	3086	3075	3075
query88	4014	2450	2426	2426
query89	470	371	392	371
query90	1786	191	191	191
query91	127	101	99	99
query92	63	47	50	47
query93	1988	512	518	512
query94	1216	193	197	193
query95	402	315	317	315
query96	592	280	278	278
query97	3158	3006	3023	3006
query98	219	198	197	197
query99	1106	843	838	838
Total cold run time: 278594 ms
Total hot run time: 174064 ms

doris-robot · 2024-07-02T13:32:52Z

ClickBench: Total hot run time: 31.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b65f6e95998bf2c9b3678953d2049585aa8871c, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.66	0.09	0.07
query5	0.49	0.47	0.49
query6	1.15	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.51	0.49
query10	0.54	0.54	0.53
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.60	0.60
query14	0.76	0.78	0.77
query15	0.84	0.81	0.81
query16	0.35	0.37	0.37
query17	1.02	1.04	0.97
query18	0.23	0.26	0.26
query19	1.90	1.71	1.82
query20	0.02	0.01	0.01
query21	15.47	0.75	0.65
query22	3.73	7.16	2.79
query23	18.30	1.35	1.30
query24	2.10	0.24	0.24
query25	0.16	0.09	0.09
query26	0.25	0.18	0.18
query27	0.08	0.08	0.08
query28	13.50	1.02	1.00
query29	12.92	3.33	3.34
query30	0.26	0.06	0.06
query31	2.88	0.39	0.40
query32	3.26	0.48	0.46
query33	2.85	2.87	2.95
query34	17.31	4.44	4.45
query35	4.41	4.50	4.54
query36	0.65	0.46	0.47
query37	0.18	0.16	0.16
query38	0.15	0.16	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.04
query42	0.06	0.06	0.05
query43	0.05	0.05	0.04
Total cold run time: 109.69 s
Total hot run time: 31.65 s

be/src/olap/data_dir.cpp

yujun777

need update

yujun777 · 2024-07-03T02:10:06Z

be/src/common/config.cpp

@@ -322,7 +322,7 @@ DEFINE_mInt32(garbage_sweep_batch_size, "100");
 DEFINE_mInt32(snapshot_expire_time_sec, "172800");
 // It is only a recommended value. When the disk space is insufficient,
 // the file storage period under trash dose not have to comply with this parameter.
-DEFINE_mInt32(trash_file_expire_time_sec, "259200");
+DEFINE_mInt32(trash_file_expire_time_sec, "86400");


maybe can change this default value to 0. not found any user case recover tablet from trash. it user really want to use trash, they need to manually set this.

default 0 might be too aggressive?
it's a behavior change

trash 很费，实际中没用，trash 的副本在fe中元数据都是必定不存在的，即使手工把它从trash 中拉回来，在fe 元数据也不含它，把它搞回来后，都还不能用。以前doris 很多代码bug，包括副本迁移等是不是丢数据、一个tablet的所有副本都可能被删干净，留着trash 可能是防一手的。但现在这些bug 都已经解决了，线上用户也没遇到过要从trash 来捞回来的情况。

真正的回收站是 fe catalog recycle bin，这部分的数据在BE中是一直在data目录中的，跟trash没关系，可以通过recover 命令把 fe recycle bin 中的数据给恢复过来。

线上给用户建议时，都是建议直接把这个值设置为0的。

后续再提个pr改下这个默认值吧

zhannngchen · 2024-07-03T03:05:35Z

run buildall

github-actions · 2024-07-03T03:11:04Z

clang-tidy review says "All clean, LGTM! 👍"

zhannngchen · 2024-07-03T03:19:41Z

run buildall

github-actions · 2024-07-03T03:21:11Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-07-03T03:38:29Z

TPC-H: Total hot run time: 39765 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 400fdc559f845177039004eadccd08cb8f6eb9bb, data reload: false

------ Round 1 ----------------------------------
q1	17642	4375	4297	4297
q2	2008	186	180	180
q3	10465	1258	1168	1168
q4	10195	785	823	785
q5	7486	2647	2651	2647
q6	222	136	134	134
q7	954	592	591	591
q8	9227	2048	2080	2048
q9	9031	6504	6447	6447
q10	8939	3715	3683	3683
q11	445	240	249	240
q12	438	232	232	232
q13	17774	2988	2985	2985
q14	269	230	222	222
q15	516	489	480	480
q16	503	377	380	377
q17	967	657	670	657
q18	8074	7370	7416	7370
q19	7945	1480	1438	1438
q20	652	314	311	311
q21	4895	3145	3934	3145
q22	380	329	328	328
Total cold run time: 119027 ms
Total hot run time: 39765 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4462	4253	4258	4253
q2	377	265	260	260
q3	3066	2940	2934	2934
q4	1954	1728	1694	1694
q5	5482	5503	5450	5450
q6	223	137	131	131
q7	2266	1882	1828	1828
q8	3258	3429	3363	3363
q9	8720	8754	8805	8754
q10	4202	3694	3818	3694
q11	585	497	499	497
q12	879	669	628	628
q13	16213	3124	3147	3124
q14	302	268	255	255
q15	524	474	513	474
q16	476	429	425	425
q17	1820	1524	1505	1505
q18	8103	7921	7735	7735
q19	1791	1638	1725	1638
q20	2185	1872	1824	1824
q21	4986	4857	4695	4695
q22	607	548	528	528
Total cold run time: 72481 ms
Total hot run time: 55689 ms

doris-robot · 2024-07-03T03:49:30Z

TPC-DS: Total hot run time: 173763 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 400fdc559f845177039004eadccd08cb8f6eb9bb, data reload: false

query1	931	369	368	368
query2	6448	2500	2287	2287
query3	6635	209	227	209
query4	20651	17483	17302	17302
query5	3620	463	482	463
query6	255	175	163	163
query7	4588	287	287	287
query8	329	277	271	271
query9	8509	2384	2361	2361
query10	558	299	286	286
query11	11436	10074	9854	9854
query12	120	87	85	85
query13	1642	370	363	363
query14	10190	7849	7761	7761
query15	274	179	189	179
query16	7968	262	265	262
query17	1916	537	553	537
query18	2090	268	267	267
query19	187	149	151	149
query20	88	79	80	79
query21	210	122	131	122
query22	4267	4122	3875	3875
query23	33833	33660	33568	33568
query24	10700	2909	2845	2845
query25	574	388	367	367
query26	697	163	151	151
query27	2291	326	317	317
query28	6047	2120	2097	2097
query29	887	622	641	622
query30	258	166	163	163
query31	986	770	781	770
query32	96	55	53	53
query33	658	298	286	286
query34	899	483	481	481
query35	737	660	625	625
query36	1129	977	993	977
query37	139	81	77	77
query38	2935	2788	2879	2788
query39	885	837	838	837
query40	212	129	138	129
query41	51	60	50	50
query42	115	101	104	101
query43	613	552	519	519
query44	1072	706	719	706
query45	192	172	166	166
query46	1066	750	719	719
query47	1851	1770	1760	1760
query48	382	300	301	300
query49	841	410	404	404
query50	761	379	386	379
query51	6864	6724	6784	6724
query52	98	102	87	87
query53	361	298	295	295
query54	876	440	447	440
query55	73	73	74	73
query56	278	259	256	256
query57	1133	1045	1040	1040
query58	247	234	257	234
query59	3253	3163	3262	3163
query60	294	268	284	268
query61	92	87	90	87
query62	596	451	451	451
query63	318	293	289	289
query64	8542	2292	1740	1740
query65	3164	3125	3088	3088
query66	740	315	324	315
query67	15598	15151	14888	14888
query68	5468	520	538	520
query69	639	407	325	325
query70	1101	1155	1054	1054
query71	429	278	274	274
query72	7183	5438	5674	5438
query73	767	327	333	327
query74	5876	5447	5468	5447
query75	3607	2672	2658	2658
query76	3120	980	968	968
query77	626	304	289	289
query78	10310	9822	9752	9752
query79	2384	505	501	501
query80	2233	472	457	457
query81	558	218	217	217
query82	1450	108	109	108
query83	300	167	165	165
query84	269	89	86	86
query85	1339	274	262	262
query86	472	305	320	305
query87	3258	3054	3083	3054
query88	3770	2357	2335	2335
query89	480	384	387	384
query90	1721	186	191	186
query91	131	99	100	99
query92	58	50	46	46
query93	2327	497	495	495
query94	1073	185	189	185
query95	409	317	315	315
query96	602	265	263	263
query97	3239	3002	3043	3002
query98	214	200	204	200
query99	1185	821	846	821
Total cold run time: 272944 ms
Total hot run time: 173763 ms

doris-robot · 2024-07-03T03:55:01Z

ClickBench: Total hot run time: 30.95 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 400fdc559f845177039004eadccd08cb8f6eb9bb, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.03	0.04
query3	0.23	0.05	0.06
query4	1.67	0.07	0.07
query5	0.47	0.50	0.47
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.48	0.48
query10	0.55	0.55	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.13
query13	0.59	0.59	0.58
query14	0.76	0.77	0.78
query15	0.84	0.81	0.80
query16	0.36	0.36	0.37
query17	0.94	0.98	1.01
query18	0.23	0.27	0.25
query19	1.89	1.73	1.78
query20	0.01	0.01	0.00
query21	15.43	0.76	0.65
query22	4.25	6.61	2.34
query23	18.33	1.36	1.19
query24	2.22	0.23	0.22
query25	0.15	0.10	0.09
query26	0.28	0.18	0.17
query27	0.08	0.08	0.08
query28	13.20	1.01	0.99
query29	12.62	3.30	3.30
query30	0.24	0.06	0.06
query31	2.88	0.38	0.39
query32	3.28	0.47	0.46
query33	2.91	2.88	2.90
query34	17.17	4.42	4.41
query35	4.48	4.50	4.57
query36	0.65	0.46	0.49
query37	0.19	0.16	0.16
query38	0.15	0.15	0.14
query39	0.04	0.03	0.03
query40	0.17	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.64 s
Total hot run time: 30.95 s

yujun777

LGTM

dataroaring

LGTM

github-actions · 2024-07-03T14:48:28Z

PR approved by at least one committer and no changes requested.

…time (apache#37170) ## Proposed changes Issue Number: close #xxx 1. if user set `trash_file_expire_time_sec` to 0, skip trash directly 2. update trash_file_expire_time_sec default value from 3 days to 1 day docs PR: apache/doris-website#817

…time (#37170) (#37409) cherry-pick #37170

update trash default expire time to 1 day see apache/doris#37170

…time (#37170) ## Proposed changes Issue Number: close #xxx 1. if user set `trash_file_expire_time_sec` to 0, skip trash directly 2. update trash_file_expire_time_sec default value from 3 days to 1 day docs PR: apache/doris-website#817

skip trash

4b65f6e

dataroaring previously approved these changes Jul 2, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024

github-actions bot added the reviewed label Jul 2, 2024

zhannngchen mentioned this pull request Jul 2, 2024

[docs](trash) update trash default expire time apache/doris-website#817

Merged

gavinchou previously approved these changes Jul 2, 2024

View reviewed changes

yujun777 reviewed Jul 3, 2024

View reviewed changes

be/src/olap/data_dir.cpp Outdated Show resolved Hide resolved

yujun777 reviewed Jul 3, 2024

View reviewed changes

be/src/olap/data_dir.cpp Outdated Show resolved Hide resolved

yujun777 suggested changes Jul 3, 2024

View reviewed changes

yujun777 reviewed Jul 3, 2024

View reviewed changes

address comments

2a43a68

zhannngchen dismissed stale reviews from gavinchou and dataroaring via 2a43a68 July 3, 2024 03:05

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 3, 2024

fix

400fdc5

yujun777 approved these changes Jul 3, 2024

View reviewed changes

zhannngchen requested review from dataroaring and gavinchou July 3, 2024 11:54

dataroaring approved these changes Jul 3, 2024

View reviewed changes

dataroaring added dev/2.1.x kind/behavior-changed labels Jul 3, 2024

dataroaring merged commit 4453bfd into apache:master Jul 3, 2024
26 of 31 checks passed

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 3, 2024

zhannngchen mentioned this pull request Jul 8, 2024

[enhancement](trash) support skip trash, update trash default expire time (#37170) #37409

Merged

zhannngchen added a commit that referenced this pull request Jul 8, 2024

[enhancement](trash) support skip trash, update trash default expire …

494b54a

…time (#37170) (#37409) cherry-pick #37170

zhannngchen added dev/2.1.5-merged and removed dev/2.1.x labels Jul 8, 2024

luzhijing pushed a commit to apache/doris-website that referenced this pull request Jul 16, 2024

[docs](trash) update trash default expire time (#817)

24c4925

update trash default expire time to 1 day see apache/doris#37170

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[enhancement](trash) support skip trash, update trash default expire time #37170

[enhancement](trash) support skip trash, update trash default expire time #37170

zhannngchen commented Jul 2, 2024 •

edited

Loading

doris-robot commented Jul 2, 2024

dataroaring left a comment

dataroaring commented Jul 2, 2024

github-actions bot commented Jul 2, 2024

github-actions bot commented Jul 2, 2024

github-actions bot commented Jul 2, 2024

doris-robot commented Jul 2, 2024

doris-robot commented Jul 2, 2024

doris-robot commented Jul 2, 2024

yujun777 left a comment

yujun777 Jul 3, 2024 •

edited

Loading

zhannngchen Jul 3, 2024

yujun777 Jul 3, 2024 •

edited

Loading

zhannngchen commented Jul 3, 2024

github-actions bot commented Jul 3, 2024

zhannngchen commented Jul 3, 2024

github-actions bot commented Jul 3, 2024

doris-robot commented Jul 3, 2024

doris-robot commented Jul 3, 2024

doris-robot commented Jul 3, 2024

yujun777 left a comment

dataroaring left a comment

github-actions bot commented Jul 3, 2024

[enhancement](trash) support skip trash, update trash default expire time #37170

[enhancement](trash) support skip trash, update trash default expire time #37170

Conversation

zhannngchen commented Jul 2, 2024 • edited Loading

Proposed changes

doris-robot commented Jul 2, 2024

dataroaring left a comment

Choose a reason for hiding this comment

dataroaring commented Jul 2, 2024

github-actions bot commented Jul 2, 2024

github-actions bot commented Jul 2, 2024

github-actions bot commented Jul 2, 2024

doris-robot commented Jul 2, 2024

doris-robot commented Jul 2, 2024

doris-robot commented Jul 2, 2024

yujun777 left a comment

Choose a reason for hiding this comment

yujun777 Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

zhannngchen Jul 3, 2024

Choose a reason for hiding this comment

yujun777 Jul 3, 2024 • edited Loading

Choose a reason for hiding this comment

zhannngchen commented Jul 3, 2024

github-actions bot commented Jul 3, 2024

zhannngchen commented Jul 3, 2024

github-actions bot commented Jul 3, 2024

doris-robot commented Jul 3, 2024

doris-robot commented Jul 3, 2024

doris-robot commented Jul 3, 2024

yujun777 left a comment

Choose a reason for hiding this comment

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 3, 2024

zhannngchen commented Jul 2, 2024 •

edited

Loading

yujun777 Jul 3, 2024 •

edited

Loading

yujun777 Jul 3, 2024 •

edited

Loading