Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement](trash) support skip trash, update trash default expire time #37170

Merged
merged 3 commits into from
Jul 3, 2024
Merged

[enhancement](trash) support skip trash, update trash default expire time #37170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4b65f6e
skip trash
zhannngchen Jul 2, 2024
2a43a68
address comments
zhannngchen Jul 3, 2024
400fdc5
fix
zhannngchen Jul 3, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion be/src/common/config.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,7 +322,7 @@ DEFINE_mInt32(garbage_sweep_batch_size, "100");
DEFINE_mInt32(snapshot_expire_time_sec, "172800");
// It is only a recommended value. When the disk space is insufficient,
// the file storage period under trash dose not have to comply with this parameter.
DEFINE_mInt32(trash_file_expire_time_sec, "259200");
DEFINE_mInt32(trash_file_expire_time_sec, "86400");
Copy link
Collaborator

@yujun777 yujun777 Jul 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe can change this default value to 0. not found any user case recover tablet from trash. it user really want to use trash, they need to manually set this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default 0 might be too aggressive?
it's a behavior change

Copy link
Collaborator

@yujun777 yujun777 Jul 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

trash 很费，实际中没用，trash 的副本在fe中元数据都是必定不存在的，即使手工把它从trash 中拉回来，在fe 元数据也不含它， 把它搞回来后， 都还不能用。 以前doris 很多代码bug， 包括副本迁移等是不是丢数据、 一个tablet的所有副本都可能被删干净，留着trash 可能是防一手的。 但现在这些bug 都已经解决了， 线上用户也没遇到过要从trash 来捞回来的情况。

真正的回收站是 fe catalog recycle bin， 这部分的数据在BE中是一直在data目录中的，跟trash没关系， 可以通过recover 命令把 fe recycle bin 中的数据给恢复过来。

线上给用户建议时，都是建议直接把这个值设置为0的。

后续再提个pr改下这个默认值吧

// minimum file descriptor number
// modify them upon necessity
DEFINE_Int32(min_file_descriptor_number, "60000");
Expand Down
8 changes: 7 additions & 1 deletion be/src/olap/data_dir.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -906,8 +906,14 @@ void DataDir::disks_compaction_num_increment(int64_t delta) {
}

Status DataDir::move_to_trash(const std::string& tablet_path) {
Status res = Status::OK();
if (config::trash_file_expire_time_sec <= 0) {
LOG(INFO) << "delete tablet dir " << tablet_path
<< " directly due to trash_file_expire_time_sec is 0";
RETURN_IF_ERROR(io::global_local_filesystem()->delete_directory(tablet_path));
return delete_tablet_parent_path_if_empty(tablet_path);
}

Status res = Status::OK();
// 1. get timestamp string
string time_str;
if ((res = gen_timestamp_string(&time_str)) != Status::OK()) {
Expand Down
Loading