-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Enhancement] Support use local file to accelerate the broker load #11196
base: main
Are you sure you want to change the base?
Conversation
90c78d8
to
0f37a48
Compare
0e766a5
to
a74f256
Compare
maybe set config::use_local_filecache_for_broker_random_access_file in broker load's PROPERTIES is a better choice? like the following:
Then user don't need to restart BE if they want to change use_local_filecache_for_broker_random_access_file |
a74f256
to
a73507d
Compare
a73507d
to
9f96381
Compare
ok, but |
run starrocks_fe_unittest |
Kudos, SonarCloud Quality Gate passed! |
[FE PR Coverage Check]😞 fail : 14 / 19 (73.68%) file detail
|
@@ -218,6 +218,8 @@ struct TBrokerScanRangeParams { | |||
15: optional i32 hdfs_read_buffer_size_kb = 0 | |||
// properties from hdfs-site.xml, core-site.xml and load_properties | |||
16: THdfsProperties hdfs_properties | |||
// If use_local_cache is set, we will use local file for broker random access | |||
17: optional bool use_local_cache = false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If an orc file is split into tasks and has some parallelism, will it download one file for many files?
What type of PR is this:
Which issues of this PR fixes :
Fixes #
Problem Summary(Required) :
When there is obvious network latency(e.g. ping command cost time greater than 40ms) between BE and broker file(such as in different idc), broker load ran into slow.
This PR set
use_local_cache
in broker load option to accelerate it.test result:
orc broker load(file count:101,total data size:13GB,table field count:25)
Checklist: