Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate SegmentDirectoryLoader and tierBackend concepts #7737

Merged
merged 7 commits into from
Nov 10, 2021

Conversation

npawar
Copy link
Contributor

@npawar npawar commented Nov 9, 2021

Right now the concept of tier storage backend and segmentDirectoryLoader have gotten mixed. Cleaning that up to have SegmentDirectoryLoader identifiable by the segment.directory.loader property and moving tier.backend and tier.properties to the table config.

@codecov-commenter
Copy link

codecov-commenter commented Nov 9, 2021

Codecov Report

Merging #7737 (8874b62) into master (920093a) will increase coverage by 1.17%.
The diff coverage is 72.72%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #7737      +/-   ##
============================================
+ Coverage     70.18%   71.35%   +1.17%     
  Complexity     4034     4034              
============================================
  Files          1575     1575              
  Lines         80289    80298       +9     
  Branches      11938    11938              
============================================
+ Hits          56353    57300     +947     
+ Misses        20061    19112     -949     
- Partials       3875     3886      +11     
Flag Coverage Δ
integration1 29.11% <1.81%> (?)
integration2 27.76% <1.81%> (+0.01%) ⬆️
unittests1 68.47% <71.69%> (-0.02%) ⬇️
unittests2 14.57% <1.85%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...c/main/java/org/apache/pinot/common/tier/Tier.java 88.88% <0.00%> (-11.12%) ⬇️
...inot/common/tier/TimeBasedTierSegmentSelector.java 87.50% <0.00%> (-5.84%) ⬇️
...che/pinot/common/utils/config/TierConfigUtils.java 45.83% <0.00%> (-15.28%) ⬇️
...ment/spi/loader/SegmentDirectoryLoaderContext.java 70.00% <50.00%> (-5.00%) ⬇️
...ent/spi/loader/SegmentDirectoryLoaderRegistry.java 70.37% <50.00%> (-2.71%) ⬇️
...ache/pinot/common/tier/PinotServerTierStorage.java 70.00% <62.50%> (-30.00%) ⬇️
...local/segment/index/loader/IndexLoadingConfig.java 65.93% <83.33%> (+0.52%) ⬆️
...java/org/apache/pinot/common/tier/TierFactory.java 100.00% <100.00%> (ø)
...ntroller/helix/core/rebalance/TableRebalancer.java 76.20% <100.00%> (+1.69%) ⬆️
...indexsegment/immutable/ImmutableSegmentLoader.java 90.90% <100.00%> (-0.35%) ⬇️
... and 107 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 920093a...8874b62. Read the comment docs.

*/
public static SegmentDirectoryLoader getSegmentDirectoryLoader(String tierBackend) {
return SEGMENT_DIRECTORY_LOADER_MAP.get(tierBackend);
public static void setSegmentDirectoryLoader(String segmentDirectoryLoaderName, SegmentDirectoryLoader loader) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@npawar npawar merged commit c9ba48e into apache:master Nov 10, 2021
@npawar npawar deleted the tier_config branch November 10, 2021 01:36
kriti-sc pushed a commit to kriti-sc/incubator-pinot that referenced this pull request Dec 12, 2021
Right now the concept of tier storage backend and segmentDirectoryLoader have gotten mixed. Cleaning that up to have SegmentDirectoryLoader identifiable by the segment.directory.loader property and moving tier.backend and tier.properties to the table config.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants