-
Notifications
You must be signed in to change notification settings - Fork 3.4k
HBASE-26304: Reflect out of band locality improvements in metrics and balancer #3803
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
f8672b0
to
1ed26b9
Compare
...-server/src/main/java/org/apache/hadoop/hbase/regionserver/InputStreamBlockDistribution.java
Show resolved
Hide resolved
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
Not sure what's up with the HBase bot, but the spotbugs failure is for |
💔 -1 overall
This message was automatically generated. |
1ed26b9
to
c29a907
Compare
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStoreFile.java
Outdated
Show resolved
Hide resolved
c29a907
to
8fa20ba
Compare
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
8fa20ba
to
d287c16
Compare
💔 -1 overall
This message was automatically generated. |
d287c16
to
71ed96a
Compare
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
...-server/src/main/java/org/apache/hadoop/hbase/regionserver/InputStreamBlockDistribution.java
Show resolved
Hide resolved
Please fix the javac and checkstyle warnings? Thanks. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
@Apache9 looks like some timeouts. Any chance you could rerun tests? This was succeeding before I made those simple changes. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
Please fix the checkstyle issues? Thanks. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
Sorry that I missed those. They are fixed now. The latest failure again seems to simply be a timeout in TestReplicationDroppedTables. Not sure if that's a known issue, but seems unrelated to this change. |
🎊 +1 overall
This message was automatically generated. |
…balancer (apache#3803) Signed-off-by: Duo Zhang <zhangduo@apache.org>
…balancer (apache#3803) Signed-off-by: Duo Zhang <zhangduo@apache.org>
See https://issues.apache.org/jira/browse/HBASE-26304 for more details.
This is part of my work on porting our internal LocalityHealer tool to the hbase project.
HBase data is stored in HDFS Blocks, and HBase tracks the distribution of those blocks' replicas in the HDFSBlockDistribution objects in each StoreFileInfo. The current assumption is that those blocks never move after an HStore is opened, but this is not true. Even without the LocalityHealer, blocks can move if a datanode is decommissioned or someone inadvertently runs the hdfs balancer or mover. But the LocalityHealer is special in that it makes moves with an effort to improve locality for HBase.
Due to the above assumption, HBase cannot take advantage of (or properly reflect) out-of-band locality changes. This affects the balancer, some compaction decisions, any downstream users of ClusterMetrics, and metric reporting.
This PR will not actually improve read performance, just the reporting of locality and balancer decisions. The DFSClient will heal itself because it refetches block locations when replicas are not found where expected. I've also submitted apache/hadoop#3527, which makes that more explicit for those users running modern versions of hadoop.
Part 1
The first part of this PR is updating the actual HDFSBlockDistribution used in the RegionServer to compute locality. In my opinion the best option is to have our locality metrics directly reflect the actual input streams doing the reading.
The input streams have a cache of LocatedBlocks which are what they use to drive which datanode to access data from. The LocatedBlocks are exposed to the client through a
getAllBlocks()
method. This method is available in all supported versions of hadoop (per our compatibility matrix). I added an option to the RegionServer which will derive HDFSBlockDistribution from that list of blocks. This value is cached because it does not need to be perfectly in-sync and may cause a call to the namenode.With this change deployed, the locality metrics dynamically go up and down as I move blocks around, exposing exactly what our reads are being subject to.
Part 2
With localityIndex/dataLocality properly reporting, the Balancer also keeps a cache of HDFSBlockDistributions. The solution to keeping that up-to-date here is relatively simple. Upon calling
setClusterMetrics
(which occurs before every balancer run), we compare the new metrics against the old metrics. Any region whose locality has changed will be refreshed.I think it is a pretty fair bet that any time locality changes, the balancer's cache of locations is out of date for the region.