HBASE-28192 Master should recover if meta region state is inconsistent #5513
Conversation
|
🎊 +1 overall
This message was automatically generated. |
| // pending SCPs. It's worth waiting for ~20s before arriving at the conclusion, rather | ||
| // than looping through procedures to figure out how/when/why they are able to or not | ||
| // able to make any progress and eventually abort master initialization anyway. | ||
| Threads.sleep(20000); |
There was a problem hiding this comment.
Interrupt is not handled here.
But, more generally, do not unconditionally wait for so long. Use the Waiter pattern. Why 20 seconds and not some other value? Why not make this configurable?
There was a problem hiding this comment.
introduced 2 configs to improve the retry logic
| // If already many SCPs are scheduled, but they are not progressing because of | ||
| // meta's unavailability, the best action item is to throw PleaseRestartMasterException | ||
| // and let new active master init take care of on-lining meta and process all other | ||
| // pending SCPs. It's worth waiting for ~20s before arriving at the conclusion, rather |
There was a problem hiding this comment.
Is this true? Have we really thought about prioritizing execution of the procedures related to meta and namespace before the others? I agree this could lead to complex code.
There was a problem hiding this comment.
added proc state based waiting
| if (this.getServerManager().isServerOnline(rs.getServerName())) { | ||
| return true; | ||
| } else { | ||
| throw new PleaseRestartMasterException("meta is still not online on live server yet"); |
There was a problem hiding this comment.
What you check here, for this particular region, is if it is supposed to be online on a server, that the server is considered online. This does not seem to check anything specifically pertaining to the meta table. If you want to break out of this loop because meta is not available you should make an explicit check of the region state for the meta regions.
Also, this is the same logic as if we would just loop around again, so why not do that? D.R.Y.
Finally, do we really need to restart the master here or is there some other way to address this?
There was a problem hiding this comment.
This does not seem to check anything specifically pertaining to the meta table.
This utility is used by both meta and namespace regions only.
Finally, do we really need to restart the master here or is there some other way to address this?
That's what will guarantee prioritizing SCP of the server where meta was hosted. So we schedule SCP of the server hosting meta and abort master, new master failover is guaranteed to process SCP of that server, and then continue processing other SCPs that were stuck due to unavailable meta (due to inconsistent meta state).
There was a problem hiding this comment.
Tried improving this overall in the latest revision.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Added some comments on Jira, still it's suspicious, not a guaranteed root cause and maybe this can happen only during upgrade from 2.4 to 2.5? Let me check what happened to SCP of old server. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
There was a problem hiding this comment.
I agree with the other comments, this doesn't seem to address the real issue, which is: why meta SCP was missed?
And even if we don't understand what caused the above and just want to make master resilient to abnormal situations, shouldn't we try to identify if meta is not yet online and if there isn't any SCP for meta, trigger a new one, rather than checking for SCPs for any regions and crashing master if we don't find any such SCP?
| return true; | ||
| } | ||
| } | ||
| throw new PleaseRestartMasterException("Scheduled SCP for old server for region " |
There was a problem hiding this comment.
So the solution here is to "crash" master once we never find an SCP for this region's RS?
There was a problem hiding this comment.
Here we crash only because even after scheduling SCP for the region (meta or namespace), we still find the region not online.
| .filter(p -> p instanceof ServerCrashProcedure | ||
| && serverNameForRegion.equals(((ServerCrashProcedure) p).getServerName())) | ||
| .findFirst(); | ||
| if (scpForServer.isPresent()) { |
There was a problem hiding this comment.
What if an SCP is triggered and finished successfully whilst we are sleeping? Won't we miss the update and keep retrying?
There was a problem hiding this comment.
That is also fine because eventually this will be successful:
if (numRetries == 0) {
rs = this.assignmentManager.getRegionStates().getRegionState(ri);
if (rs != null && rs.isOpened()) {
if (this.getServerManager().isServerOnline(rs.getServerName())) {
return true;
}
}
So if the region (meta/namespace) was assigned successfully, we will return true from here eventually.
|
I would definitely prefer automated solutions rather than rely on HBCK. IMO anything requiring hbck is a bug. |
I agree. For this case, we now know that the root cause was (upgrade to 2.5 + downgrade to 2.4 + meta move + upgrade to 2.5) and hence master did not have correct server address in master local region. I wonder if there is anything else that could also ever cause this problem. |
Could you explain more on the root cause? Why this could cause this problem? Because the downgrading to 2.4 does not do all the necessary rollbacks? |
|
Correct, downgrading to 2.4 does not remove meta's address from master local region's info:sn. Hence, any downgrade from 2.5 to older versions has this risk: it neither removes info CF from master local region, nor do they use master local region to update meta location (since HBASE-26193 is only applicable to 2.5.0+ releases). |
So I think we should add a tool in HBCK2 to for deleting data from master local region? Or at least, remove meta locations from master local region. So after downgrading to 2.4, we need a manual step to remove the location. |
Jira: HBASE-28192