HBASE-28584 RS SIGSEGV under heavy replication load #6124
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Deep clone the cells that are used to apply mutations on the local cluster. Some operations may still be in flight even as we fail to apply some other in-flight mutations and trigger failure handling including a release of the buffer underlying the cellScanner that is sourcing the cells.
|
🎊 +1 overall
This message was automatically generated. |
Apache9
approved these changes
Jul 28, 2024
There was a problem hiding this comment.
Let's get this in to fix the crash issue first.
IIRC, we have a similar problem when async wal is enabled. The rpc call will finish before we actually finish the WAL writing and cause we write out corrupt wal entries.
Let me check how we deal with the problem there. I guess the same trick can be used here too.
Thanks @apurtell for the analyzing.
|
OK, in ServerCall class, we have a retainByWAL method, where we will count the extra references of the ServerCall, mainly the CellScanners. I think we can just change the method to retain, which means we want to retain it for other usage even after the rpc call is done, and also use it here. Anyway, since the current PR has been well tested in producation, I think we can apply it first, and open another issue for optimizing. Thanks. |
|
🎊 +1 overall
This message was automatically generated. |
mnpoonia
approved these changes
Jul 29, 2024
virajjasani
reviewed
Jul 29, 2024
Comment on lines
+2265
to
+2267
| * Deep clones the given cell if the cell supports deep cloning | ||
| * @param cell the cell to be cloned | ||
| * @return the cloned cell |
There was a problem hiding this comment.
nit: javadoc to include @throws
d-c-manning
approved these changes
Aug 3, 2024
|
Any updates here? Thanks. |
|
I think we should fix this in newer releases. If you are all OK, I could try to implement the reference counting way to solve the problem. @apurtell @virajjasani Thoughts? Thanks. |
|
I thought refcounting would be complex but am not opposed to it as a different solution. When and if we have that we could remove the copying. |
|
We would not need this change if #6263 solves the problem instead. |
|
Fixed by #6263 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Clone the cells that are used to apply mutations on the local cluster. Some operations may still be in flight even as we fail to apply some other in-flight mutations and trigger failure handling including a release of the buffer underlying the cellScanner that is sourcing the cells.