Improve fault injection to MultiRead #8937
Conversation
Summary: Right now BlockBasedTable::RetrieveMultipleBlocks() override error from MaybeReadBlockAndLoadToCache() with checksum checking error. MaybeReadBlockAndLoadToCache() only inserts blocks to cache in this path, so error can happen when checksum mismatch or block cache full. It's not clear whether it is a problem if we swallow checksum mismatch error and recheck it. I think it is a better practice stop operation we see an error. Test Plan: Run all existing tests.
Summary: Several improvements to MultiRead: 1. Add two more types of fault that should be handled: empty read results and checksum mismatch 2. Add a message indicating which type of fault is injected 3. Increase the failure rate Test Plan:
|
The problem is that, sometimes the stress test fails. Not sure it's the test issue or a bug. |
siying
changed the title
|
@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
…d be fore prefetching
|
@siying has updated the pull request. You must reimport the pull request before landing. |
|
@siying has updated the pull request. You must reimport the pull request before landing. |
utilities/fault_injection_fs.h
Outdated
| @@ -497,6 +503,13 @@ class FaultInjectionTestFS : public FileSystemWrapper { | |||
| // saved callstack | |||
| void PrintFaultBacktrace(); | |||
|
|
|||
| void AddThreadLocalMessage(const std::string& m) { | |||
There was a problem hiding this comment.
This is not being called from anywhere. Dead code?
There was a problem hiding this comment.
Yes I can remove it. I used it for adding manual code in all the places to debug...
|
@siying has updated the pull request. You must reimport the pull request before landing. |
|
@siying has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator. |
|
This pull request has been merged in 9320067. |
Summary: Several improvements to MultiRead: 1. Fix a bug in stress test which causes false positive when both MultiRead() return and individual read request have failure injected. 2. Add two more types of fault that should be handled: empty read results and checksum mismatch 3. Add a message indicating which type of fault is injected 4. Increase the failure rate Pull Request resolved: facebook/rocksdb#8937 Reviewed By: anand1976 Differential Revision: D31085930 fbshipit-source-id: 3a04994a3cadebf9a64d25e1fe12b14b7a272fba
Summary:
Several improvements to MultiRead:
Test Plan: