Consider nemeses used in Foundation DB #27
Description
FoundationDB nemeses:
- Laggy communication between nodes
- Network routing errors
- Swizzle clogging*
- Simulated software bugs**
- Upgrades from old server/file versions
Incomplete writes to disk(see Implement filesystem fault-injections #75)Corrupted writes to disk(see Implement filesystem fault-injections #75)Disk drive runs out of space(see Implement filesystem fault-injections #75)Single node in a cluster terminates(done in Add Kill nemesis #16)Network partitions between nodes(done in Add partition nemesis #17)Processes freeze for random durations(done in Add pause nemesis #21)
"For a while, there was an informal competition within the engineering team to design failures that found the toughest bugs and issues the most easily. After a period of one-upsmanship, the reigning champion is called “swizzle-clogging”. To swizzle-clog, you first pick a random subset of nodes in the cluster. Then, you “clog” (stop) each of their network connections one by one over a few seconds. Finally, you unclog them in a random order, again one by one, until they are all up. This pattern seems to be particularly good at finding deep issues that only happen in the rarest real-world cases." https://apple.github.io/foundationdb/testing.html