Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Avoid returning errors where possible
We were previously unsure whether it was better to return an error and exit the fuzz target early, or to append "fake" data (like all zeros) when an `Arbitrary` implementation requested more data than the fuzzer gave us. We originally chose to exit early, and this commit is the first step in reversing that decision. We didn't have any data to motivate either choice when we originally made the decision to exit early. What we had was an intuition that we shouldn't repeatedly waste time on test cases that are very similar to each other because they are all padded with the same fake data at the end. If we exited early, our thinking went, we would yield our time back to the fuzzer, letting it more efficiently explore the state space. In practice, it has worked out okay, but not great. What we've been missing out on are chance mutations that cause us to explore new code paths. A random mutation made by the fuzzer that would otherwise take us to a new code path happens to have an invalid UTF-8 code point or not quite enough data, and then we return an error and exit early. If we avoided returning errors, then that random mutation would have led us to new code paths. A corollary is that test case reduction is more efficient as well, since the reduced input bytes are also more likely to succeed in generating a valid instance of the `Arbitrary` type. In summary, exiting early yields time back to the fuzzer, giving it more chances to try new mutations, while avoiding early exits is more forgiving to random mutations, making them more likely to succeed at finding new code paths. This is a trade off, and we can't have both in the limit. My local, not-super-rigorous experiments are telling me that we made the wrong trade off originally, and that being forgiving with mutations to find new code paths easier is the better choice. Next breaking release that we make, I think we can remove the fallibility from the `Arbitrary` trait and all the `Unstructured` methods. The only tricky one is `Unstructured::get_bytes`, but I think we can either loosen its contract so that it doesn't always return a slice of length `size`, or remove the method completely in favor of `Unstructured::fill_buffer`.
- Loading branch information