integrated fuzz testing

[andrewrk]

Make it so that unit tests can ask for fuzz input:

test "foo" {
    const input_bytes = std.testing.fuzzInput(.{});
    try std.testing.expect(!std.mem.eql(u8, "canyoufindme", input_bytes));
}

Introduce flags to the compiler: -ffuzz, -fno-fuzz. These end up passing -fsanitize=fuzzer-no-link to Clang for C/C++ files. Introduce build system equivalent API.

However, neither the CLI interface nor the build system interface is needed in order to enable fuzzing. The only thing that is needed is to ask for fuzz input in unit tests, as in the above example.

When the build runner interacts with the test runner, it learns which tests, if any, are fuzz tests. Then when unit tests pass, it moves on to fuzz testing, by providing our own implementation of the genetic algorithms that drive the input bytes (similar to libFuzzer or AFL), and re-compiling the unit test binary with -ffuzz enabled.

Fuzz testing is level-driven so we will need some CLI to operate those options. For example, zig build --fuzz might start fuzzing indefinitely, while zig build --fuzz=300s declares success after fuzzing for five minutes. When fuzz testing is not requested, it defaults to a small number of iterations just to smoke test that it's all working.

Some sort of UI would be nice. For starters this could just be std.Progress. In the future perhaps there could be a live-updating HTML page to visualize progress and code coverage in realtime. How cool would it be to watch source code turn from red to green live as the fuzzer finds new branches?

I think there's value in being able to fuzz test a mix of Zig and C/C++ source code, so let's start with evaluating LLVM's instrumentation and perhaps being compatible with it, or at least supporting it. First step is to implement the support library in Zig.

-ffuzz will be made available as a comptime flag in @import("builtin") so that it can be used, for example, to choose the naive implementation of std.mem.eql which helps the fuzzer to find interesting branches.

Comments are welcome. Note this is an enhancement not a proposal. The question is not "whether?" but "how?".

Related:

• support code coverage when testing #352
• Proposal: Add a way for a test to expect a panic #1356
• -fsanitize=fuzzer support #5484
• expose random seed to the test runner #17609
• Consider running tests in parallel by deafult #15953

[squeek502]

Note that fuzz testing benefits a lot from starting with an input corpus of short, unique, and relevant inputs:

https://github.com/AFLplusplus/AFLplusplus/blob/stable/docs/fuzzing_in_depth.md#2-preparing-the-fuzzing-campaign

So Zig will likely want to have a way to provide the build/test runner with seed inputs as well. Imaginary syntax:

test "foo" {
    std.testing.fuzzCorpus(&.{
        @embedFile("inputs/input01"),
        @embedFile("inputs/input02"),
    });
    const input_bytes = std.testing.fuzzInput();
    try std.testing.expect(!std.mem.eql(u8, "canyoufindme", input_bytes));
}

But this may not be ideal since it's part of the test code itself. Separating things out into some sort of separate "setup the fuzzer" + "provide a function to repeatedly call" might be worthwhile.

(side note: dictionaries can also be helpful and would similarly ideally be provided in some sort of setup phase)

FWIW here's what Go's integrated fuzz testing looks like:

func FuzzReverse(f *testing.F) {
    testcases := []string{"Hello, world", " ", "!12345"}
    for _, tc := range testcases {
        f.Add(tc)  // Use f.Add to provide a seed corpus
    }
    f.Fuzz(func(t *testing.T, orig string) {
        // ...
    })
}

[mlugg]

When the build runner interacts with the test runner, it learns which tests, if any, are fuzz tests.

How does this mechanism work? If you've not thought about this yet, as a random (possibly bad) idea: perhaps std.testing.fuzzInput returns error{NeedFuzz}![]const u8 or similar, and if -ffuzz is not provided, it just returns error.NeedFuzz, which the caller propagates with try and the test runner can then report to the build runner.

[nektro]

example of a repo that does it today (with afl integration) (with separate zig build test and zig build fuzz steps)
https://github.com/nektro/zig-json

[andrewrk]

How does this mechanism work?

The build runner already runs the test runner as a child process with the test runner protocol over stdio, so that it can keep running unit tests when one of them crashes the process, and check that a unit test triggered a safety panic as expected (#1356). It also makes the parent process know which test was being executed if the unit test crashes the process.

Doing this over stdio is super handy because it even works in strange environments such as via QEMU, wine, or wasmtime.

The function can set a flag indicating that a fuzz test was encountered, then return random bytes (smoke test). Before the test runner sends EOF to the parent process it will send a message indicating metadata about the fuzz tests in the compilation. The build runner then has all the information it needs to enter Fuzz Mode after the main build pipeline is done.

[mlugg]

That makes sense -- nicely designed.

Here's a tangentially related question. Like other parts of the compiler, our testing infrastructure is moving towards a strong bias to running via the build system. Is there, perhaps, an argument to be made for renaming zig test to zig build-test and maybe even eliminating the non-compiler-protocol test runner functionality? The standalone command provides a worse UX, but its name can kind of indicate to people that it's "the way" to test their code; this often leads to people doing incorrect things like trying to zig test individual files within a project (when the correct thing is to test their entire project with a test filter set).

This fuzzing stuff is another example of very tight integration between the build system and compiler, where directly running zig test would at the very least provide a worse UX. (I don't quite understand what the -ffuzz option is intended to do to Zig code, if anything, so don't have a solid grasp of whether it would work at all; does the test runner or the build runner provide the fuzzing inputs?)

Perhaps this is a silly idea; but if you think it has some merit, I'll spin it off into a separate proposal.

[andrewrk]

This would also apply to build-exe, build-lib, build-obj, translate-c, and objcopy. I think there is value in supporting both workflows; the simplicity of using the lower-level commands is quite handy when troubleshooting. I think it's fine if people use zig test to test a single file, as long as it works, but of course the build system is there for managing more complex invocations as well as multiplexing.

The fuzz tests in zig test mode would still run but would only do 1 iteration each, with (probably useless) random input. Perfect for writing the fuzz test before you actually want to give it a spin with zig build --fuzz, and for noticing when you broke it.

To answer your question about -ffuzz, it enables instrumentation in the generated code so that the fuzzer gets feedback on the branches that were taken based on its generated input. This helps it search the state space much more efficiently. The idea here is that there would be two builds of the unit tests - one without this instrumentation for unit tests, and one with the instrumentation that also links in the support library code, for doing fuzz testing.

Edit: now that I think about it, I don't think it would be that hard to make -ffuzz work in combination with zig test as well, although my driving motivation is still the all-powerful zig build integration.

[squeek502]

The fuzz tests in zig test mode would still run but would only do 1 iteration each, with (probably useless) random input. Perfect for writing the fuzz test before you actually want to give it a spin with zig build --fuzz, and for noticing when you broke it.

IMO the ideal would be that in zig test mode it would run the test once with each of the provided input corpus. For a well constructed corpus, this would actually test many different code paths (while being finite + quick).

However, I can't really think of a way to make defining an input corpus work with Zig's current test syntax, so a proof-of-concept that always fuzzes starting with an empty input is probably the way to go.

[matklad]

Some sort of UI would be nice. For starters this could just be std.Progress. In the future perhaps there could be a live-updating HTML page to visualize progress and code coverage in realtime. How cool would it be to watch source code turn from red to green live as the fuzzer finds new branches?

Fuzzing often is done in a distributed manner: ten machines simultaneously running the fuzzer. To enable these kind of use-cases, it would be useful to access the results from the build system. Eg, fuzz step could produce a report in JSON file, which you then can use as an input to “CreateGitHubIssueStep” or some such.

[kristoff-it]

Here's a half-baked idea that maybe somebody could turn into something workable: have a mechanism to ensure that fuzzing hits a certain line of code and that shows a failure otherwise.