make fuzzing a first class citizen in Go

[dvyukov]

There is a proposal for making fuzzing a first class citizen in Go:
golang/go#19109
First they are trying to understand if there's interest. I would appreciate if you drop a line there if you found fuzzing useful and a brief of your success story.
Thanks

[DavidVorick]

Yes, we've found fuzzing useful in our projects multiple times. Especially sensitive code, the fuzzer will frequently find edge cases that we missed.

I will say that most of the benefit is usually seen in the first tiny bit of fuzzing. There's a pretty strong diminishing returns as you continue to fuzz, at least that's what we've found.

[dvyukov]

@DavidVorick Please post this at golang/go#19109

[dvyukov]

FYI here is a detailed proposal:
https://docs.google.com/document/u/1/d/1zXR-TFL3BfnceEAWytV8bnzB2Tfp6EPFinWVJ5V4QC8/pub
If you have any comments, post them to golang/go#19109

[andybons]

I’m actively working on this.

[andybons]

An update on progress:

I’ve been a bit busy with the China launch so this has taken a back seat to that. I’m going to try to page this stuff back into my brain this week to get a better understanding of the code and what is required.

As stated on golang/go#19109, the next step is to get go-fuzz the closest it can be to our desired API in the go command so that we can better evaluate it as a candidate in the go tool, so I’ll be working on that first. This will also help me get more familiar with what is needed if it’s integrated into the go tool.

[dvyukov]

There are 2 things about the current go-fuzz code:

1. It's overly complex in some places.
2. It's dirty in lots of places.

So I am thinking a good way forward can be:
You create a new branch and start working on the API part from scratch first. Do something that implements the proposed API, clean and close to upstreamable. The fuzzing part can be as simple as "let's just give it a random byte slice".
After this part it should already be usable and should be able to find trivial bugs. At this point we can integrate with OSS-Fuzz already.
Then we can add coverage and very basic mutation logic.
At this point we can upstream this.
Then we probably will need to shake out some stuff.
And later we can make fuzzing logic more complex incrementally.

This is incremental plan with several milestones. We will deliver early and solve the important problems early without burring in complexities of fuzzing logic.
And you obviously can look at the existing code and take any parts you need.
Otherwise I hardly see how we can progress from the current go-fuzz state.
Thoughts?

[andybons]

Sounds good to me!

[kcc]

I would advocate for a slightly different order:

• define the fuzz target API
• write a few micro fuzz targets using that API (similar to libFuzzer's tests
• write a few real fuzz targets for testing parts of the Go run-time
• implement edge (or at least basic block) coverage, similar to https://clang.llvm.org/docs/SanitizerCoverage.html#inline-8bit-counters
• plug some coverage-guided fuzzing engine to this (libFuzzer, go-fuzz, AFL, whatever)
• commit everything upstream, including the tests, make the tests run in CI

Then iterate.

[AlekSi]

One thing I would like to see in the integrated version is #65. In my experience, that is the biggest stopper for people trying to use go-fuzz.

[dvyukov]

Re #65, agree. The current state-of-art in fuzzing seems to be moving in this direction (libfuzzer's protobuf-based mutation, syzkaller). But we again can do the most awesome support for this across other languages as:

func Fuzz(f testing.F, ... arbitrary args here ...) {...}

The args can recursively include arbitrary types, including pointers, structs, slices (except for map/chan probably). Then we use reflect to capture the signature and store inputs along the lines of: {"abc", 0.57, {true, [0, 1, 2]}}.

[guidovranken]

Some time ago I appropriated @dvyukov 's Go instrumentation for usage with libFuzzer for my own needs: https://github.com/guidovranken/libfuzzer-go

I now have an oss-fuzz branch ready to be merged.
It uses libFuzzer's extra counters for coverage.
See: guidovranken/oss-fuzz@0b48d41

It now only features a basic JSON decoder but can easily be extended.

Are you interested in running this on oss-fuzz? If so, I'll make a PR, and transfer ownership to you (Golang team) once it's up. If not, no hard feelings.