Use standalone main in reproduce

[mikea]

https://reviews.llvm.org/rL285135 adds standalone main
Reproduce command should link with that one instead of full libfuzzer.

[Dor1s]

That seems to be needed for issue #799. Not sure about reproducing with standalone target, but for coverage it definitely would be useful. For example, a test report for zlib contains more libFuzzer coverage data than stuff from zlib itself:

Some fuzz targets have more than 32K units in corpus, but I think our Linux images can handle more, so current implementation of https://github.com/llvm-mirror/compiler-rt/blob/master/lib/fuzzer/standalone/StandaloneFuzzTargetMain.c#L24 shouldn't fail for those.

[kcc]

We can also feed parts of the corpus to StandaloneFuzzTargetMain.c and then merge the results.
Or enhance StandaloneFuzzTargetMain.c

[oliverchang]

What would be the benefits of this?

It feels that this adds more complexity (more builds and configuration changes) than the benefit it adds (a bit less overhead?)

[Dor1s]

I see the following benefits (all of them don't seem strong enough):

• faster and smaller coverage builds (i.e. with profiler instrumentation)
• reproducibility that is independent from libFuzzer revisions (e.g. imagine a bad revision that crashes or exits), standalone target is less fragile
• can be slightly better for debugging a single crash / issue

But I tend to Oliver's opinion that we don't really need this...

[oliverchang]

Thanks for outlining these.

faster and smaller coverage builds (i.e. with profiler instrumentation)

I would imagine the bulk of the time it takes to generate coverage would come from actually running individual inputs? there shouldn't be much (time) overhead from libFuzzer driving this right?

reproducibility that is independent from libFuzzer revisions (e.g. imagine a bad revision that crashes or exits), standalone target is less fragile

This complicates many things on CF end, and we would need it to support timeout and OOM detection.

Right now I'm not seeing the benefits :/ over the downsides:

• This adds 3 more builds to our build matrix.
• The standalone main has limitations: e.g. with the maximum number of units that it can run at a time because of commandline length limitations.

[Dor1s]

Yes, the overheard shouldn't be significant for most of the targets.

[Dor1s]

@kcc @inferno-chromium do you have any ideas why we might need a standalone build? I don't remember why Mike had created this issue in the first place...

[kcc]

a standalone build is easier to reason about for anyone who is not familiar with our tools.
It's also much easier to reproduce bugs with it outside of our environment (especially if e.g. msan is involved). Since we already use different compiler flags to build for coverage, why not also link it against a different main?

standalone main has limitations:

Yes, I don't want to make this main() more complex because a) it will force us to write a non-portable code and b) it will complicate what needs to be very simple.

But this complication is very easy to overcome. Suppose C* are the corpus dirs.
All you need to do is:

find C* -type f | split -l 1000
for f in x??; do ./a.out $(cat $f); done
# merge the profiles

[oliverchang]

a standalone build is easier to reason about for anyone who is not familiar with our tools. It's also much easier to reproduce bugs with it outside of our environment (especially if e.g. msan is involved).

That should be part of the upstream build system right? The proposed changes here will require interacting with our tools anyway (docker/helper etc).

Since we already use different compiler flags to build for coverage, why not also link it against a different main?

Why use a more limited main when we can just use libFuzzer's? :)

But this complication is very easy to overcome.

This still seems unnecessarily complicated from an infra standpoint...

[kcc]

That should be part of the upstream build system right?

Should, but we can't really enforce it.

Why use a more limited main when we can just use libFuzzer's? :)

Because it adds complexity :)
E.g. you either should not instrument it, or you should filter out the results of instrumenting it.