add fuzzit.dev continous fuzzing integration

[yevgenypats]

Hi Team,

The PR is ready for review: more instruction in a later comment

This PR adds a continuous fuzzing integration to go-ethereum's travis pipeline via https://fuzzit.dev service.

This continuous fuzzing workflow consist of

Continuously fuzzing up-to-date master generating new test cases, crashes and corpus.

Running quick regression tests with the generated corpus and fixed crashes on every PR to catch bugs earlier

I'm the founder of fuzzit.dev and the service is free for open source projects (you can see some great projects that we serve on our website).

To make integration as easy as possible, we've created an organisation at https://app.fuzzit.dev/orgs/ethereum/dashboard.

To take ownership of the organisation, please login to https://app.fuzzit.dev and let us know your account.

build/fuzzit.sh script is an additional step in Travis pipeline which builds fuzzit targets and uploads them to fuzzit.dev servers for continuous fuzzing.

This PR uses a temporary API key. After merging, please get the API key from https://app.fuzzit.dev/orgs/ethereum/settings and set FUZZIT_API_KEY env variable via Travis UI (https://travis-ci.org/ethereum/go-ethereum/settings).

Let me know if you have any questions.

[karalabe]

What's the difference between Fuzzit and Fuzzbuzz?

[yevgenypats]

Hi Peter, So in General I think the idea is the same. Though - We do have some important differences: 1) Our CLI is open source which I think is crucial especially for sensitive projects like Eth but also a best practice. 2) we run quick fuzz regressions inline on PRs with your CI so you can catch bugs early on before they are merged. This one of our most used features. 2) In my biased opinion we have a bit wider adoption with OSS projects (you can see on our site) 3) In general I guess you should try both and maybe even keep both depending what works best for you and the team. Cheers, Yevgeny

…

On Thu, Aug 8, 2019, 10:49 AM Péter Szilágyi ***@***.***> wrote: What's the difference between Fuzzit and Fuzzbuzz? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19927?email_source=notifications&email_token=AD52CDVFS7QV6OPY65I34XTQDPFYNA5CNFSM4IKDZNXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD32YWPA#issuecomment-519408444>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD52CDR6GXMQF57G4G52YDDQDPFYNANCNFSM4IKDZNXA> .

[yevgenypats]

Also, do you have any idea why the fuzzing step is blocked? We can't test if our PR works.

…

On Thu, Aug 8, 2019, 11:01 AM Yevgeny Pats ***@***.***> wrote: Hi Peter, So in General I think the idea is the same. Though - We do have some important differences: 1) Our CLI is open source which I think is crucial especially for sensitive projects like Eth but also a best practice. 2) we run quick fuzz regressions inline on PRs with your CI so you can catch bugs early on before they are merged. This one of our most used features. 2) In my biased opinion we have a bit wider adoption with OSS projects (you can see on our site) 3) In general I guess you should try both and maybe even keep both depending what works best for you and the team. Cheers, Yevgeny On Thu, Aug 8, 2019, 10:49 AM Péter Szilágyi ***@***.***> wrote: > What's the difference between Fuzzit and Fuzzbuzz? > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#19927?email_source=notifications&email_token=AD52CDVFS7QV6OPY65I34XTQDPFYNA5CNFSM4IKDZNXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD32YWPA#issuecomment-519408444>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AD52CDR6GXMQF57G4G52YDDQDPFYNANCNFSM4IKDZNXA> > . >

[holiman]

Interesting. I guess you've seen this PR: #19910 . To do the same fuzzing on fuzzit.dev, would we define the individual fuzzers within that sh-script somehow?

[yevgenypats]

Yes, Indeed. We wrote it in this PR but for some reason the step is blocked in travis. Do you have any idea why ?

…

On Thu, Aug 8, 2019, 11:16 AM Martin Holst Swende ***@***.***> wrote: Interesting. I guess you've seen this PR: #19910 <#19910> . To do the same fuzzing on fuzzit.dev, would we define the individual fuzzers within that sh-script somehow? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19927?email_source=notifications&email_token=AD52CDRGJSPD3VUCRDLS273QDPI6HA5CNFSM4IKDZNXKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3223UA#issuecomment-519417296>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD52CDX7JPE5DKTI3E6AEOLQDPI6HANCNFSM4IKDZNXA> .

[holiman]

It's building now

[yevgenypats]

cool. I'm on it. probably some travis glitch

[yevgenypats]

Hi Team,

The PR is ready for review.

I don't have access to the wiki so I can't add the instructions of how to compile and run the fuzzers, so I'll put them here in the meantime.

Building the fuzzers

Currently there are 4 go-fuzz fuzzers. I've added a script that compiles all the targets with libFuzzer (so this currently only supported on Linux) ./build/libfuzzer_targets

Continuous Fuzzing Integration with Fuzzit

1. Every push of new code to master build new fuzz targets and updates them on fuzzit servers. The fuzzers run continuously and alert the relevant people of any new crashes. (someone will need to login so I can add him/her to the Eth team and configure the email alerts).
2. The first process generates a corpus which we use during regression. For every pull-request all test-cases and crashes (that were fixed) are downloaded with fuzzit CLI and run against the built fuzzer targets of the PR to catch bugs early on. This will catch both new bugs that the PR might introduce or old bugs that were fixed already.

Local Development

Developers can also run the regression locally via fuzzit CLI to try and debug issues that arise in their PR.

# Current Fuzz Target TARGETS=("bitutil-fuzzer" "bn256-add-fuzzer" "bn256-mul-fuzzer" "bn256-pair-fuzzer")
./fuzzit create job --local ethereum/$TARGET $PATH_TO_FUZZER

Attached is a screenshot of the main page of the dashboard.

Please review.

Cheers,
Yevgeny

[yevgenypats]

ping @holiman @karalabe .

[adamschmideg]

Fuzzbuzz integration looks simpler. So we'll come back to this one after #19910 is sorted out.

[karalabe]

In the mean time go-ethereum was integrated into Google's OSS Fuzz project, so that should cover all continuous fuzzing needs for free.