Seed experiment

[tokatoka]

This is a experiment to see how five fuzzers (afl, aflpp, libfuzzer, honggfuzz, libafl) behaves when supplied with a empty seeds and saturated seed corpus.

First I want do an empty seed experiment.
Later i will change it to use ossfuzz corpus

[tokatoka]

The command is

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty_seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty_seeds-2 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

@DonggeLiu
Could you run the command?

[DonggeLiu]

gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-empty_seeds --fuzzers aflplusplus libafl honggfuzz libfuzzer afl aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

Sure, but there are 2 issues:

1. Could you please follow the same date format in experiment names? 2024-08-14. This help us keep the cloud storage more organized.
2. Like last time, this experiment requests too many fuzzers. Could you please split them into 2 experiments. Let's keep #fuzzers <= 5 in each exp.

[tokatoka]

ok i edited and fixed the issues
#2025 (comment)

[tokatoka]

this is dependent on #2023 #2021

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-15-empty_seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

[DonggeLiu]

Ops, experiment name cannot contain _.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-15-empty-seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-14-empty-seeds-2 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

[tokatoka]

it looks like it didn't run
can you check what is the reason?

[DonggeLiu]

2024-08-14-empty-seeds-2

it looks like it didn't run
can you check what is the reason?

I think it is because of the same issue I fixed in #2023
Now that you have updated your branch, let me re-run the exp.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-empty-seeds-1 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-17-empty-seeds-2 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

[DonggeLiu]

Experiment 2024-08-17-empty-seeds-1 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

Experiment 2024-08-17-empty-seeds-2 data and results will be available later at:
The experiment data.
The experiment report.
The experiment report(experimental).

[DonggeLiu]

Hi @tokatoka, I started two test exp above.
If they work well, could you please rebase to adopt this change?
I forgot to revert temp changes in a previous PR.

Thanks!

[tokatoka]

hi, i merged the master branch to revert it.
now it's ready

[tokatoka]

now it's truly okay (with the green ci)
could you run the same command? @DonggeLiu

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-21-empty-seeds-1 --fuzzers aflplusplus libafl honggfuzz libfuzzer afl

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-21-empty-seeds-2 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

[tokatoka]

hmm the experiment-data has data in its folder but the report is not generated
do you know the reason? @DonggeLiu

[DonggeLiu]

Let's debug this on the other PR #2028
This experiment also had many runtime crashes, for example.

[tokatoka]

I updated libafl commit here too.
Can we start the experiment here?

[DonggeLiu]

I updated libafl commit here too. Can we start the experiment here?

Sure, I wonder if that heisenbug will haunt us here.
Let's run for once, and if the bug reappears, I will create a new PR with my debug logs.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2024-08-27-empty-seeds-1 --fuzzers aflplusplus_empty_seeds libafl_empty_seeds honggfuzz_empty_seeds libfuzzer_empty_seeds afl_empty_seeds

[tokatoka]

No it doesn't work.
Just like before. The experiment-data is there but the report is missing

[tokatoka]

@DonggeLiu
How can I help to debug this stuff..?

[DanBlackwell]

@DonggeLiu How can I help to debug this stuff..?

#2000 (comment) Could it be related to this? maybe you can try run just zlib on it's own and see if there's the same error?

[tokatoka]

@DonggeLiu Can you create a debug branch for this too..?
For this one, 0 reports were created

[DonggeLiu]

yep I saw a lot of "Fuzz target binary not found." errors:

It's likely the same error again.
Weird that it stopped occurring for a year and suddenly came back.

[DonggeLiu]

@DonggeLiu How can I help to debug this stuff..?

#2000 (comment) Could it be related to this? maybe you can try run just zlib on it's own and see if there's the same error?

Thanks for helping @DanBlackwell .
We will investigate more in a new PR.

[DonggeLiu]

@DonggeLiu Can you create a debug branch for this too..? For this one, 0 reports were created

I will need your help to resolve the conflict before I can rebase and create a new PR : )

Auto-merging fuzzers/libafl/builder.Dockerfile
CONFLICT (content): Merge conflict in fuzzers/libafl/builder.Dockerfile
error: could not apply 8b14d84f... Update LibAFL (#2036)
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Could not apply 8b14d84f... Update LibAFL (#2036)

Or you could cherry-pick my commits from #2038.

[tokatoka]

resolved it