sfuzz

[zerokay]

compare against core fuzzers

[jonathanmetzman]

Can you tell us a little about sfuzz and yourself?

[zerokay]

Can you tell us a little about sfuzz and yourself?

I am a security researcher and I had a PR before, but it is possible that I accidentally closed and deleted it. I have modified the energy scheduling algorithm, so I want to test its effectiveness.

[zerokay]

@jonathanmetzman Is there any problem? If not, can you run the experiment? Thank you.

[vanhauser-thc]

I was curious what this changes :)

this change only affects the RARE schedule, a schedule that is great to add in a fuzzing campaign, but is outperformed here on fuzzbench by fast and explore easily:

    fuzz_p2 = 3* next_pow2(afl->n_fuzz[q->n_fuzz_entry]);  / from 1*

this will make a too small difference to be visible in the results:

      if (q->favored) factor *= 1.25;  // from 1.15

[zerokay]

Yep, I made minor modifications and would like to see the effect of this modification. In the future, i will make improvements based on the benchmark.

[DonggeLiu]

There was an error in cloud build, it has been fixed now.
I will close&reopen the PR to apply the fix.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-08-04-sfuzz --fuzzers sfuzz

[DonggeLiu]

BTW, you don't have to add experiment requests to service/experiment-requests.yaml anymore, as we are running experiments from this PR with /gcbrun commands :)
If you'd like, you can write the experiment request in the format above.

I did not include the core fuzzers in the command, because FB can use their statistics from their previous experiments.

[DonggeLiu]

/gcbrun run_experiment.py -a --experiment-config /opt/fuzzbench/service/experiment-config.yaml --experiment-name 2023-08-04-sfuzz --fuzzers sfuzz

[DonggeLiu]

I am pretty sure the experiment is running, despite that the CI says it failed.

Experiment data and results will be available later at:
The experiment data.
The experiment report.

[zerokay]

@alan32liu Hi, I browsed the experimental data and found that it does not include woff2_convert_woff2ttf_fuzzer, but the generated report contains its comparison. Could you please remove this experimental comparison (woff2_convert_woff2ttf_fuzzer) and generate a new report?

[DonggeLiu]

@alan32liu Hi, I browsed the experimental data and found that it does not include woff2_convert_woff2ttf_fuzzer, but the generated report contains its comparison. Could you please remove this experimental comparison (woff2_convert_woff2ttf_fuzzer) and generate a new report?

Hi @zerokay, the report can be regenerated based on this guide : )

The root cause of the missing experiment result of woff2_convert_woff2ttf_fuzzer is unclear to me, but I submitted a PR (above) that should be able to fix it.

If you'd like to re-run sfuzz on the benchmark, we can also start another experiment and merge the results later : )

[zerokay]

@alan32liu Hi, I browsed the experimental data and found that it does not include woff2_convert_woff2ttf_fuzzer, but the generated report contains its comparison. Could you please remove this experimental comparison (woff2_convert_woff2ttf_fuzzer) and generate a new report?

Hi @zerokay, the report can be regenerated based on this guide : )

The root cause of the missing experiment result of woff2_convert_woff2ttf_fuzzer is unclear to me, but I submitted a PR (above) that should be able to fix it.

If you'd like to re-run sfuzz on the benchmark, we can also start another experiment and merge the results later : )

Please re-run sfuzz on the benchmark, thanks.

[DonggeLiu]

Please re-run sfuzz on the benchmark, thanks.

Sure, let's do that after my fix is merged.

[zerokay]

Sure, let's do that after my fix is merged.

@alan32liu Hi，is it okay now? If not, could you remove the experiment and generate a new report?

[DonggeLiu]

Sure, let's do that after my fix is merged.

@alan32liu Hi，is it okay now? If not, could you remove the experiment and generate a new report?

It's not ready yet, we are looking into the root case of the bug.
Meanwhile, you can always generate the report by yourself via the steps on this guide.
If you don't include the benchmark in the parameters, it won't affect the overall result.

[zerokay]

It's not ready yet, we are looking into the root case of the bug. Meanwhile, you can always generate the report by yourself via the steps on this guide. If you don't include the benchmark in the parameters, it won't affect the overall result.

Okay, I'll generate the report locally first. And I want to know all the experimental results, so please re-run sfuzz on the benchmark after fixing the error, thank you.

[zerokay]

@alan32liu Hi, is it okay now? If not, could you run the whole experiment again?

[DonggeLiu]

@alan32liu Hi, is it okay now?

I did not have the chance to close that PR due to my limited time working on this.
That PR shows a way to mitigate the bug, but I did not investigate the root cause and fix it.
If you are interested in it (we did not see this in other experiments, though), we would appreciate it if you could suggest some possible causes/fixes.

If not, could you run the whole experiment again?

Re-running the experiment will not give us any new results without fixing the bug.
As mentioned before, you can always generate the report by yourself via the steps in this guide.
It won't take a lot of effort, and all you need is the data.csv.gz file, which is already here.