Skip to content

Display predicted coverage curves for different generation strategies #189

@Liam-DeVoe

Description

@Liam-DeVoe

(Originally: #177 (comment))

We should communicate "how good" fuzzing is vs just running blackbox Hypothesis for longer. One nice way to do this is displaying the estimated coverage curves on each per-test page for blackbox Hypothesis vs coverage-guided HypoFuzz. (We could also split coverage-guided into "pure mutation" and "whatever mixture HypoFuzz uses", but it's not clear to me that pure-mutation is a useful metric).

The hard part here is getting estimators for campaign progress. Blackbox coverage is stationary and so a standard curve fit should perform relatively well, but anything based on mutation is non-stationary. #11 has related discussion and research about better progress estimators for fuzzing.

Note that deciding the origin of "new coverage" is biased if done naively: a branch may be covered first by a mutation-generated input, and is then not considered new for blackbox the first time a blackbox-generated input hits that branch. We should track behaviors and fingerprints separately by generation strategy for the purposes of calculating generation strategy effectiveness.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions