Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

commands to interact with corpus test results #326

Open
0xalpharush opened this issue Mar 11, 2024 · 3 comments
Open

commands to interact with corpus test results #326

0xalpharush opened this issue Mar 11, 2024 · 3 comments

Comments

@0xalpharush
Copy link
Contributor

I'd like to have some of the features in test-fuzz for working with fuzz test failures:

  • a command to replay failures medusa fuzz --replay [all, $TEST...]
    • it would be nice for each failure to give the path/test name of the entry in $CORPUS_DIR/test_results/ so that it can be replayed individually
    • TBD if granularity for property/ assertion/ specific panic codes is desirable
  • a command, medusa fuzz --display [all, $TEST...` to display the ABI decoded call sequence with function signatures (A.b(1), C.d(2))

It doesn't necessarily have to be a flag like test-fuzz

@anishnaik
Copy link
Collaborator

I think we would make it a separate subcommand. Maybe something along the lines of:
medusa corpus replay [all, $TEST...]

Then, we can also support other subcommands like
medusa corpus shrink to shrink the corpus elements

@0xalpharush
Copy link
Contributor Author

subcommand to replay failures, manually shrink test cases, and maybe shrinking the corpus. I figure the ability to replay one trace and add --trace-all would be useful.

In order to shrink the corpus, will be faster to compare buckets of edges as opposed to all active PC's. I think we should move from the PC for the "is interesting coverage" metric and use the PC, jumpdest
https://github.com/h0mbre/Lucid/blob/3026e7323c52b30b3cf12563954ac1eaa9c6981e/src/coverage.rs#L29-L56

Optionally, the minimization should take into account the gas usage or execution time.

Then, we just run like medusa corpus coverage and re-use all that line based coverage for the reports (we can run it after the campaign by default, too).

So we'd basically have an array of edges hit that are binned into set ranges and only consider changes between the bins interesting like Lucid (AFL style coverage). It helps for stuff like loops where +1 iteration isn't interesting but maybe like +10 is

Sketch of what it looks like: https://github.com/crytic/medusa/tree/wip/edge-coverage-and-hit-count

@0xalpharush
Copy link
Contributor Author

I started working on the shrink command but it still needs work to get the testing providers registered to the chain. The API and hook patterns made this kind of awkward and create a lot of duplicated code without more invasive changes. For shrink the corpus, there needs to be a way to tell whether a "smaller" tx subsumes the coverage of another tx in place of the test case provider. I think this will be easier with edges over PC's but it can likely be done naively with a loop or some bisection maybe...

https://github.com/crytic/medusa/tree/feat/shrink-subcommand

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants