Skip to content

feat(forge): Remove proptest runner from fuzzed tests #11061

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 58 commits into from
Aug 15, 2025
Merged

feat(forge): Remove proptest runner from fuzzed tests #11061

merged 58 commits into from
Aug 15, 2025

Conversation

grandizzy
Copy link
Collaborator

@grandizzy grandizzy commented Jul 22, 2025 •
edited
Loading

Motivation

Solution

  • consolidate corpus config in FuzzCorpusConfig and use for both fuzz and invariants
  • decouple CorpusManager from invariant config and reuse in fuzz tests (fuzz corpus is same as invariant but with a single tx; BasicTxDetails used in for both strategies and same corpus management). move logic for abi mutate in abi_mutate and use in both fuzz / invariants
  • abi mutate strategies for uint, int, address and fixed bytes inspired from ityfuzz: BitFlipMutator / ByteInterestingMutator / WordInterestingMutator / DwordInterestingMutator / IncDecValue (applicable for uint, int only). Mutation of custom structs, tuples and dynamic arrays (increase / decrease size, mutate random element) https://github.com/fuzzland/ityfuzz/blob/master/src/mutation_utils.rs#L449-L452
  • remove fuzz proptest, introduce time based campaigns
  • remove test data refcells for invariants and fuzz, no longer needed without proptest runner (should also improve perf)
  • unit tests

Maze comparison

Maze PR coverage guided & abi mutations v1.2.3
0 15 (invariants 16, 17, 18, 19, 27, 28, 29, 30, 34, 35, 38, 41, 42, 5, 9) 13 (invariants 16, 17, 18, 19, 27, 29, 30, 34, 35, 38, 42, 5, 9)
1 15 (invariants 12, 16, 17, 25, 26, 31, 32, 33, 37, 38, 39, 46, 47, 48, 5) 14 (invariants 12, 16, 17, 25, 26, 31, 33, 38, 39, 46, 47, 48, 5, 7)
2 16 (invariants 13, 14, 2, 26, 27, 29, 31, 33, 38, 39, 4, 40, 42, 43, 44, 46) 16 (invariants 13, 14, 2, 26, 27, 29, 31, 33, 38, 39, 4, 40, 42, 43, 44, 46)
3 13 (invariants 1, 10, 12, 16, 25, 3, 33, 36, 40, 41, 6, 8, 9) 12 (invariants 10, 12, 16, 25, 3, 33, 36, 40, 41, 6, 8, 9)
4 18 (invariants 10, 11, 17, 22, 24, 27, 28, 32, 33, 36, 37, 38, 39, 41, 43, 45, 5, 6) 15 (invariants 10, 11, 17, 22, 27, 28, 32, 33, 36, 37, 41, 43, 5, 6)

PR Checklist

  • Added Tests
  • Added Documentation
  • Breaking changes

@grandizzy grandizzy changed the title WIP: Remove proptest from fuzzed tests WIP: Remove proptest runner from fuzzed tests Jul 22, 2025
@grandizzy grandizzy self-assigned this Jul 22, 2025
@grandizzy grandizzy moved this to In Progress in Foundry Jul 22, 2025
@grandizzy grandizzy mentioned this pull request Jul 24, 2025
Merged
@grandizzy grandizzy added this to the v1.4.0 milestone Jul 29, 2025
DaniPopes
DaniPopes reviewed Aug 12, 2025
View reviewed changes
grandizzy and others added 6 commits August 12, 2025 16:24
@grandizzy @DaniPopes
Update crates/evm/evm/src/executors/corpus.rs
3ead4b3 
Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>
@grandizzy @DaniPopes
Update crates/evm/evm/src/executors/corpus.rs
b3b06ed 
Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>
@grandizzy @DaniPopes
Update crates/evm/evm/src/executors/corpus.rs
1a4ab5e 
Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>
@grandizzy
Fix typo
423f2d9
@grandizzy
use wrapping add / sub for inc/dec mutation
387db1c
@grandizzy
Validate inc/dec, better word mutation, tests
3757c7e
@grandizzy grandizzy requested a review from DaniPopes August 13, 2025 07:25
DaniPopes
DaniPopes reviewed Aug 13, 2025
View reviewed changes
DaniPopes
DaniPopes reviewed Aug 13, 2025
View reviewed changes
DaniPopes
DaniPopes reviewed Aug 13, 2025
View reviewed changes
DaniPopes
DaniPopes previously approved these changes Aug 14, 2025
View reviewed changes
Copy link
Member

@DaniPopes DaniPopes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, last nits

@grandizzy @DaniPopes
Update crates/evm/fuzz/src/strategies/param.rs
f777b95 
Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>
grandizzy and others added 2 commits August 14, 2025 16:05
@grandizzy @DaniPopes
Update crates/evm/fuzz/src/strategies/param.rs
f33b356 
Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>
@grandizzy
Rename mutate_random_array/tuple_value
89189da
@grandizzy grandizzy merged commit 3412508 into foundry-rs:master Aug 15, 2025
22 checks passed
@github-project-automation github-project-automation bot moved this from In Progress to Done in Foundry Aug 15, 2025
@grandizzy grandizzy deleted the no-fuzz-proptest branch August 15, 2025 05:17
@grandizzy grandizzy moved this from Done to Completed in Foundry Aug 18, 2025
MerkleBoy pushed a commit to MerkleBoy/foundry that referenced this pull request Sep 17, 2025
@grandizzy @DaniPopes @MerkleBoy
feat(forge): Remove proptest runner from fuzzed tests (foundry-rs#11061)
2de9fc7 
* Remove proptest from fuzzed tests

* Persist and replay fuzz failure

* Nits

* add fuzz coverage metrics config, cleanup

* rm unec RefCell, refactor

* Add inline fuzz coverage metrics

* Same corpus manager for fuzz and invariant tests

* Fix fmt

* Unify configs

* Use corpus manager / abi mutation for fuzz tests - Move hit count in corpus - add mutate_abi and evict_oldes_corpus fns TODO: load persisted corpus from file on fuzz test start

* Replay corpus for fuzz tests as well

* Add corpus replay failures for fuzz tests too

* cleanup fuzz corpus on forge clean, display errors

* more abi mutations

* rand bit flip mutation + add support for more types

* clippy

* More mutations - interesting byte and word

* More mutations, traces

* Review changes

* add mutators traits

* Simplify

* Update crates/evm/fuzz/src/strategies/mod.rs

Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>

* Avoid clone when mutating array and tuples

* Update crates/evm/evm/src/executors/corpus.rs

Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>

* Update crates/evm/evm/src/executors/corpus.rs

Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>

* Update crates/evm/evm/src/executors/corpus.rs

Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>

* Fix typo

* use wrapping add / sub for inc/dec mutation

* Validate inc/dec, better word mutation, tests

* Remove unused validation, early return on validation

* mutators to operate on &mut [u8], instrument, cleanup

* Mutate fixedbytes, more address mutations

* Unify mutators

* Add word mutation unit test

* Address nits

* Update crates/evm/fuzz/src/strategies/param.rs

Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>

* Update crates/evm/fuzz/src/strategies/param.rs

Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>

* Rename mutate_random_array/tuple_value

---------

Co-authored-by: DaniPopes <57450786+DaniPopes@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DaniPopes DaniPopes DaniPopes approved these changes

@klkvr klkvr Awaiting requested review from klkvr

@mattsse mattsse Awaiting requested review from mattsse mattsse is a code owner

@yash-atreya yash-atreya Awaiting requested review from yash-atreya

@zerosnacks zerosnacks Awaiting requested review from zerosnacks zerosnacks is a code owner

Assignees

@grandizzy grandizzy

Labels

None yet

Projects

Foundry
Status: Completed

Milestone

v1.4.0

Development

Successfully merging this pull request may close these issues.

feat(fuzz): add coverage guided fuzzing for stateless tests (currently only invariant mode works)

2 participants

@grandizzy @DaniPopes