Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for optimization mode #75

Merged
merged 12 commits into from
Jun 21, 2023
Merged

Add support for optimization mode #75

merged 12 commits into from
Jun 21, 2023

Conversation

tarunbhm
Copy link
Contributor

@tarunbhm tarunbhm commented Dec 27, 2022

closes #71

  • unit test cases

…g considerations:

- optimization functions can have a different prefix that can be configred in init mode
- optimization test functions should be view/pure
- optimization test cases fail in case of an error in EVM call
- optimization test cases never without if there is no err anywhere
- print a message for passed test cases which shows final optimized value
- unit test cases are not yet implemented, manual local testing is done
@tarunbhm tarunbhm linked an issue Dec 27, 2022 that may be closed by this pull request
@tarunbhm
Copy link
Contributor Author

tarunbhm commented Dec 28, 2022

Seems like there is race condition in the value being computed because the same value is updated by all the workers. I think ideally we should have an instance of value for every worked index and reset it on WorkerDestroyed event. One more thing which is not clear to me is how do we decide if we have reached the maximum possible value for the optimization test case? I mean our fuzzer may stop before reaching the max value because of test limit or time limit.

Looking through the code I also see that that testCase.workerStates is reused between workers on same index because it is not reset when a worker is destroyed. It may not be a problem because all the contracts are deployed again when a new worker is created and this triggers testCase.onWorkerDeployedContractAdded method which updates the testCase.workerStates with new values. It could become a problem if in future we make changes in initial deployment procedure based on worker index or some other parameter. cc @anishnaik @Jaime-Iglesias @Xenomega

Update: I have fixed it with the mutex for now

@tarunbhm tarunbhm marked this pull request as ready for review December 28, 2022 07:31
@Xenomega Xenomega mentioned this pull request Mar 31, 2023
7 tasks
@anishnaik
Copy link
Collaborator

@tarunbhm do you want to go ahead and update this PR with the latest master branch? Happy to get this merged ASAP. Apologies for the delay.

@CLAassistant
Copy link

CLAassistant commented May 8, 2023

CLA assistant check
All committers have signed the CLA.

@ggrieco-tob
Copy link
Member

I'm taking care of this. It seems that it should be ready for another review. I added some code to actually show the sequence of transactions that maximized certain function (otherwise, it was not useful, since these tests should never fail)

@anishnaik
Copy link
Collaborator

anishnaik commented Jun 20, 2023

Hey @tarunbhm + @ggrieco-tob so I did a review of this PR and I added the capability to shrink a call sequence that optimized the test case's value. This was something that was missing from the PR. However, during this development I realized that we will have to put this PR on hold for now b/c of the inefficiency of the call sequence shrinking. The reason it is inefficient is because every time the value is maximized, we send a shrink request. however, after the shrinking is done, we do not mark the test case as failed b/c the next call or call sequence could in fact maximize the value even more. Then, another computationally intensive shrink request can be triggered on the next call in the call sequence. Thus, optimization mode doesn't really have a "stop" - it keeps going as long as the fuzzer keeps going. Since we cannot do a one-time shrink request at the moment, this PR will have to be on hold.

Ideally, we could call FuzzerWorker.shrinkCallSequence directly but that does not work given the way medusa is structured and how the fuzzing loop is set up. We will have to wait until we integrate a higher-level testing API so that we can support arbitrary shrinking requests. This would allow us to only shrink optimization call sequences one time instead of a gajillion times.

@anishnaik anishnaik merged commit 5a7d044 into master Jun 21, 2023
@anishnaik anishnaik deleted the 71-optimization-mode branch June 21, 2023 17:09
damilolaedwards pushed a commit that referenced this pull request Nov 16, 2023
integrate optimization mode into fuzzer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add support for optimization mode
4 participants