Execution traces for call sequences failing tests #105

Xenomega · 2023-03-13T06:17:28Z

This PR adds execution traces to call sequences which fail tests. All shrunken sequences will now be replayed to capture execution traces for discern what caused failures within a single transaction.

It currently supports:

Other TODOs:

Add a --trace-all flag and associated config option to indicate the fuzzer should attach execution traces to all finalized shrunken sequences.
Add to tests to target remaining untested trace operations (self destruction)

A later PR should add support for on-chain console.log(), which should be incorporated in the trace. An issue should be opened to track that prior to merging this PR.

…est call sequences

* Added helpers to extract Solidity errors/panics (now used by assertion test provider and execution tracer) * Added fuzzer tests to verify execution trace output contains revert reasons and assertion failures * Updated README with required solidity version for unit tests

# Conflicts: # fuzzing/calls/call_sequence.go # fuzzing/fuzzer_worker.go

README.md

chain/test_chain.go

fuzzing/executiontracer/execution_tracer.go

…-execute a given call sequence without any custom checks * Updated FuzzerWorker to re-execute the finalized shrunken sequence before performing the "finalized shrunken sequence" callback, so all providers can assume state is post-execution. * Updated assertion test provider to attach an execution trace for the last call in a failing sequence * Updated property test provider to attach execution traces for the last call in a failing sequence, and for its property test * Updated the fuzzer worker so if a "trace all" flag is provided, it will attach execution traces to all finalized shrunken call sequence elements prior to returning the results to any test provider * Updated TestChain to provide a method to get the post-execution state root hash for a given block number

Xenomega · 2023-03-15T02:29:25Z

The last commit I pushed makes the following changes:

FuzzerWorker will re-execute the finalized shrunken sequence before calling back to test providers (there was previously a deviation where the worker's chain state would be post-execution of call sequences for the initial test call back, and the shrink verifier, but not the final callback). This will let test providers consistently assume the state is post-execution of the call sequence.
[If "trace all" is enabled] FuzzerWorker attach traces to all elements of a finalized shrunken sequence, before returning it back to the test provider
AssertionTestCaseProvider attaches a trace to the last call always
PropertyTestCaseProvider attaches a trace to the last call, and the property test call, always

The "trace all" option is still currently a hardcoded bool and needs to be added to the config, along with a CLI flag to allow override enabling of it (even if config doesn't have it set).

…and "code"

fuzzing/calls/call_sequence.go

- Removed the hardcoding of traceAll in fuzzerWorker. - Added TraceAll config option - Added trace-all flag to CLI that can override config option if necessary

anishnaik · 2023-03-15T23:25:05Z

The following has changed in the last 3 commits:

We have added the traceAll config variable as well as the --trace-all CLI flag that can override the traceAll config variable. traceAll works as described by @Xenomega in his previous comment
For call frames where no code is executed, we updated the visual formatting of it to make it look a bit cleaner
Fix a small bug with error handling in call_sequence.go.

Xenomega · 2023-03-18T10:19:56Z

The last commit added support for cheat code contracts in execution traces. I might do some further cleanup later today, otherwise it should be good to go.

Note: CI failed on the last commit because markdown lint was flagging URLs in this repo as dead links, though it should have passed. I'm going to re-run it in a few hours and assume it was some glitch (as they are valid links). Otherwise I'll look into if a new release treats private repos differently.

EDIT: CI is failing due to tcort/markdown-link-check#246 , a new release broke the config file support...

EDIT (2): Applied a temporary fix for CI to master and merged it into this branch.

…trace code)

…pilation-related "abiutils" package.

* Added test for contract deployments

…o track which part of call data represents ABI constructor args during deployment

anishnaik · 2023-03-21T00:24:43Z

compilation/abiutils/solidity_events.go

+	coreTypes "github.com/ethereum/go-ethereum/core/types"
+)
+
+// UnpackEventAndValues takes a given contract ABI, and VM return values, and


incomplete comment

Updated in 56caf88

anishnaik · 2023-03-21T00:56:54Z

fuzzing/test_case_property_provider.go

@@ -76,47 +78,61 @@ func (t *PropertyTestCaseProvider) isPropertyTest(method abi.Method) bool {

 // checkPropertyTestFailed executes a given property test method to see if it returns a failed status. This is used to
 // facilitate testing of property test methods after every call the Fuzzer makes when testing call sequences.
-func (t *PropertyTestCaseProvider) checkPropertyTestFailed(worker *FuzzerWorker, propertyTestMethod *contracts.DeployedContractMethod) (bool, error) {
+// The property test is executed over the state indicating by the provided state root hash.


not sure what we mean by "provided state root hash" here?

This was old from temporary code changes, removed in 56caf88

anishnaik

bada bing bada boom

* Introduction of execution trace attaching and displaying for failed test call sequences * Added helpers to extract Solidity errors/panics (now used by assertion test provider and execution tracer) * Added fuzzer tests to verify execution trace output * Updated README with required solidity version for unit tests * Provided a wrapper for ExecuteCallSequenceIteratively, to simply re-execute a given call sequence without any custom checks * Updated FuzzerWorker to re-execute the finalized shrunken sequence before performing the "finalized shrunken sequence" callback, so all providers can assume state is post-execution. * Updated assertion test provider to attach an execution trace for the last call in a failing sequence * Updated property test provider to attach execution traces for the last call in a failing sequence, and for its property test * Updated the fuzzer worker so if a "trace all" flag is provided, it will attach execution traces to all finalized shrunken call sequence elements prior to returning the results to any test provider * Updated TestChain to provide a method to get the post-execution state root hash for a given block number * Cleaned up ABI error and event helpers by splitting them off to a compilation-related "abiutils" package. --------- Co-authored-by: anishnaik <anish.r.naik@gmail.com>

anishnaik and others added 11 commits March 7, 2023 19:14

useless commit

f65bcb0

hacky way to get revert reason

f046909

Introduction of execution trace attaching and displaying for failed t…

3960df8

…est call sequences

Merge remote-tracking branch 'origin/master' into temp/event-logging

7134493

# Conflicts: # fuzzing/calls/call_sequence.go # fuzzing/fuzzer_worker.go

Added event support to execution tracer and traces

600193e

Removed old version of event emission test

c59cd55

Removed old call sequence event code

56d0f1d

Improve indentation of execution traces for better readability

b0a02a7

Fix lint issue

b32085d

Return data unpacking fix

30dfa43

Xenomega requested a review from anishnaik March 13, 2023 09:14

anishnaik reviewed Mar 13, 2023

View reviewed changes

README.md Show resolved Hide resolved

anishnaik reviewed Mar 13, 2023

View reviewed changes

chain/test_chain.go Show resolved Hide resolved

anishnaik reviewed Mar 14, 2023

View reviewed changes

fuzzing/executiontracer/execution_tracer.go Outdated Show resolved Hide resolved

Xenomega added 2 commits March 14, 2023 22:41

Updated execution tracer to better record runtime bytecodes for "to" …

7663797

…and "code"

Added self destruct test

863ed0f

anishnaik reviewed Mar 15, 2023

View reviewed changes

fuzzing/calls/call_sequence.go Show resolved Hide resolved

anishnaik added 3 commits March 15, 2023 12:03

add traceAll config option

fc95b94

- Removed the hardcoding of traceAll in fuzzerWorker. - Added TraceAll config option - Added trace-all flag to CLI that can override config option if necessary

updated formatting for call frames that do not execute code

a0c9e5f

fix bug in attachExecutionTraces

f26f976

anishnaik and others added 5 commits March 15, 2023 19:27

comment update

45ecd9e

lint updates

4951cd6

another formatting update

8209b6a

remove ChildCallFrames parameter and add as getter

f7739b5

First pass at cheat code contract support in execution traces

9b3e407

Xenomega added 7 commits March 18, 2023 18:29

Merge branch 'master' into dev/execution-tracing

555f6c6

Added support for custom solidity errors (TODO: cleanup of execution …

77b2760

…trace code)

Cleaned up ABI error and event helpers by splitting them off to a com…

2e9044c

…pilation-related "abiutils" package.

* More cleanup of execution tracing code

704bcf2

* Added test for contract deployments

Added a ConstructorArgsData field to execution tracing's CallFrame, t…

10c291a

…o track which part of call data represents ABI constructor args during deployment

Fix lint issue

d593592

Add a test case for call argument resolution

1365540

Xenomega mentioned this pull request Mar 20, 2023

Milestone tracking: Initial release #101

Closed

11 tasks

anishnaik reviewed Mar 21, 2023

View reviewed changes

Xenomega added 2 commits March 20, 2023 22:12

Added support for proxy calls to execution tracer + tests

837b88b

Updated doc comments

56caf88

anishnaik approved these changes Mar 22, 2023

View reviewed changes

Xenomega merged commit 3da5b3c into master Mar 22, 2023

Xenomega deleted the dev/execution-tracing branch March 22, 2023 14:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Execution traces for call sequences failing tests #105

Execution traces for call sequences failing tests #105

Xenomega commented Mar 13, 2023 •

edited

Loading

Xenomega commented Mar 15, 2023 •

edited

Loading

anishnaik commented Mar 15, 2023

Xenomega commented Mar 18, 2023 •

edited

Loading

anishnaik Mar 21, 2023

Xenomega Mar 21, 2023

anishnaik Mar 21, 2023

Xenomega Mar 21, 2023

anishnaik left a comment

Execution traces for call sequences failing tests #105

Execution traces for call sequences failing tests #105

Conversation

Xenomega commented Mar 13, 2023 • edited Loading

Xenomega commented Mar 15, 2023 • edited Loading

anishnaik commented Mar 15, 2023

Xenomega commented Mar 18, 2023 • edited Loading

anishnaik Mar 21, 2023

Choose a reason for hiding this comment

Xenomega Mar 21, 2023

Choose a reason for hiding this comment

anishnaik Mar 21, 2023

Choose a reason for hiding this comment

Xenomega Mar 21, 2023

Choose a reason for hiding this comment

anishnaik left a comment

Choose a reason for hiding this comment

Xenomega commented Mar 13, 2023 •

edited

Loading

Xenomega commented Mar 15, 2023 •

edited

Loading

Xenomega commented Mar 18, 2023 •

edited

Loading