feat(forge): simulating multiple transactions in a single test with a dependency tree

[gsalzer]

Component

Forge

Describe the feature you would like

With forge, is it possible to simulate multiple transactions (in a single block or in multiple ones) within a single test? If not, are there plans to extend forge into this direction?

Currently, the answer to the first question seems to be yes and no. Yes, because we can set the block number, the sender, tx.origin etc to multiple values within a test. No, because not all aspects of a transaction are simulated faithfully. As an example, the code of a contract that has self-destructed is not removed, even if we increase the block number (in the hope to start a new transaction with the new block). Confusingly, the code is removed if the contract already selfdestructs in function setUp().

Can we assume that each test is a separate transaction? If yes, is there a way of running tests in a fixed order, without resetting the environment in between? Would this be a way to simulate a sequence of transactions? How precise is this simulation?

Additional context

No response

[gsalzer]

As a concrete example: Suppose I'm testing a wallet that, within the same transaction, is created at a specific address, performs some tasks, and then selfdestructs. How do I check that after the transaction, the code is indeed gone?

[grandizzy]

that's quite similar with #1543 (see comment there why code is removed if selfdestruct in setup. I think would be neat to be able to chain multiple tests, like testB to operate on state after testA and should also solve #1543 will look into complexity of this change and discuss with team

[gsalzer]

Thanks for considering an extension. Yes, this looks like the same issue.

But actually, my desires would go even further. Suppose we want to demonstrate that the order of transactions has/does not have an effect. Then I would like to run a test in forge, remember the results, run another test (same initial setup, but different (trans)actions), and then compare the results of the two tests. So the idea would be to have not only a single sequence of transactions within a test (like we are just discussing), but several of them in parallel from the same initial conditions. Is this something that forge possibly can do, or is this totally against the basic concept of forge?

[grandizzy]

what I was thinking at was to have capability of inline configuring a test like

/// forge-config: default.unit.prereqs=create_contract,selfdestroy_contract`
function testSelfdestroy() public

where testSelfdestroy has create_contract and selfdestroy_contract functions as prereqs, in specified order. I think this should cover your scenario?

[gsalzer]

what I was thinking at was to have capability of inline configuring a test like [...]. I think this should cover your scenario?

Looks great, seems to solve my original problem!

Do you see a possibility to use forge for a test that compares the results of several sequences of transactions that are executed from the same initial conditions, e.g., to demonstrate that there is a transaction order dependence? As I understand it, your extension would allow us to execute a sequence of transactions within a single test, which improves the current situation with at most two transactions (the setup and the test).

[grandizzy]

what I was thinking at was to have capability of inline configuring a test like [...]. I think this should cover your scenario?

Looks great, seems to solve my original problem!

Do you see a possibility to use forge for a test that compares the results of several sequences of transactions that are executed from the same initial conditions, e.g., to demonstrate that there is a transaction order dependence? As I understand it, your extension would allow us to execute a sequence of transactions within a single test, which improves the current situation with at most two transactions (the setup and the test).

hm, you mean like instead
setup -> [tx1, tx2,.. ] -> test
to have something like
setup -> shuffle[tx1, tx2,.. ] -> test
?
Are you starting from a known failure or you want to assert invariants (if so check the invariant testing support in foundry)

[gsalzer]

My current use case is to demonstrate existential properties (mostly known failures: There is a sequence of calls that leads to some problematic behavior). One case that doesn't work well with forge at the moment is the demonstration that there is a transaction order dependence, since this would require

          / txa1, txa2, ... \
setup -> |                   | -> test
          \ txb1, txb2, ... /

test would need access to the states after the txa and txb chain, respectively, in order to compare them and assert a difference. For many cases, it would be sufficient to have just one transaction in each sequence (txa1 and txb1), as it is often possible to simulate the transaction sequence tx1,tx2,... in a single transaction (say, a constructor performing a sequence of calls).

setup -> shuffle[tx1, tx2,.. ] -> test
might be a cool feature, too. If shuffle means that forge tests all permutations, then one could use it to demonstrate certain universal properties, like transaction order independence for certain cases. If shuffle just selects one random sequence, then one could still rerun it a certain number of times (like with fuzz tests) to find a transaction order dependence, or assert its absence, with some degree of certainty.

[grandizzy]

setup -> shuffle[tx1, tx2,.. ] -> test might be a cool feature, too. If shuffle means that forge tests all permutations, then one could use it to demonstrate certain universal properties, like transaction order independence for certain cases.

I think we could have a strategy to test all permutations, let's have basic support in and we can expand after.

[gsalzer]

@grandizzy Thanks for adding beforeTestSetup, works great. I also experimented with the --isolate flag, but prefer your approach. With --isolate, each top-level call becomes a separate transaction (without being visible in the trace as such), and I found it tricky if I wanted to get several calls executed in a single transaction. Moreover, it doesn't seem to possible to specify in the test file itself that it should be run in isolation mode, which may give misleading results if one forgets to specify the flag.

[gsalzer]

Do you have any ideas how one could have a test comparing the results of two tests? Say, run several transactions in one order and also in a another, different one, and then assert that the two states differ (thus demonstrating a transaction order dependence)?

[grandizzy]

Do you have any ideas how one could have a test comparing the results of two tests? Say, run several transactions in one order and also in a another, different one, and then assert that the two states differ (thus demonstrating a transaction order dependence)?

hey @gsalzer maybe you could use startStateDiffRecording / stopAndReturnStateDiff cheatcodes to accomplish such?
https://book.getfoundry.sh/cheatcodes/start-state-diff-recording
https://book.getfoundry.sh/cheatcodes/stop-and-return-state-diff

[stalinMacias]

Hey @grandizzy Do you know how could I run thousands of tokens transfers on a single test, and that each transfer would be treated as an individual tx?
The problem I'm having is that the test runs out of gas while doing the transfers, and what I want to test is something that involves doing thousands of transfers to an account over different timestamps, let's say, I'd want to reproduce in the same test what would happen if I'd execute thousands of individual token transfers on a chain.

• I'd like to run a test where each transfer would be an individual tx and after I've completed all the transfers, to test if the execution of a function reverts because of all the updates that were made by each individual transfer.

[grandizzy]

@stalinMacias did you try isolation mode?

--isolate
          Whether to enable isolation of calls. In isolation mode all top-level calls are executed
          as a separate transaction in a separate EVM context, enabling more precise gas accounting
          and transaction state changes

This or beforeTestSetup should work I think

[stalinMacias]

@grandizzy Thanks for the super quick answer. I've just tried but the out of gas error still shows up.
This is what I'm trying to do, but execution reverts after aprox 15k transfers

vm.startPrank(maliciousUser);
    for(uint i = 0; i < 12_000; i++) {
      token.transfer(legitUser, 1);    }
    for(uint i = 0; i < 12_000; i++) {
      token.transfer(legitUser, 1);
    }
    vm.stopPrank();

Any other idea that comes up to your mind?

[grandizzy]

hm, maybe bump the gas limit
gas_limit = "18446744073709551615"

Or split the 2 for loops in multiple loops and calls / txes and configure them in beforeTestSetup