Table tests

[onbjerg]

Component

Forge

Describe the feature you would like

Table tests are a way to generate test cases based on a dataset of parameters (the table), enabling code reuse in tests that use the same assertions but with predefined data. An example of a table test can be found here.

There are a few ways to go about this (after brainstorming with @gakonst) these seem like the most viable options:

Setup/run functions

This approach uses a setUpTable* function that returns the table with testdata and a testTable* that runs assertions on each entry in the table. Due to constraints with Solidity (again) the UX is going to suck a bit:

function setUpTableSums() public returns (bytes[] memory) {
  bytes[] memory entries = new bytes[](2);
  entries[0] = abi.encode(1, 2, 3);
  entries[1] = abi.encode(4, 5, 9);
}

function testTableSums(uint256 a, uint256 b, uint256 expected) public {
  assertEq(a + b, expected);
}

The setUpTable functions are called once during setup and the table is stored in-memory. The testTable function is then called in parallel with each entry in the table. If an entry fails we should mark the table test as a failure and provide some info on what entries failed.

The setUpTable functions returns an array of bytes: each entry in this array is calldata we pass directly to the table test function by prepending the signature of the table test function.

Pros:

• Solidity-only

Cons:

• Uses ABI encode

External files

Same as above, but the table is in a file named after some convention. Each entry is ABI encoded and then passed to the table test function

Pros:

• No ABI encode

Cons:

• Uses external files

Additional context

No response

[gakonst]

Any reason why we cannot avoid the abi encode like this?

struct Case {
   uint256 a;
   uint256 b;
   uint256 expected;
}

function setUpTableSums() public returns (Case[] memory) {
  Case[] memory entries = new Case[](2);
  entries[0] = Case(1, 2, 3);
  entries[1] = Case(4, 5, 9);
}

function testTableSums(Case memory case) public {
  assertEq(case.a + case.b, case.expected);
}

[onbjerg]

Hmm, thinking about it, you're right. Since we have the ABI we would know how to serialize/deserialize the return data

[d-xo]

Maybe you could use some magic modifiers that you pull out of the solidity AST at test time? The following could be interpreted as "fuzz testTableSums, always using the provided concrete inputs as a part of the fuzzing campaign".

modifier inputSums(uint a, uint b, uint expected) { _; }

function testTableSums(uint a, uint b, uint expected) 
  public
  inputSums(1, 2, 3)
  inputSums(4, 5, 9)
{
  assertEq(a + b, expected);
}

[d-xo]

I guess it should also be possible to parse some custom natspec comments with a list of inputs?

[emilianobonassi]

+1 to this feature

as a workaround i am actually using a modifier iterating over cases

struct Case {
  uint256 a;
  uint256 b;
  uint256 expected;
}

Case[] entries;

Case c;

modifier parametrizedTest() {
  for (uint256 i = 0; i < entries.length; i++) {
    c = entries[i];
    _;
  }
}

function setUp() public override {
  entries.push(Case(1, 2, 3));
  entries.push(Case(4, 5, 9));
}

function testTableSums() public parametrizedTest {
  assertEq(c.a + c.b, c.expected);
}

[sambacha]

I guess it should also be possible to parse some custom natspec comments with a list of inputs?

0.8.13 supports the natspec directive of custom now

// from https://github.com/contractshark/vscode-solidity-extenstion/pull/7/files#diff-9b75242a14d24953968395cc5a946df5d3bdda1043d761a91e8d3f10157aa016R117-R120

// https://docs.soliditylang.org/en/latest/natspec-format.html#tags
  "natspec-tag-custom": {
            "match": "(@custom:)\\b",
            "name": "storage.type.custom.natspec"
        },

something like @custom:requires $TEST_ACCOUNT etc etc see #979