`fail_on_revert` does not fail test suite when revert happens inside handler function

[Rubilmax]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (95a93cd 2023-08-25T00:25:25.985662923Z)

What command(s) is the bug in?

forge test

Operating System

None

Describe the bug

The following test passes:

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.0;

import "forge-std/Test.sol";

contract BrokenInvariantTest is Test {
    bytes4[] internal selectors;

    function setUp() public {
        selectors.push(this.doSomething.selector);

        targetSelector(FuzzSelector({addr: address(this), selectors: selectors}));
    }

    function doSomething() public {
        require(false);
    }

    function invariantBroken() public {}
}

using the command: forge test

Expected behavior would be: the test suite fails because there's a revert in a handler function

[0xfarhaan]

Same would have expected this to fail with fail_on_revert set true. Updated the example @Rubilmax posted to use the assertion helper which does post in the logs an assertion failure but the test suite passes.

Code:

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.0;

import "forge-std/Test.sol";

contract Handler is Test {

    function doSomething() public {
        assertTrue(false);
    }

}

contract BrokenInvariantTest is Test {
    
    bytes4[] internal selectors;

    Handler handler;

    function setUp() public {
        handler = new Handler();

        selectors.push(Handler.doSomething.selector);

        targetSelector(FuzzSelector({addr: address(handler), selectors: selectors}));
    }

    function statefulFuzz_BrokenInvariant() public {}

}

~/rep/personal/forge-playground main ?1 ❯ forge t --mt statefulFuzz_Broken                           
[⠒] Compiling...
No files changed, compilation skipped

Running 1 test for tests/unit/5.t.sol:BrokenInvariantTest
[PASS] statefulFuzz_BrokenInvariant() (runs: 1, calls: 1, reverts: 1)
Logs:
  Error: Assertion Failed
  
Test result: ok. 1 passed; 0 failed; 0 skipped; finished in 2.20ms
Ran 1 test suites: 1 tests passed, 0 failed, 0 skipped (1 total tests

We did ask for this behaviour in #4718 with the solution implemented in #5445.

[Rubilmax]

This regression may have been introduced in #5676

[lucas-manuel]

Just ran into this issue as well.

[Evalir]

hey @lucas-manuel @0xfarhaan could y'all confirm this is still happening? trying to repro on latest foundry and the repros do fail with fail_on_revert = true.

[Rubilmax]

hey @lucas-manuel @0xfarhaan could y'all confirm this is still happening? trying to repro on latest foundry and the repros do fail with fail_on_revert = true.

I can confirm that the issue is still happening on version: forge 0.2.0 (893fc9f 2023-09-12T00:22:45.184609469Z)

I just had to run the following test contract (as described in this issue's description):

// SPDX-License-Identifier: UNLICENSED
pragma solidity ^0.8.0;

import "forge-std/Test.sol";

contract BrokenInvariantTest is Test {
    bytes4[] internal selectors;

    function setUp() public {
        selectors.push(this.doSomething.selector);

        targetSelector(FuzzSelector({addr: address(this), selectors: selectors}));
    }

    function doSomething() public {
        require(false);
    }

    function invariantBroken() public {}
}

Output is:

[alex-alra-arteaga]

Same issue, any plan on the roadmap on fixing it?

[PatrickAlphaC]

I also just ran into this. The workaround for now is just to consider any reverts as a failed test, even though it says pass. For CI/CD you'd have to read the text of the number of reverts, and if it's not 0 consider it failed.

[wafflemakr]

I had this same issue with my tests using forge 0.2.0 (80df71f 2023-09-26T18:05:26.004416000Z) but tried it in a pc with a previous release forge 0.2.0 (11b08e9 2023-08-16) and now it shows the test failing.

[Rubilmax]

@Evalir @gakonst can this bugfix be prioritized pls?
The reproducible example is the simplest possible and it is confirmed by several independent developers. This is definitely not an expected behavior of foundry