Tests fail with --coverage while they are 100% pass without --coverage #1777

Doc-Pixel · 2023-12-17T06:16:29Z

Environment information

OS: ubuntu 22.*
Python Version: 3.8.10
ape and plugin versions: 0.6.20

Installed Plugins:
  infura       0.6.0
  alchemy      0.6.0
  template     0.6.1
  foundry      0.6.11
  solidity     0.6.6
  tokens       0.6.1
  vyper        0.6.11
  hardhat      0.6.13
  polygon      0.6.5
  etherscan    0.6.7
  ens          0.6.1

Contents of your ape-config.yaml (NOTE: do not post anything private like RPC urls or secrets!):

name: myContract

default_ecosystem: ethereum

# number_of_accounts: 5

plugins:
  - name: infura
  - name: alchemy
  - name: vyper
  - name: ens
  - name: etherscan 
  - name: hardhat

vyper:
  evm_version: istanbul  # bor doesn't support latest forks


geth:
  ethereum:
    mainnet:
      uri: http://myServer:8545
      chain_id: 109

hardhat:
  request_timeout: 30  # Defaults to 30
  fork_request_timeout: 600  # Defaults to 300
  fork:
    ethereum:
      mainnet:
        upstream_provider: geth

test:
  coverage:
    reports:
      terminal:
        verbose: true  # Show verbose coverage information in the terminal.

What went wrong?

When I run my tests without --coverage, I get 100% PASS as a result. When I add --coverage, I get errors.
It happens when I call my function that accepts an address as the only input parameter.

So in short:

def myFunction(_address: address):

And I test it with myContract.myFunction(address_to_check, sender=address_in_adminlist)

Ape tells me:

from field must match key's 0x1e59ce931B4CFea3fe4B875411e280e173cB7A9C, but it was 0xc89D42189f0450C2b2c3c61f58Ec5d628176A1E7

Long version below

def test_Profile(my_contract, owner, not_owner):
    my_contract.storeProfile('mango', owner , 1, 'hi Im nobody bla bla bla', sender=owner)
    assert my_contract.getProfile(owner, sender=owner) == ('mango', owner, 1,'hi Im nobody bla bla bla')
    assert my_contract.getProfile(owner, sender=not_owner) == ('mango', owner, 1,'hi Im nobody bla bla bla')

The contract function is literally this: Return a stored struct for the account that got passed.
There is no assertion that only the owner address can request their profile.

@view
@external
def getProfile(account: address) -> Profile:
    return self.profileStore[account]

And the error message is

/home/ethbox/Vyper-Main/myContract/tests/test_myContract.py:70: in test_Profile
    assert my_contract.getProfile(owner, sender=not_owner) == ('mango', owner, 1,'hi Im nobody bla bla bla')
E   TypeError: from field must match key's 0x1e59ce931B4CFea3fe4B875411e280e173cB7A9C, but it was 0xc89D42189f0450C2b2c3c61f58Ec5d628176A1E7
        myContract = <myContract 0x274b028b03A250cA03644E6c578D81f019eE1323>
        not_owner  = <TestAccount 0xc89D42189f0450C2b2c3c61f58Ec5d628176A1E7>
        owner      = <TestAccount 0x1e59ce931B4CFea3fe4B875411e280e173cB7A9C>

And the way profiles are stored:

struct Profile:
    # userID: uint256 
    userName: String[64]
    pfpContract: address
    pfpID: uint256
    bio: String[256]

profileStore: HashMap[address, Profile]

So for some reason it doesn't allow not_owner to call the getProfile function and pass not_owner as the sender and requesting the profile created by sender. Which doesn't make sense with the vyper function.
Also it should either succeed or fail in both scenarios, wether I test with or without --coverage.
So not consistent.

To make things more exciting I fork shibarium with ape-hardhat (even though it's called ethereum in the ape-config. I just flipped the IP)
ape test --network ethereum:mainnet-fork:hardhat --coverage

How can it be fixed?

Not entirely sure, seems like the address passed as a parameter (owner) is overwritten with the sender= address (not_owner).
This only gets triggered when I use --coverage.

The text was updated successfully, but these errors were encountered:

linear · 2023-12-17T06:16:32Z

APE-1620 Tests fail with --coverage while they are 100% pass without --coverage

antazoey · 2023-12-20T22:01:03Z

Can you try again on 0.7 if possible?

We have fixed some things, at least.. failing to get coverage on a tx should no longer cause the test to fail anyway.

Doc-Pixel · 2023-12-23T18:51:56Z

Can you try again on 0.7 if possible?

We have fixed some things, at least.. failing to get coverage on a tx should no longer cause the test to fail anyway.

I've tested with this setup:

Python 3.10.13
Ape 0.7.1
Vyper 0.3.10

I upgraded the entire toolchain, deleted old python versions / libs like this sudo rm -rf /usr/local/lib/python3.8 and every occurence everywhere because I had some weird interference going on with ape/vyper still trying to pull stuff from those locations.

Uninstalled then reinstalled all the ape plugins. Coverage is working and I no longer get the scenario where test fail with a non-owner/deployer account on a function that doesn't check for msg.sender at all.

:-)

Doc-Pixel · 2023-12-31T06:24:39Z

Me again, reopening again :-)

getting a did not revert using coverage while the test passes without coverage.

Tried with the 0.7.1 version and also did a pip3 install git+
Also set the version pragma to vyper 0.3.7 and back to 0.3.10 to see if that had any effect which it didn't.

Bit of a peculiar error messages, depending if we have set a revert message or not in the _checkAccess() function

    # with a revert message:
    with ape.reverts():
E   AssertionError: Transaction did not revert.
E   However, an exception of type <class 'OverflowError'> occurred: Python int too large to convert to C ssize_t.

    # without a revert message
    with ape.reverts():
E   AssertionError: Transaction did not revert.
E   However, an exception of type <class 'ape.exceptions.DecodingError'> occurred: Tried to read 32 bytes, only got 0 bytes..

It's checking

denyList: HashMap[address, bool]

@view
@internal
def _denied() -> bool:
    return self.denyList[msg.sender]

@view
@internal
def _checkAccess() -> bool:
    assert not self._denied(), 'optional revert message'
    # rest of checks

I call an external function extFunction that calls _checkAccess() to determine if someone is admin, on allow/deny lists etc depending on contract mode. and one of the checks is to call _denied() to check if someone is on the denylist before the body is executed

Running the tests without --coverage is 100% pass.

Doc-Pixel · 2023-12-31T08:05:45Z

with -pdb I get some more hints

exc_value = DecodingError('Tried to read 32 bytes, only got 0 bytes.'), traceback = <traceback object at 0x7feb1c98ae00>

    def __exit__(self, exc_type: Type, exc_value: Exception, traceback) -> bool:
        if exc_type is None:
            raise AssertionError("Transaction did not revert.")
    
        if not isinstance(exc_value, ContractLogicError):
>           raise AssertionError(
                f"Transaction did not revert.\n"
                f"However, an exception of type {type(exc_value)} occurred: {exc_value}."
            ) from exc_value
E           AssertionError: Transaction did not revert.
E           However, an exception of type <class 'ape.exceptions.DecodingError'> occurred: Tried to read 32 bytes, only got 0 bytes..

../../../.local/lib/python3.10/site-packages/ape/pytest/contextmanagers.py:162: AssertionError```

antazoey · 2024-01-02T15:42:55Z

If you run with -v debug, it will print out Ape's traceback which will really help me. I still haven't reproduced!

Doc-Pixel · 2024-01-03T02:07:16Z

I think this is the relevant section, let me know if you need more details :-)

with --coverage:

DEBUG: hardhat_metadata
DEBUG: Getting response HTTP. URI: http://127.0.0.1:8545/[hidden] Method: hardhat_metadata, Response: {'jsonrpc': '2.0', 'id': 4, 'result': {'clientVersion': 'HardhatNetwork/2.19.4/@nomicfoundation/ethereumjs-vm/7.0.2', 'chainId': 31337, 'instanceId': '0x54a18089bc9e3a55f490e4b67117ac892f2b4a1b51ac3d1b8ea4f52624cb9bc9', 'latestBlockNumber': 2446853, 'latestBlockHash': '0xf91ca2a5d082a6018eaeee793b51f2b72aa410612727f9a3d6ce332f9af289c0', 'forkedNetwork': {'chainId': 109, 'forkBlockNumber': 2446853, 'forkBlockHash': '0xf91ca2a5d082a6018eaeee793b51f2b72aa410612727f9a3d6ce332f9af289c0'}}}

tests/test_bricker.py .DEBUG: eth_getStorageAt (4)
F
/home/user/Projects/Vyper/shibber/tests/test_bricker.py:115: in test_all_modes
    with ape.reverts():
E   AssertionError: Transaction did not revert.
E   However, an exception of type <class 'ape.exceptions.DecodingError'> occurred: Tried to read 32 bytes, only got 0 bytes..
        allowed_user = <TestAccount 0x9965507D1a55bcC2695C58ba16FB37d819B0A4dc>
        bricker_contract = <bricker 0x5FC8d32690cc91D4c39d9d3abcBD16989F875707>
        denied_user = <TestAccount 0x15d34AAf54267DB7D7c367839AAf71A00a2C6A65>
        mode       = 1
        owner      = <TestAccount 0xf39Fd6e51aad88F6F4ce6aB8827279cffFb92266>

Starting interactive mode. Type `exit` to halt current test.

In [1]:

and without --coverage:

DEBUG: hardhat_metadata
DEBUG: Getting response HTTP. URI: http://127.0.0.1:8545/[hidden] Method: hardhat_metadata, Response: {'jsonrpc': '2.0', 'id': 4, 'result': {'clientVersion': 'HardhatNetwork/2.19.4/@nomicfoundation/ethereumjs-vm/7.0.2', 'chainId': 31337, 'instanceId': '0x3e8533557458be475c2f5f93b8e3a8ebf88c301301af3f627a53cc5e7f3cb671', 'latestBlockNumber': 2446967, 'latestBlockHash': '0xce630e41cd1a36bbd68629dbeff2eef47ad243eeaa61ee5c5ec56c9d7c23b472', 'forkedNetwork': {'chainId': 109, 'forkBlockNumber': 2446967, 'forkBlockHash': '0xce630e41cd1a36bbd68629dbeff2eef47ad243eeaa61ee5c5ec56c9d7c23b472'}}}

tests/test_bricker.py DEBUG: Contract call:       <UnrecognizedContract>
DEBUG: From:                0xf39fd6e51aad88f6f4ce6ab8827279cfffb92266
.DEBUG: eth_getStorageAt (4)
DEBUG: Contract call:       <UnrecognizedContract>
DEBUG: From:                0x9965507d1a55bcc2695c58ba16fb37d819b0a4dc
DEBUG: To:                  0x5fc8d32690cc91d4c39d9d3abcbd16989f875707
DEBUG: 
DEBUG: Error: Transaction reverted without a reason string
DEBUG: at <UnrecognizedContract>.<unknown> (0x5fc8d32690cc91d4c39d9d3abcbd16989f875707)
DEBUG: at processTicksAndRejections (node:internal/process/task_queues:95:5)
DEBUG: at HardhatNode.runCall (/home/user/Projects/Vyper/shibber/node_modules/hardhat/src/internal/hardhat-network/provider/node.ts:679:20)
DEBUG: at EthModule._callAction (/home/user/Projects/Vyper/shibber/node_modules/hardhat/src/internal/hardhat-network/provider/modules/eth.ts:360:9)
DEBUG: at HardhatNetworkProvider._sendWithLogging (/home/user/Projects/Vyper/shibber/node_modules/hardhat/src/internal/hardhat-network/provider/provider.ts:145:22)
DEBUG: at HardhatNetworkProvider.request (/home/user/Projects/Vyper/shibber/node_modules/hardhat/src/internal/hardhat-network/provider/provider.ts:122:18)
DEBUG: at JsonRpcHandler._handleRequest (/home/user/Projects/Vyper/shibber/node_modules/hardhat/src/internal/hardhat-network/jsonrpc/handler.ts:191:20)
DEBUG: at JsonRpcHandler._handleSingleRequest (/home/user/Projects/Vyper/shibber/node_modules/hardhat/src/internal/hardhat-network/jsonrpc/handler.ts:152:17)
DEBUG: 
.DEBUG: eth_getStorageAt (4)
.DEBUG: eth_getStorageAt (4)
DEBUG: Contract call:       <UnrecognizedContract>
DEBUG: From:                0x70997970c51812dc3a010c7d01b50e0d17dc79c8
DEBUG: To:                  0x5fc8d32690cc91d4c39d9d3abcbd16989f875707
DEBUG: 
....                                                                                                                                                                 [100%]

======================================================================================== 7 passed in 11.73s =========================================================================================
INFO: Stopping 'Hardhat node' process.
user@laptop:~/Projects/Vyper/shibber$

Interesting enough without coverage there's still (sort of) an error but it seems to pass the test.

antazoey · 2024-01-03T18:38:53Z

I am jus having trouble reproducing any of this :(

Are you able to possible create a reproduction repo I can work off of?

antazoey · 2024-01-03T18:56:47Z

here is my example project, piecing together bits here to try and reproduce: https://github.com/antazoey/ape-playground/pull/3/files

but the tests never fail, not even with --coverage, so im not sure

Doc-Pixel · 2024-01-03T20:24:08Z

The plot thickens:

I am running the tests against shibarium, which is a bor/heimdalld based chain. Pretty much polygon with a different base token.

I switched to my ethereum node and now it forks mainnet and tests with coverage without any issues.

Polygon's bor doesn't support the latest EVM, but (I think) up to the London fork and I know they're working to release an update to bor to support newer EVM versions. Not sure if that is mainnet already.

Currently I'm not in a position to test with polygon if the behaviour is the same. I'll try as soon as I get a chance.

antazoey · 2024-01-03T21:04:47Z

I just tried polygon mainnet fork and ran the tests but still didnt have an issue, but maybe i should use something already deployed to mainnet before forking?

I pushed my config changes to the repro branch on the PR I commented above.

Doc-Pixel added the category: bug Something isn't working label Dec 17, 2023

Doc-Pixel closed this as completed Dec 23, 2023

Doc-Pixel reopened this Dec 31, 2023

This was referenced Jan 2, 2024

fix: issue where NotImplmeneted error was not raised #1802

Merged

fix: decoding issues that affect tracing features such as coverage #1803

Merged

antazoey closed this as completed in #1803 Jan 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tests fail with --coverage while they are 100% pass without --coverage #1777

Tests fail with --coverage while they are 100% pass without --coverage #1777

Doc-Pixel commented Dec 17, 2023

linear bot commented Dec 17, 2023

antazoey commented Dec 20, 2023 •

edited

Loading

Doc-Pixel commented Dec 23, 2023

Doc-Pixel commented Dec 31, 2023

Doc-Pixel commented Dec 31, 2023 •

edited

Loading

antazoey commented Jan 2, 2024

Doc-Pixel commented Jan 3, 2024 •

edited

Loading

antazoey commented Jan 3, 2024

antazoey commented Jan 3, 2024 •

edited

Loading

Doc-Pixel commented Jan 3, 2024

antazoey commented Jan 3, 2024

Tests fail with --coverage while they are 100% pass without --coverage #1777

Tests fail with --coverage while they are 100% pass without --coverage #1777

Comments

Doc-Pixel commented Dec 17, 2023

Environment information

What went wrong?

How can it be fixed?

linear bot commented Dec 17, 2023

antazoey commented Dec 20, 2023 • edited Loading

Doc-Pixel commented Dec 23, 2023

Doc-Pixel commented Dec 31, 2023

Doc-Pixel commented Dec 31, 2023 • edited Loading

antazoey commented Jan 2, 2024

Doc-Pixel commented Jan 3, 2024 • edited Loading

antazoey commented Jan 3, 2024

antazoey commented Jan 3, 2024 • edited Loading

Doc-Pixel commented Jan 3, 2024

antazoey commented Jan 3, 2024

antazoey commented Dec 20, 2023 •

edited

Loading

Doc-Pixel commented Dec 31, 2023 •

edited

Loading

Doc-Pixel commented Jan 3, 2024 •

edited

Loading

antazoey commented Jan 3, 2024 •

edited

Loading