TransactionExecutionError: Headers Timeout Error

[Pzixel]

Bug Description

When running long tests on harhat network all transactions start failing and then it stalls.

Reproducible By

This snippet should do the trick:

for (let j = 0; j < 1000000; j++) {
    const amountIn = '1000';
    const [signer1] = await ethers.getSigners();
    const swapRouter = ISwapRouter__factory.connect('0xE592427A0AEce92De3Edee1F18E0157C05861564', signer1);
    const recipient = signer1.address;
    await waitEthersTx(swapRouter.exactInput({
        // WETH -> USDC 3000 fee
        path: "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2000bb8A0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48",
        amountIn: amountIn,
        deadline: constants.MaxUint256,
        recipient,
        amountOutMinimum: 1,
    }, {
        value: amountIn,
    }));
}

Expected Behavior

It should be working without issues

Logs & Screenshots

Transaction fails with

TransactionExecutionError: Headers Timeout Error
    at HardhatNode.mineBlock (/Users/pzixel/Documents/Repos/test/node_modules/hardhat/src/internal/hardhat-network/provider/node.ts:478:15)
    at async EthModule._sendTransactionAndReturnHash (/Users/pzixel/Documents/Repos/test/node_modules/hardhat/src/internal/hardhat-network/provider/modules/eth.ts:1496:18)
    at async HardhatNetworkProvider.request (/Users/pzixel/Documents/Repos/test/node_modules/hardhat/src/internal/hardhat-network/provider/provider.ts:118:18)
    at async EthersProviderWrapper.send (/Users/pzixel/Documents/Repos/test/node_modules/@nomiclabs/hardhat-ethers/src/internal/ethers-provider-wrapper.ts:13:20)

    Caused by: HeadersTimeoutError: Headers Timeout Error
        at Timeout.onParserTimeout [as _onTimeout] (/Users/pzixel/Documents/Repos/test/node_modules/undici/lib/client.js:1005:28)
        at listOnTimeout (node:internal/timers:561:11)
        at processTimers (node:internal/timers:502:7)

Environment

Node 16.5.0, Macos 12.3.1 M1 Pro (but I'm pretty sure it's doesn't matter)

Hardhat config if relevant:

const networks: NetworksUserConfig = {
    hardhat: {
        forking: {
            url: myurl,
        },
        chainId: 1,
    },
};

[github-actions]

This issue is also being tracked on Linear.

We use Linear to manage our development process, but we keep the conversations on Github.

LINEAR-ID: d203952e-41c3-4764-a380-1abb7cff8d1e

[feuGeneA]

Thank you for using Hardhat. Can you provide a more complete reproduction scenario? Perhaps in a separate repo or in an attachment? Something I can easily just run. Thanks.

[NiKrause]

I also have this issue. I tried upgradeProxy with an impersonated instance of the proxy contract to be upgraded.
It's not really long running but quite some work to upgrade an upgradable contract.

[therawk]

I'm struggling with this, too. I call a function on a smart contract which takes a while to finish and get this error:

HeadersTimeoutError: Headers Timeout Error
    at Timeout.onParserTimeout [as _onTimeout] (/dev/collab/node_modules/undici/lib/client.js:1005:28)
    at listOnTimeout (node:internal/timers:559:11)
    at processTimers (node:internal/timers:500:7) {
  code: 'UND_ERR_HEADERS_TIMEOUT'
}

What exactly is causing this Timeout Error? I'm using mainnet fork via Alchemy. Is this timeout caused by Alchemy, Hardhat or the undici lib?

[Pzixel]

I've got it on two different endpoints (non-alchemy ones) so this must be a hardhat issue.

I might try creating a MRE but I will only be able to do so in 2 weeks or so. I hope someone comes with better example earlier.

[NiKrause]

I only got it on a hardhat local node when I was forking from a Polygon mainnet via a public node. There might be a timeout when getting the data from the live node. If I repeat the command a couple of times it disappears. Obviously when all data are downloaded the the local node eventually.

[feuGeneA]

I'm closing this because it sounds like Hardhat is behaving as expected.

This error message is simply the result of a network timeout, and the mere presence of this message does not indicate a problem with Hardhat, and GitHub Issues are intended to be used exclusively for problems in Hardhat itself.

There are many reasons why such a timeout may occur, including a problem on the endpoint (eg public node). It could also be the case that the computation you're requesting the node to execute may simply be a longer-running computation than the timeout values that have been configured at various points in the networking stack (both on the client and on the server).

If anyone suspects a problem that actually lies within Hardhat, please feel free to re-open this Issue, and, ideally, provide a minimal, reproducible example.

[3commascapital]

i am also seeing this when forking an ethereum node

[bretep]

I'm also seeing it forking an ethereum archive node of mainnet. :/

[bretep]

I've taken a pcap, 100% of the RPC calls were responded to. There was one call that looked like it took 1 second to respond.

Here is each request and the latency of the request.

RPC HTTP Status codes

[bretep]

I'm not seeing any timeouts on the wire or from the server, of which I'm the only client using.

[therawk]

Would you be interested in a repo I'm having this issue with?

[feuGeneA]

@bretep thank you for this. I've re-opened the issue. Clearly this isn't necessarily an issue connecting to the remote node. In particular, the presence of HardhatNode.mineBlock() in the call stack, which I hadn't noticed before your confirmation here, points a finger at Hardhat.

Are you able to configure Wireshark to also capture loopback traffic? (I know that is easier on some systems than others.) That would capture this activity, if I'm understanding the scenario correctly.

Also, can you confirm that you have the same symptoms as in the original Issue description? In particular, I'm interested in hearing whether HardhatNode.mineBlock() appears in your call stack.

Finally, would you happen to have a reproducible scenario for this? Even with all these wonderful clues, without a reproduction it will be difficult to even diagnose this, much less get it fixed.

Thanks again!