Support tracing of synced, not-processed blocks

[gituser]

Describe the bug
I've synced Nethermind on the mainnet to replace OpenEthereum client, but trace_filter returns empty response for some reason:

curl --data '{"method":"trace_filter","params":[{"fromBlock":"0x1","toBlock":"0x50"}],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8545
{"jsonrpc":"2.0","result":[],"id":1}

On my old OpenEthereum node the result is correct though:

curl --data '{"method":"trace_filter","params":[{"fromBlock":"0x1","toBlock":"0x50"}],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST 127.0.0.1:9110
{"jsonrpc":"2.0","result":[{"action":{"author":"0x05a56e2d52c817161883f50c441c3228cfe54d9f","rewardType":"block","value":"0x4563918244f40000"},"blockHash":"0x88e96d4537bea4d9c05d12549907b32561d3bf31f45aae734cdc119f13406cb6","blockNumber":1,"result":null,"subtraces":0,"traceAddress":...":[],"transactionHash":null,"transactionPosition":null,"type":"reward"}],"id":1}

I've tried enumerating through latest blocks and found out that the only available blocks for trace_filter are last 2910 +/- 10 blocks from the latest synced block.

E.g. If I try to trace_filter block beyound that range latest_block - 3000 nethermind gives me an error:

{
   "error" : {
      "code" : -32603,
      "data" : "Nethermind.Trie.TrieException: Failed to load key 17e18854d6c1cee0ac2cee66833b0c4ff4c250224b30519d295e3e503d9f5bab from root hash 0x16492ad5cecdae541bccf891c2e8ce664bb4ccc5be2cd74e7ce806d68302942b.\n ---> Nethermind.Trie.TrieException: Node 0x16492ad5cecdae541bccf891c2e8ce664bb4ccc5be2cd74e7ce806d68302942b is missing from the DB\n   at Nethermind.Trie.Pruning.TrieStore.LoadRlp(Keccak keccak, IKeyValueStore keyValueStore)\n   at Nethermind.Trie.TrieNode.ResolveNode(ITrieNodeResolver tree)\n   at Nethermind.Trie.PatriciaTree.Run(Span`1 updatePath, Int32 nibblesCount, Byte[] updateValue, Boolean isUpdate, Boolean ignoreMissingDelete, Keccak startRootHash)\n   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)\n   --- End of inner exception stack trace ---\n   at Nethermind.Trie.PatriciaTree.Get(Span`1 rawKey, Keccak rootHash)\n   at Nethermind.State.StateProvider.GetState(Address address)\n   at Nethermind.State.StateProvider.GetAndAddToCache(Address address)\n   at Nethermind.State.StateProvider.GetNonce(Address address)\n   at Nethermind.Consensus.Processing.BlockProcessor.BlockValidationTransactionsExecutor.ProcessTransactions(Block block, ProcessingOptions processingOptions, BlockReceiptsTracer receiptsTracer, IReleaseSpec spec)\n   at Nethermind.Consensus.Processing.BlockProcessor.ProcessBlock(Block block, IBlockTracer blockTracer, ProcessingOptions options)\n   at Nethermind.Consensus.Processing.BlockProcessor.ProcessOne(Block suggestedBlock, ProcessingOptions options, IBlockTracer blockTracer)\n   at Nethermind.Consensus.Processing.BlockProcessor.Process(Keccak newBranchStateRoot, List`1 suggestedBlocks, ProcessingOptions options, IBlockTracer blockTracer)\n   at Nethermind.Consensus.Processing.BlockchainProcessor.ProcessBranch(ProcessingBranch processingBranch, Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer)\n   at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer)\n   at Nethermind.Consensus.Processing.OneTimeChainProcessor.Process(Block block, ProcessingOptions options, IBlockTracer tracer)\n   at Nethermind.Consensus.Tracing.Tracer.Trace(Block block, IBlockTracer blockTracer)\n   at Nethermind.JsonRpc.Modules.Trace.TraceRpcModule.TraceBlock(Block block, ParityTraceTypes traceTypes, TxTraceFilter txTraceFilter)\n   at Nethermind.JsonRpc.Modules.Trace.TraceRpcModule.trace_filter(TraceFilterForRpc traceFilterForRpc)",
      "message" : "Internal error"
   },
   "jsonrpc" : "2.0",
   "id" : 1
}

NOTE: old OpenEthereum node was running with fast sync (NOT archive!) and trace_filter worked just fine there and consumed ~1.1T, new Nethermind node consumes little bit less ~729G but tracing seems to be not working. I do not want to run archive node, simply because of the storage requirements.

To Reproduce
Steps to reproduce the behavior:

1. Fully sync mainnet node
2. Use this configuration:

{
  "Init": {
    "ChainSpecPath": "chainspec/foundation.json",
    "GenesisHash": "0xd4e56740f876aef8c010b86a40d5f56745a118d0906a34e69aec8c0db1cb8fa3",
    "BaseDbPath": "nethermind_db/mainnet",
    "LogFileName": "mainnet.log",
    "MemoryHint": 10480000000
  },
  "Network": {
    "ActivePeersMaxCount": 100
  },
  "Sync": {
    "FastSync": true,
    "SnapSync": true,
    "PivotNumber": 14957000,
    "PivotHash": "0x7ce2c9cb601721e33ee6313e91b30c9f035b74ee282402b5bdfb1095c572acd7",
    "PivotTotalDifficulty": "51613742220579521774353",
    "FastBlocks": true,
    "WitnessProtocolEnabled": true
  },
  "JsonRpc": {
    "Enabled": true,
    "Timeout": 20000,
    "Host": "0.0.0.0",
    "Port": 8545,
    "EnabledModules": [
      "Eth",
      "Subscribe",
      "Trace",
      "TxPool",
      "Web3",
      "Personal",
      "Proof",
      "Net",
      "Parity",
      "Health",
      "Rpc",
    ],
    "AdditionalRpcUrls": [
      "http://127.0.0.1:8551|http;ws|net;eth;subscribe;engine;web3;client;rpc|"
    ],
    "IpcUnixDomainSocketPath":"/home/nethermind/.nethermind/nethermind.ipc"
  },
  "Metrics": {
    "NodeName": "Mainnet"
  }
}

3. Try to query the node with the above curl

Expected behavior
Nethermind should return trace filter info same as OpenEthereum

Desktop (please complete the following information):

• OS: Ubuntu 18.04
• Version: 1.13.3-14-g2500d3daf (built from master branch)

[gituser]

Related issue: #4143

[LukaszRozmej]

This is fairly unlikely to be implemented, would require spinning up database for traces as well as developing a p2p syncing protocol for those traces, which only nethermind would support. I think that what we actually want to do is to have archive node footprint be minimal with state db redesign.

[gituser]

@LukaszRozmej is there any workaround in the configuration to let's say store last 30K states instead of running archive node, so last 30K blocks will be available for tracing?

trace_filter is essential feature in our blocks processing but we can't afford to run full archive node, on OpenEthereum it worked just fine with separate traces DB.

[LukaszRozmej]

@gituser we will consider adding this, but nothing we can do short-term

[gituser]

@LukaszRozmej this is quite crucial feature stopping us from switching completely from OpenEthereum to Nethermind.

I think many other users are having similar issue, cause there are many relying on trace_filter functionality on pruned node.

How hard it will be to implement this?

Thanks.

[gituser]

Any update on this or maybe there is some other mechanism to check for ERC-20 transfers or internal ETH transactions besides using tracing?

[LukaszRozmej]

@gituser created a plugin that stores the traces and retrives them through json rpc, the only thing is this needs FullSync (but not archive state).