Error syncing with mainnet on Besu 1.3.7 node - MerkleTrieException

[timbeiko]

My node is raising a MerkleTrieException and is unable to sync to mainnet with pruning enabled.

 

Hardware specs: Thinkpad T430 laptop, 8GB RAM 1TB SSD 4 cores i53320M 2.6GHz

CLI arguments: $ besu --data-path=/home/node/node --rpc-http-enabled --pruning-enabled

 

Error log:

2019-12-16 19:21:02.825+01:00 | nioEventLoopGroup-3-1 | INFO  | SyncTargetManager | Found common ancestor with peer 0x9208aa710c44ca7220... at block 9696372019-12-16 19:21:02.825+01:00 | nioEventLoopGroup-3-1 | INFO  | SyncTargetManager | Found common ancestor with peer 0x9208aa710c44ca7220... at block 9696372019-12-16 19:21:04.032+01:00 | EthScheduler-Services-0 (importBlock) | ERROR | PipelineChainDownloader | Chain download failed. Restarting after short delay.java.util.concurrent.CompletionException: org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0xdab2b5f035e693b4597a6c7d24abb3c0b675022a45a2262000fd29233016a0a4 at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?] at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376) ~[?:?] at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1019) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?] at org.hyperledger.besu.services.pipeline.Pipeline.abort(Pipeline.java:152) ~[besu-pipeline-1.3.7.jar:1.3.7] at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:134) ~[besu-pipeline-1.3.7.jar:1.3.7] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?]Caused by: org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0xdab2b5f035e693b4597a6c7d24abb3c0b675022a45a2262000fd29233016a0a4

Probability: Probable

Severity: Critical

---

Additional comments from the original Jira issue:

After deleting chain data and resyncing with pruning, I get the same error at another block. The following is an extract of the error:

2020-01-08 12:31:09.618+01:00 | nioEventLoopGroup-3-1 | INFO | SyncTargetManager | Found common ancestor with peer 0x58cf42ac9b3134a2ef... at block 2428314 2020-01-08 12:31:12.984+01:00 | EthScheduler-Services-48 (importBlock) | ERROR | PipelineChainDownloader | Chain download failed. Restarting after short delay. java.util.concurrent.CompletionException: org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0x498331f10b7c83bb32ef331f72296c4fed6d40cbcd1fba0ced59501a19b1bbfc at java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367) ~[?:?] at java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376) ~[?:?] at java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1019) ~[?:?] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[?:?] at org.hyperledger.besu.services.pipeline.Pipeline.abort(Pipeline.java:152) ~[besu-pipeline-1.3.8.jar:1.3.8] at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:134) ~[besu-pipeline-1.3.8.jar:1.3.8] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0x498331f10b7c83bb32ef331f72296c4fed6d40cbcd1fba0ced59501a19b1bbfc at org.hyperledger.besu.ethereum.trie.StoredNode.lambda$load$0(StoredNode.java:107) ~[besu-trie-1.3.8.jar:1.3.8] at java.util.Optional.orElseThrow(Optional.java:408) ~[?:?] at org.hyperledger.besu.ethereum.trie.StoredNode.load(StoredNode.java:106) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:49) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:62) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:50) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:62) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:50) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:62) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.StoredNode.accept(StoredNode.java:50) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:62) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:62) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.GetVisitor.visit(GetVisitor.java:46) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.BranchNode.accept(BranchNode.java:62) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.StoredMerklePatriciaTrie.get(StoredMerklePatriciaTrie.java:80) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.trie.StoredMerklePatriciaTrie.get(StoredMerklePatriciaTrie.java:35) ~[besu-trie-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.worldstate.DefaultMutableWorldState.get(DefaultMutableWorldState.java:111) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.core.AbstractWorldUpdater.get(AbstractWorldUpdater.java:80) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.core.AbstractWorldUpdater.get(AbstractWorldUpdater.java:80) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.core.AbstractWorldUpdater.get(AbstractWorldUpdater.java:80) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.core.AbstractWorldUpdater.get(AbstractWorldUpdater.java:80) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.vm.operations.SelfDestructOperation.cost(SelfDestructOperation.java:42) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.vm.ehalt.InsufficientGasExceptionalHaltPredicate.exceptionalHaltCondition(InsufficientGasExceptionalHaltPredicate.java:35) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.vm.ehalt.ExceptionalHaltManager.evaluateAll(ExceptionalHaltManager.java:38) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.vm.EVM.evaluateExceptionalHaltReasons(EVM.java:89) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.vm.EVM.executeNextOperation(EVM.java:73) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.vm.EVM.runToHalt(EVM.java:51) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.mainnet.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:156) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.mainnet.AbstractMessageProcessor.process(AbstractMessageProcessor.java:170) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:345) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:291) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:42) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.mainnet.TransactionProcessor.processTransaction(TransactionProcessor.java:133) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:132) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:36) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.mainnet.BlockProcessor.processBlock(BlockProcessor.java:59) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:88) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter.importBlock(MainnetBlockImporter.java:46) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.core.BlockImporter.importBlock(BlockImporter.java:44) ~[besu-core-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:45) ~[besu-eth-1.3.8.jar:1.3.8] at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:29) ~[besu-eth-1.3.8.jar:1.3.8] at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37) ~[besu-pipeline-1.3.8.jar:1.3.8] at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:130) ~[besu-pipeline-1.3.8.jar:1.3.8] ... 5 more 2020-01-08 12:31:15.083+01:00 | nioEventLoopGroup-3-1 | INFO | SyncTargetManager | Found common ancestor with peer 0x58cf42ac9b3134a2ef... at block 2428314 

Possible cause of error might be found here:

java.lang.Thread.run(Thread.java:834) [?:?] Caused by: org.hyperledger.besu.ethereum.trie.MerkleTrieException: Unable to load trie node value for hash 0x498331f10b7c83bb32ef331f72296c4fed6d40cbcd1fba0ced59501a19b1bbfc at org.hyperledger.besu.ethereum.trie.StoredNode.lambda$load$0(StoredNode.java:107) ~[besu-trie-1.3.8.jar:1.3.8] at java.util.Optional.orElseThrow(Optional.java:408) ~[?:?] at