Bonsai trie initial sync fails with `World State Root does not match expected value` (testing only)

[yorickdowne]

Description

With an initial mainnet fast sync using bonsai tries, the sync fails with this error message:

eth1_1  | 2021-05-05 01:40:17.950+00:00 | EthScheduler-Services-4 (requestCompleteTask) | INFO  | CompleteTaskStep | Downloaded 883300000 world state nodes. At least 82 nodes remaining.
eth1_1  | 2021-05-05 01:40:47.085+00:00 | EthScheduler-Services-4 (requestCompleteTask) | INFO  | DefaultSynchronizer | Fast sync completed successfully with pivot block 12358887
eth1_1  | 2021-05-05 01:40:47.091+00:00 | EthScheduler-Services-4 (requestCompleteTask) | INFO  | FullSyncDownloader | Starting full sync.
eth1_1  | 2021-05-05 01:40:47.099+00:00 | EthScheduler-Services-4 (requestCompleteTask) | INFO  | WorldDownloadState | Finished downloading world state from peers
eth1_1  | 2021-05-05 01:40:47.570+00:00 | nioEventLoopGroup-3-1 | INFO  | SyncTargetManager | Found common ancestor with peer Peer 0x017cb4270271491285... at block 12358887
eth1_1  | 2021-05-05 01:41:05.053+00:00 | EthScheduler-Services-34 (importBlock) | ERROR | PipelineChainDownloader | Chain download failed. Restarting after short delay.
eth1_1  | java.util.concurrent.CompletionException: java.lang.RuntimeException: World State Root does not match expected value, header 0xec071e09803825ae0438af2fd562ef168f7a301df7e4b4867f1108c125a2742d calculated 0xc4374cf503d80e0ec722776b55e7f505c290bd55ef2cbbcf78fc1847a0b691b4
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367)
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture.completeRelay(CompletableFuture.java:376)
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture$UniRelay.tryFire(CompletableFuture.java:1093)
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2152)
eth1_1  |       at org.hyperledger.besu.services.pipeline.Pipeline.abort(Pipeline.java:179)
eth1_1  |       at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:158)
eth1_1  |       at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
eth1_1  |       at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
eth1_1  |       at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
eth1_1  |       at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
eth1_1  |       at java.base/java.lang.Thread.run(Thread.java:832)
eth1_1  | Caused by: java.lang.RuntimeException: World State Root does not match expected value, header 0xec071e09803825ae0438af2fd562ef168f7a301df7e4b4867f1108c125a2742d calculated 0xc4374cf503d80e0ec722776b55e7f505c290bd55ef2cbbcf78fc1847a0b691b4

I understand Bonsai tries are not production ready, this sync was done for testing purposes only.

Versions (Add all that apply)

• Software version: besu/v21.1.6-dev-425db2a9/linux-x86_64/oracle_openjdk-java-15
• Java version: 15
• OS Name & Version: Ubuntu 20.04

[matkt]

Thank you for your ticket . I have some questions to help us understand the bug.

• Do you have more stracktrace below the one you put (another stacktrace etc)?
• Have you mixed different versions of Besu during sync or have you always used this one besu/v21.1.6-dev-425db2a9/linux-x86_64/oracle_openjdk-java-15 ?
• If you restart Besu does the synchronization resume without issues or do you still have the stacktrace ?

I also did some modification on this PR #2261 that may fix the issue. Could you tell me if it's better with this one ?

Thanks for your help

[yorickdowne]

• Yes, this stacktrace basically repeats ad infinitum
• I only ever synced with that version, this was a test
• If I restart Besu I get the same stacktrace

I'll try the PR you linked and see how it behaves for me!
Edit: Compiled without issue and now syncing from scratch; give it a couple days now.

[matkt]

Compiled without issue and now syncing from scratch; give it a couple days now.

Ok thanks I also tried to start new fastsync node to test if I can reproduce this issue locally

If you still have the issue it will help me if you can send me some traces

• Get he hash of the badblock : https://besu.hyperledger.org/en/stable/Reference/API-Methods/#debug_getbadblocks
• Get traces of this block thanks to the hash https://besu.hyperledger.org/en/stable/Reference/API-Methods/#debug_standardtraceblocktofile

[yorickdowne]

Shall do. One marked difference between this sync attempt and the last one: During the last one I had a restart of Besu halfway through, which started world state over. So last attempt, first fast sync completed, then world state did; this sync, world state is done before fast sync has completed.

If this sync is successful, I may start over again and try to get the behavior during the first run, that is, stop and start Besu around 400 mil state.

[matkt]

indeed may be that stopping the fastsync has made the database inconsistent even if I do not yet understand how. I will also try on my side on a smaller network (goerli) to stop and start to see if there is an impact

[yorickdowne]

Success. Note the java exception, might be worth doing something about. I'll start over now and stop it partway through, then restart, to see whether that causes issues.

eth1_1  | 2021-05-20 00:16:25.827+00:00 | EthScheduler-Services-33 (importBlock) | INFO  | FastImportBlocksStep | Completed importing chain segment 12450752 to 12450951, Peers: 26
eth1_1  | 2021-05-20 00:16:39.702+00:00 | vertx-blocked-thread-checker | WARN  | BlockedThreadChecker | Thread Thread[vert.x-eventloop-thread-2,5,main]=Thread[vert.x-eventloop-thread-2,5,main] has been blocked for 6008 ms, time limit is 2000 ms
eth1_1  | io.vertx.core.VertxException: Thread blocked
eth1_1  |       at java.base@15.0.2/sun.nio.ch.DatagramChannelImpl.send0(Native Method)
eth1_1  |       at java.base@15.0.2/sun.nio.ch.DatagramChannelImpl.sendFromNativeBuffer(DatagramChannelImpl.java:897)
eth1_1  |       at java.base@15.0.2/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:859)
eth1_1  |       at java.base@15.0.2/sun.nio.ch.DatagramChannelImpl.send(DatagramChannelImpl.java:817)
eth1_1  |       at app//io.netty.channel.socket.nio.NioDatagramChannel.doWriteMessage(NioDatagramChannel.java:295)
eth1_1  |       at app//io.netty.channel.nio.AbstractNioMessageChannel.doWrite(AbstractNioMessageChannel.java:142)
eth1_1  |       at app//io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:941)
eth1_1  |       at app//io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.flush0(AbstractNioChannel.java:354)
eth1_1  |       at app//io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:905)
eth1_1  |       at app//io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
eth1_1  |       at app//io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
eth1_1  |       at app//io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:765)
eth1_1  |       at app//io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:790)
eth1_1  |       at app//io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:758)
eth1_1  |       at app//io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:808)
eth1_1  |       at app//io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1025)
eth1_1  |       at app//io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:294)
eth1_1  |       at app//io.vertx.core.datagram.impl.DatagramSocketImpl.doSend(DatagramSocketImpl.java:280)
eth1_1  |       at app//io.vertx.core.datagram.impl.DatagramSocketImpl.lambda$send$2(DatagramSocketImpl.java:268)
eth1_1  |       at app//io.vertx.core.datagram.impl.DatagramSocketImpl$$Lambda$810/0x0000000800fffa98.handle(Unknown Source)
eth1_1  |       at app//io.vertx.core.impl.AddressResolver.lambda$null$0(AddressResolver.java:88)
eth1_1  |       at app//io.vertx.core.impl.AddressResolver$$Lambda$717/0x0000000800fc5e08.handle(Unknown Source)
eth1_1  |       at app//io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)
eth1_1  |       at app//io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)
eth1_1  |       at app//io.vertx.core.impl.EventLoopContext$$Lambda$590/0x0000000800f2aab8.run(Unknown Source)
eth1_1  |       at app//io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
eth1_1  |       at app//io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
eth1_1  |       at app//io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
eth1_1  |       at app//io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
eth1_1  |       at app//io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
eth1_1  |       at app//io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
eth1_1  |       at java.base@15.0.2/java.lang.Thread.run(Thread.java:832)
eth1_1  | 2021-05-20 00:16:43.515+00:00 | EthScheduler-Services-33 (importBlock) | INFO  | FastImportBlocksStep | Completed importing chain segment 12450952 to 12451151, Peers: 26
eth1_1  | 2021-05-20 00:16:46.045+00:00 | EthScheduler-Services-33 (importBlock) | INFO  | FastImportBlocksStep | Completed importing chain segment 12451152 to 12451351, Peers: 26
eth1_1  | 2021-05-20 00:16:49.324+00:00 | EthScheduler-Services-33 (importBlock) | INFO  | FastImportBlocksStep | Completed importing chain segment 12451352 to 12451550, Peers: 26
eth1_1  | 2021-05-20 00:16:49.324+00:00 | EthScheduler-Services-33 (importBlock) | INFO  | PipelineChainDownloader | Chain download complete
eth1_1  | 2021-05-20 00:16:49.807+00:00 | EthScheduler-Services-33 (importBlock) | INFO  | DefaultSynchronizer | Fast sync completed successfully with pivot block 12451550
eth1_1  | 2021-05-20 00:16:49.825+00:00 | EthScheduler-Services-33 (importBlock) | INFO  | FullSyncDownloader | Starting full sync.
eth1_1  | 2021-05-20 00:16:50.661+00:00 | nioEventLoopGroup-3-4 | INFO  | SyncTargetManager | Found common ancestor with peer Peer 0x3402e9107f1112f7e6... at block 12451550
eth1_1  | 2021-05-20 00:23:12.642+00:00 | EthScheduler-Services-37 (importBlock) | INFO  | FullImportBlockStep | Import reached block 12451600 (0xa8a70bf649e0f46886e992535d28feef13d6e8eb5061f38d5849ba0b0f299511), - Mg/s, Peers: 26
eth1_1  | 2021-05-20 00:28:28.682+00:00 | EthScheduler-Services-37 (importBlock) | INFO  | FullImportBlockStep | Import reached block 12451800 (0x5b0cfc833663abb1f488115097780d7e5fcd32f6bf1e4202b5d1ad0892962e4f), 9.322 Mg/s, Peers: 26
eth1_1  | 2021-05-20 00:33:00.886+00:00 | EthScheduler-Services-37 (importBlock) | INFO  | FullImportBlockStep | Import reached block 12452000 (0x0103c2070ec9a958ede4315874137e3ef1f34982af30ae02b5aae63ce10977b9), 10.929 Mg/s, Peers: 26

[matkt]

I think have something that looks similar with a ropsten node that had to restart the fastsync several times. I will investigate and try to find the reason

EDIT : I investigated and the reason of my issue is I have a bad pivot block with this node. the stateroot of my pivot block not match the stateroot, parenthash, .. on etherscan

I wonder if this is related to the massive reorg that there was on ropsten. do you have your old node to check the stateroot of the pivot block ?

"number" : "0x9cbdbb",
"hash" : "0xeb8fdc97536d7a2b5cc9d75051b02c3823ba5a9660131b9054370ad14f533220",
"mixHash" : "0x3a6aea9cfd1e964c94c7c08eef50d69171684a7da963e0e540f0bb3c38b0c669",
"parentHash" : "0x24a5d260c6a8ae63116772c4280d68240fe8692d13ffe9cea9191ea14a95340a",
"nonce" : "0x511545e9eb17749e",
"sha3Uncles" : "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
"stateRoot" : "0x0aeb913ddf6d189e1d4e87ef28c25c0ad22a042de7aed2d9b8c466f07a892478",
"receiptsRoot" : "0xd194b6b2db7f90717142911d3bd3d89510b7e1c193474af11d67a245d6f74401",
"miner" : "0x977546fc3c7d8295282f1eaa943441eb498022c7",
"difficulty" : "0x4516301",
"totalDifficulty" : "0x778a5af2b80902",
"extraData" : "0x307834383639373636353666366532303530366636663663",
"size" : "0x1008",
"gasLimit" : "0x7a3083",
"gasUsed" : "0xfa112",
"timestamp" : "0x60a6b0b3",

[yorickdowne]

I do not have my old node, alas.

[yorickdowne]

I have this reproduced! The only difference to the successful sync was that Besu was restarted once, while World State was not completely "in" yet.

eth1_1  | 2021-05-23 14:20:56.272+00:00 | EthScheduler-Services-5 (requestCompleteTask) | INFO  | CompleteTaskStep | Downloaded 846800000 world state nodes. At least 22341 nodes remaining.
eth1_1  | 2021-05-23 14:21:13.958+00:00 | EthScheduler-Services-5 (requestCompleteTask) | INFO  | CompleteTaskStep | Downloaded 846900000 world state nodes. At least 1 nodes remaining.
eth1_1  | 2021-05-23 14:21:24.376+00:00 | EthScheduler-Services-5 (requestCompleteTask) | INFO  | DefaultSynchronizer | Fast sync completed successfully with pivot block 12475948
eth1_1  | 2021-05-23 14:21:24.381+00:00 | EthScheduler-Services-5 (requestCompleteTask) | INFO  | FullSyncDownloader | Starting full sync.
eth1_1  | 2021-05-23 14:21:24.393+00:00 | EthScheduler-Services-5 (requestCompleteTask) | INFO  | WorldDownloadState | Finished downloading world state from peers
eth1_1  | 2021-05-23 14:21:25.855+00:00 | nioEventLoopGroup-3-5 | INFO  | SyncTargetManager | Found common ancestor with peer Peer 0x03d6d0b0d64530db20... at block 12475948
eth1_1  | 2021-05-23 14:21:37.379+00:00 | EthScheduler-Services-34 (importBlock) | ERROR | PipelineChainDownloader | Chain download failed. Restarting after short delay.
eth1_1  | java.util.concurrent.CompletionException: java.lang.RuntimeException: World State Root does not match expected value, header 0xa2792be39009e2b019cd0ed857b7dab92f1607917a37567ce6a576d8ec9d602a calculated 0x8cdcfca1f16f793daf37a921ba7760ef3c3224f66030463b40e7f4fbe8085e3b
eth1_1  |       at java.base/java.util.concurrent.CompletableFuture.encodeRelay(CompletableFuture.java:367)

[yorickdowne]

I had to restart Besu once more to enable the debug API, and now it's showing these kind of messages:

eth1_1  | 2021-05-24 16:52:45.490+00:00 | nioEventLoopGroup-3-4 | INFO  | SyncTargetManager | Found common ancestor with peer Peer 0x0c312d59b705ea66f4... at block 12475948
eth1_1  | 2021-05-24 16:52:45.493+00:00 | nioEventLoopGroup-3-4 | WARN  | FullSyncTargetManager | Disconnecting Peer 0x0c312d59b705ea66f4... because world state is not available at common ancestor at block 12475948 (0xb7166f29c1b0db4b77a3ef32fe3886bb1724df2f8736ad03a2f78deab8f06638)

No bad blocks:

curl -X POST --data '{"jsonrpc":"2.0","method":"debug_getBadBlocks","params":[],"id":1}' http://localhost:8547
{
  "jsonrpc" : "2.0",
  "id" : 1,
  "result" : [ ]
}

which of course might be an artefact of the restart.

Can you point me towards how to check the stateroot of the pivot block?

[matkt]

I had to restart Besu once more to enable the debug API, and now it's showing these kind of messages:

eth1_1  | 2021-05-24 16:52:45.490+00:00 | nioEventLoopGroup-3-4 | INFO  | SyncTargetManager | Found common ancestor with peer Peer 0x0c312d59b705ea66f4... at block 12475948
eth1_1  | 2021-05-24 16:52:45.493+00:00 | nioEventLoopGroup-3-4 | WARN  | FullSyncTargetManager | Disconnecting Peer 0x0c312d59b705ea66f4... because world state is not available at common ancestor at block 12475948 (0xb7166f29c1b0db4b77a3ef32fe3886bb1724df2f8736ad03a2f78deab8f06638)

No bad blocks:

curl -X POST --data '{"jsonrpc":"2.0","method":"debug_getBadBlocks","params":[],"id":1}' http://localhost:8547
{
  "jsonrpc" : "2.0",
  "id" : 1,
  "result" : [ ]
}

which of course might be an artefact of the restart.

Can you point me towards how to check the stateroot of the pivot block?

Yes good news. you need to call eth_getBlockByNumber with the pivot block number. And after you can compare with the same block present on etherscan

curl --location --request POST 'http://127.0.0.1:8545' --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["10272187", true], "id":1}'

[yorickdowne]

This is the output for the pivot block:

curl --location --request POST 'http://localhost:8547' --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["12475948", true], "id":1}'

  "jsonrpc" : "2.0",
  "id" : 1,
  "result" : {
    "number" : "0xbe5e2c",
    "hash" : "0xb7166f29c1b0db4b77a3ef32fe3886bb1724df2f8736ad03a2f78deab8f06638",
    "mixHash" : "0xf12dd642d093f7f46181415d9f40482ce2b6a185176a4e0eb8b6c9fbcceabfeb",
    "parentHash" : "0xcecf4bc8685c3566c0430012ddead18f91014e59e53a6aed4449d7c6de1b2a1a",
    "nonce" : "0xac8c5f6caece7bee",
    "sha3Uncles" : "0xa757be5f27a4611f860bc51e7b414e6e3f706aff0fc3836e3e875cd6a8e1838e",
    "logsBloom" : "0x0220809205020810808a54b0800050809620000010100747a0232430068002e1090c244204800080080248123001210006000000490d20214104280a02e4602430c81c81245000694c34800801302821480210400940160097800e00a90040121084048402020802010610108040484140098a000684066007241192114f1401109821088230a214154008489100080485b00b1101018238684000420818291802120188190021400a8c548880040000028042010022020c00664440010802100610201620800048a4a00480208004bc08408b42844414105a05208210922215083025108820410b00c012850212000040208000014080c41000108810021220",
    "transactionsRoot" : "0x9f1ae146f568f8c1dbc92f1b6bb01982118a90d4f4b2a1db445f7d5687a677ed",
    "stateRoot" : "0x47e9468c63a94d109a7d5fd74e01139c18bcc3baab876346a6e04d6975140332",
    "receiptsRoot" : "0x594bdc3db4d5cab2fa7decfb5ad5d9203e46df71a608f0a6de4cb041ea96f60f",
    "miner" : "0xf20b338752976878754518183873602902360704",
    "difficulty" : "0x1cb711b6732d4f",
    "totalDifficulty" : "0x54c8a5faf92b8c6fecf",
    "extraData" : "0xe4b883e5bda9e7a59ee4bb99e9b1bc0b0321",
    "size" : "0x1114f",
    "gasLimit" : "0xe4e1c0",
    "gasUsed" : "0xe4a4db",
    "timestamp" : "0x60a74b18",
    "uncles" : [ "0x7cf9200a873a7a30198eb69de284a7a3295ce4d09ed6ad6315be139a6c0b778b" ],
    "transactions" : [ {

The state root matches etherscan. Before restart it showed World State Root does not match expected value, header 0xa2792be39009e2b019cd0ed857b7dab92f1607917a37567ce6a576d8ec9d602a calculated 0x8cdcfca1f16f793daf37a921ba7760ef3c3224f66030463b40e7f4fbe8085e3b

I don't see those hex values anywhere in that block, in its parent, or the uncle.

Where to from here?

[yorickdowne]

This is the parent, and stateRoot also matches etherscan

curl --location --request POST 'http://localhost:8547' --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["12475947", true], "id":1}'

  "jsonrpc" : "2.0",
  "id" : 1,
  "result" : {
    "number" : "0xbe5e2b",
    "hash" : "0xcecf4bc8685c3566c0430012ddead18f91014e59e53a6aed4449d7c6de1b2a1a",
    "mixHash" : "0xecfda76cfa0b3cefef07e121e90c37f76ec11d229bb8801b0eb04c3be725bd52",
    "parentHash" : "0xc8eaabdbbdab72fa7ad03b78a2c5088cedb436dc5657292d01be924c6939c97c",
    "nonce" : "0xe82e91c53e09bc8b",
    "sha3Uncles" : "0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347",
    "logsBloom" : "0x00e30102440800082143944797a75630e1815e4a2855fc128511a059b00401ad3008220d9089b01c990da000b36681081b4544885b809a05065d1e96523461019d4528401ae1c081cc71061d6298d826401363039140806e75c39c8d280002a8107201484e00c403e09008081c193850361314f3318e26c05a25841410a01c26176824361366121128768004013006a81802a8a39f50215e08521748731100200f0130699883e301850560828207ed4cc8d09a200064094c00af8230c48abb70c3126182a0ce0e898022c02c8056010442060eaa8420a21f68487512c2003a341218a9a829a402111c0980038894610580704840489231048e0185424c14d8d1",
    "transactionsRoot" : "0x22dc6a5395817daf6c2874ecc00a1be34fc5a4de339b198dbebd501c43ed48ed",
    "stateRoot" : "0x2d0f20a7a65c053a270ada4451d3a675bb089e321ef303725c7d211f0640f8e7",
    "receiptsRoot" : "0x73ec93adef646ac41f5e266017def894411328c3f35f1b54dda00720342eaa07",
    "miner" : "0xb3b7874f13387d44a3398d298b075b7a3505d8d4",
    "difficulty" : "0x1cb37a472a4806",
    "totalDifficulty" : "0x54c8a42f8810253d180",
    "extraData" : "0x426162656c20687a38",
    "size" : "0xa101",
    "gasLimit" : "0xe4e1a0",
    "gasUsed" : "0xe4b258",
    "timestamp" : "0x60a74b14",
    "uncles" : [ ],
    "transactions" : [ {

[matkt]

This is the output for the pivot block:

curl --location --request POST 'http://localhost:8547' --header 'Content-Type: application/json' --data-raw '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["12475948", true], "id":1}'

  "jsonrpc" : "2.0",
  "id" : 1,
  "result" : {
    "number" : "0xbe5e2c",
    "hash" : "0xb7166f29c1b0db4b77a3ef32fe3886bb1724df2f8736ad03a2f78deab8f06638",
    "mixHash" : "0xf12dd642d093f7f46181415d9f40482ce2b6a185176a4e0eb8b6c9fbcceabfeb",
    "parentHash" : "0xcecf4bc8685c3566c0430012ddead18f91014e59e53a6aed4449d7c6de1b2a1a",
    "nonce" : "0xac8c5f6caece7bee",
    "sha3Uncles" : "0xa757be5f27a4611f860bc51e7b414e6e3f706aff0fc3836e3e875cd6a8e1838e",
    "logsBloom" : "0x0220809205020810808a54b0800050809620000010100747a0232430068002e1090c244204800080080248123001210006000000490d20214104280a02e4602430c81c81245000694c34800801302821480210400940160097800e00a90040121084048402020802010610108040484140098a000684066007241192114f1401109821088230a214154008489100080485b00b1101018238684000420818291802120188190021400a8c548880040000028042010022020c00664440010802100610201620800048a4a00480208004bc08408b42844414105a05208210922215083025108820410b00c012850212000040208000014080c41000108810021220",
    "transactionsRoot" : "0x9f1ae146f568f8c1dbc92f1b6bb01982118a90d4f4b2a1db445f7d5687a677ed",
    "stateRoot" : "0x47e9468c63a94d109a7d5fd74e01139c18bcc3baab876346a6e04d6975140332",
    "receiptsRoot" : "0x594bdc3db4d5cab2fa7decfb5ad5d9203e46df71a608f0a6de4cb041ea96f60f",
    "miner" : "0xf20b338752976878754518183873602902360704",
    "difficulty" : "0x1cb711b6732d4f",
    "totalDifficulty" : "0x54c8a5faf92b8c6fecf",
    "extraData" : "0xe4b883e5bda9e7a59ee4bb99e9b1bc0b0321",
    "size" : "0x1114f",
    "gasLimit" : "0xe4e1c0",
    "gasUsed" : "0xe4a4db",
    "timestamp" : "0x60a74b18",
    "uncles" : [ "0x7cf9200a873a7a30198eb69de284a7a3295ce4d09ed6ad6315be139a6c0b778b" ],
    "transactions" : [ {

The state root matches etherscan. Before restart it showed World State Root does not match expected value, header 0xa2792be39009e2b019cd0ed857b7dab92f1607917a37567ce6a576d8ec9d602a calculated 0x8cdcfca1f16f793daf37a921ba7760ef3c3224f66030463b40e7f4fbe8085e3b

I don't see those hex values anywhere in that block, in its parent, or the uncle.

Where to from here?

It is the new block that Besu try to import after the fastsync. So obviously if we stop the fastsync in the middle the data in database are corrupted and we are not able to validate new blocks. I'll try to investigate this. Thanks a lot for all this informations :)