State rolling failed on BonsaiWorldStateKeyValueStorage because nonces differ

[iampkuhz]

Description

As an [Actor], I want [feature] so that [why].

Acceptance Criteria

• [Criteria 1]

Steps to Reproduce (Bug)

1. start the first 3 nodes.
2. start the 4th nodes and it begin syncing

Expected behavior: [What you expect to happen]

Actual behavior: [What actually happens]

Frequency: [What percentage of the time does it occur?]
I've tried restart the node but it still stucked.

Logs (if a bug)

2023-10-15 18:00:39.863+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | FullImportBlockStep | Import reached block 278438 (0x24410b77242b34625a176865f9e28e5a7a5c205156a279b0aa7275d0432a519c), - Mg/s, Peers: 1
2023-10-15 18:00:39.863+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | FullImportBlockStep | Import reached block 278439 (0xaa8561a36acd1565ed4b96a855c0105d09c4e332ad6bdd6e1b9f052fd780591e), - Mg/s, Peers: 1
2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | DEBUG | BonsaiWorldStateProvider | Attempting Rollback of 0x533a4388377b211e995ad2f4a317cf1f446240496ceadc3446756fc2e0c25cdb
2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | DEBUG | BonsaiWorldStateProvider | State rolling failed on BonsaiWorldStateKeyValueStorage for block hash 0xaa8561a36acd1565ed4b96a855c0105d09c4e332ad6bdd6e1b9f052fd780591e java.lang.IllegalStateException: Address=0x1a3a39ccbe340eaaa3cddece27fd4dc46f6cc8f7 Prior Value in Rolling Change: nonces differ
        at org.hyperledger.besu.ethereum.bonsai.BonsaiAccount.assertCloseEnoughForDiffing(BonsaiAccount.java:313)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.rollAccountChange(BonsaiWorldStateUpdateAccumulator.java:577)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.lambda$rollBack$15(BonsaiWorldStateUpdateAccumulator.java:526)
        at java.base/java.util.HashMap$EntrySpliterator.forEachRemaining(HashMap.java:1850)
        at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:762)
        at org.hyperledger.besu.ethereum.bonsai.worldview.BonsaiWorldStateUpdateAccumulator.rollBack(BonsaiWorldStateUpdateAccumulator.java:524)
        at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateProvider.rollMutableStateToBlockHash(BonsaiWorldStateProvider.java:242)
        at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateProvider.getMutable(BonsaiWorldStateProvider.java:184)
        at org.hyperledger.besu.ethereum.bonsai.BonsaiWorldStateProvider.getMutable(BonsaiWorldStateProvider.java:160)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:124)
        at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:72)
        at org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter.importBlock(MainnetBlockImporter.java:45)
        at org.hyperledger.besu.ethereum.core.BlockImporter.importBlock(BlockImporter.java:45)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:63)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:32)
        at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37)
        at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:169)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | MainnetBlockValidator | Invalid block 278440 (0xdb4bc2f7edd10191f7efd102cd783ac9cc0c15f60bf28691115d50cb16ec0fcf): Optional[Unable to process block because parent world state 0x0bbc969cf30f94ebf0ff9a38fc0d8858a9553714308c79ed4278ca2fc00ede8b is not available]
2023-10-15 18:00:39.864+08:00 | EthScheduler-Services-2 (importBlock) | INFO  | Pipeline | Unexpected exception in pipeline. Aborting. org.hyperledger.besu.ethereum.eth.sync.tasks.exceptions.InvalidBlockException: Failed to import block: Invalid block at #278440 (0xdb4bc2f7edd10191f7efd102cd783ac9cc0c15f60bf28691115d50cb16ec0fcf)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:65)
        at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:32)
        at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37)
        at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:169)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

Versions (Add all that apply)

• Software version: 23.7.3
• Java version:openjdk version "17.0.8" 2023-07-18 LTS
• OS Name & Version: CentOS7.2
• Kernel Version: Linux 4.9.151-015.ali3000.alios7.x86_64

[non-fungible-nelson]

Can you provide us with your configuration and genesis files?

Any other details that may be relevant.

[iampkuhz]

1. here is my config file.

BTW, if I change data-storage-format from BONSAI to FOREST and clean data to re-sync, the sync works quite normal

data-path="/home/admin/node4/data" # Path

node-private-key-file="/home/admin/node4/conf/key"

# Network
bootnodes=["--(hidden)--"]


# 可以访问服务器的客户端IP白名单
host-allowlist=["*"]
# 如果客户端使用浏览器访问,可以
rpc-http-cors-origins=["all"]

p2p-host="--(hidden)--"
p2p-port=30306
max-peers=42

# HTTP JSON-RPC
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8548
rpc-http-api=["ETH","NET","QBFT", "DEBUG"]

permissions-nodes-config-file-enabled=true
permissions-nodes-config-file="/home/admin/node4/conf/permissions_config.toml"
static-nodes-file="/home/admin/node4/conf/static-nodes.json"

# WS JSON-RPC
rpc-ws-host="0.0.0.0"
rpc-ws-port=8604

# Chain
genesis-file="/home/admin/node4/conf/genesis.json" # Path to the custom genesis file


# 关闭发现模式
discovery-enabled=true
nat-method="NONE"

# data storage format
data-storage-format="BONSAI"

sync-mode="FULL"
#fast-sync-min-peers=3

tx-pool-limit-by-account-percentage=1
# 节点内存中缓存待出块的交易数量最大值
tx-pool-max-size="2000"

# Mining
#miner-enabled=true
#miner-coinbase="0xfe3b557e8fb62b89f4916b721be55ceb828dbd73"

2. here is my genesis file

{
  "config" : {
    "chainId" : 20230824001,
    "berlinBlock" : 0,
    "qbft" : {
      "blockperiodseconds" : 4,
      "epochlength" : 30000,
      "requesttimeoutseconds" : 8
    }
  },
  "nonce" : "0x0",
  "timestamp" : "0x58ee40ba",
  "gasLimit" : "0x1fffffffffffff",
  "contractSizeLimit": 2147483647,
  "difficulty" : "0x1",
  "mixHash" : "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
  "coinbase" : "0x0000000000000000000000000000000000000000",
  "alloc" : {
    "3355ff1765cea77300d79b0821c10d69bbce4029" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "0xad78ebc5ac6200000"
    },
    "5ee46326f4ee40e388730e63b9aeecaddc573771" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "90000000000000000000000"
    },
    "fab25c4214ebab8f02563d1ab821f2fa337fcdb8" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "90000000000000000000000"
    },
    "e0dbb7c3f8b8f875f01e277a7364fee98e252625" : {
      "privateKey" : "--(hidden)--",
      "comment" : "private key and this comment are ignored.  In a real chain, the private key should NOT be stored",
      "balance" : "90000000000000000000000"
    }
  },
  "extraData" : "0xf87aa00000000000000000000000000000000000000000000000000000000000000000f854948ea92c56f8157187d8441babefa146701c414fe794d94f00cc3896283e21b7481759adcc57bf0d7205941a3a39ccbe340eaaa3cddece27fd4dc46f6cc8f7946ed28d2fd48ef53a7fe39f65c14a86ceccdfb457c080c0"
}

[non-fungible-nelson]

@matthew1001 any insight into this scenario? More fodder for Bonsai --> QBFT testing...

[matthew1001]

I've not done much with Bonsai yet, only Forest, so it's not a scenario I've tried myself. Does FULL sync request all transactions from block 0 (I think it does?) and if so, can Bonsai nodes provide that?