ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected A74D98958D7A6D4EDFFBBF760EDE9A27DF9141A379DBB1E55F5A6AF71C5D3A2C, got 1F7F6326657EB801001C4E49848D882551B3F2B4CA1259CF533E3F1BE8AFD339

[IvanDedov]

Hi guys!
Node stopped working

spec is like below:

OS: Ubuntu 16.04.6 LTS

version: Binance Chain Release: 0.6.3-hf.1;Binance Chain Commit: a5abb0f; Cosmos Release: v0.25.0-binance.21; Tendermint Release: v0.32.3-binance.1;

config.toml

fast_sync = true
state_sync_reactor = true
state_sync_height = 0
hot_sync = true
hot_sync_timeout = "3s"
hot_sync_reactor = true

---

Startup error

bnb@bcs:$ /usr/local/bin/bnbchaind start --home /srv/HDD4/home/bnb/.bnbchaind --pruning breathe&
[1] 32547
bnb@bcs:$ panic: Failed to process committed block (90913099:6EAF26E7AA83DD2B2513CA432793847B50E8243EDBBF992BB79D8C2568F6FEDA): Wrong Block.Header.AppHash. Expected A74D98958D7A6D4EDFFBBF760EDE9A27DF9141A379DBB1E55F5A6AF71C5D3A2C, got 1F7F6326657EB801001C4E49848D882551B3F2B4CA1259CF533E3F1BE8AFD339

goroutine 36949 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc0152ca340)
/home/suyu/go/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.1/blockchain/v0/reactor.go:399 +0x1bf0
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
/home/suyu/go/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.1/blockchain/v0/reactor.go:126 +0x84

[jseibel-circle]

Same issue for me. Same log message and then node dies and restarts, gets to this point and dies again, repeat forever

[Alteryy]

I have the exact same error.
./state_recover 90913098 dose not work.

[unclezoro]

Hi, can you guys also post the app.toml file here as well.

[HaoyangLiu]

@IvanDedov Could you share your app.toml

[unclezoro]

I have the exact same error.
./state_recover 90913098 dose not work.

Before we locate the root cause, can you try with ./state_recover 90913097 and feed back here? Thanks @Alteryy

[IvanDedov]

# This is a TOML config file.
# For more information, see https://github.com/toml-lang/toml

[base]
# Interval blocks of breathe block, if breatheBlockInterval is 0, breathe block will be created every day.
breatheBlockInterval = 0
# Size of account cache
accountCacheSize = 30000
# Size of signature cache
signatureCacheSize = 30000
# Running mode when start up, 0: Normal, 1: TransferOnly, 2: RecoverOnly
startMode = 0
# Concurrency of OrderKeeper, should be power of 2
orderKeeperConcurrency = 2
# Days count back for breathe block
breatheBlockDaysCountBack = 7

[upgrade]
# Block height of BEP6 upgrade
BEP6Height = 20300000
# Block height of BEP9 upgrade
BEP9Height = 20300000
# Block height of BEP10 upgrade
BEP10Height = 20300000
# Block height of BEP19 upgrade
BEP19Height = 20300000
# Block height of BEP12 upgrade
BEP12Height = 25120000
# Block height of BEP3 upgrade
BEP3Height = 41277000
# Block height of FixSignBytesOverflow upgrade
FixSignBytesOverflowHeight = 51467800
# Block height of LotSizeOptimization upgrade
LotSizeUpgradeHeight = 51467800
# Block height of changing listing rule upgrade
ListingRuleUpgradeHeight = 51467800
# Block height of FixZeroBalanceHeight upgrade
FixZeroBalanceHeight = 51467800

[addr]
# Bech32PrefixAccAddr defines the Bech32 prefix of an account's address
bech32PrefixAccAddr = "bnb"
# Bech32PrefixAccPub defines the Bech32 prefix of an account's public key
bech32PrefixAccPub = "bnbp"
# Bech32PrefixValAddr defines the Bech32 prefix of a validator's operator address
bech32PrefixValAddr = "bva"
# Bech32PrefixValPub defines the Bech32 prefix of a validator's operator public key
bech32PrefixValPub = "bvap"
# Bech32PrefixConsAddr defines the Bech32 prefix of a consensus node address
bech32PrefixConsAddr = "bca"
# Bech32PrefixConsPub defines the Bech32 prefix of a consensus node public key
bech32PrefixConsPub = "bcap"

##### publication related configurations #####
[publication]
# configurations ends with Kafka can be a semi-colon separated host-port list
# Whether we want publish market data (this includes trades and order)
publishOrderUpdates = true
orderUpdatesTopic = "orders"
orderUpdatesKafka = "127.0.0.1:9092"

# Whether we want publish account balance to notify browser db indexer persist latest account balance change
publishAccountBalance = true
accountBalanceTopic = "accounts"
accountBalanceKafka = "127.0.0.1:9092"

# Whether we want publish order book changes
publishOrderBook = true
orderBookTopic = "orders"
orderBookKafka = "127.0.0.1:9092"

# Whether we want publish block fee changes
publishBlockFee = true
blockFeeTopic = "accounts"
blockFeeKafka = "127.0.0.1:9092"

# Whether we want publish transfers
publishTransfer = true
transferTopic = "transfers"
transferKafka = "127.0.0.1:9092"

# Global setting
publicationChannelSize = "10000"
publishKafka = false
publishLocal = true
# max size in megabytes of marketdata json file before rotate
localMaxSize = 1024
# max days of marketdata json files to keep before deleted
localMaxAge = 7

[log]

# Write logs to console instead of file
logToConsole = false

## The below parameters take effect only when logToConsole is false
# Log file root, if not set, use home path
logFileRoot = ""
# Log file path relative to log file root path
logFilePath = "bnc.log"
# Number of logs keep in memory before writing to file
logBuffSize = 10000

[IvanDedov]

I have the exact same error.
./state_recover 90913098 dose not work.

Before we locate the root cause, can you try with ./state_recover 90913097 and feed back here? Thanks @Alteryy

bnb@bcs:$ ./state_recover 90913097 /srv/HDD4/home/bnb/.bnbchaind
recover home_path[/srv/HDD4/home/bnb/.bnbchaind] to height[90913097]
recover success
bnb@bcs:$ /usr/local/bin/bnbchaind start --home /srv/HDD4/home/bnb/.bnbchaind --pruning breathe&
[1] 29150
bnb@bcs:~$ panic: Failed to process committed block (90913099:6EAF26E7AA83DD2B2513CA432793847B50E8243EDBBF992BB79D8C2568F6FEDA): Wrong Block.Header.AppHash. Expected A74D98958D7A6D4EDFFBBF760EDE9A27DF9141A379DBB1E55F5A6AF71C5D3A2C, got 1F7F6326657EB801001C4E49848D882551B3F2B4CA1259CF533E3F1BE8AFD339

goroutine 36896 [running]:
github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).poolRoutine(0xc01554a4e0)
/home/suyu/go/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.1/blockchain/v0/reactor.go:399 +0x1bf0
created by github.com/tendermint/tendermint/blockchain/v0.(*BlockchainReactor).OnStart
/home/suyu/go/pkg/mod/github.com/binance-chain/bnc-tendermint@v0.32.3-binance.1/blockchain/v0/reactor.go:126 +0x84

[1]+ Exit 2 /usr/local/bin/bnbchaind start --home /srv/HDD4/home/bnb/.bnbchaind --pruning breathe

[HaoyangLiu]

Could you execute df -h to show your disk information?

[IvanDedov]

Could you execute df -h to show your disk information?

bnb@bcs:~$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 32G 0 32G 0% /dev
tmpfs 6.3G 642M 5.7G 11% /run
/dev/mapper/ubuntu--vg-root 457G 96G 338G 23% /
tmpfs 32G 0 32G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/sda1 459G 306G 130G 71% /srv/HDD1
/dev/sdb1 459G 382G 54G 88% /srv/HDD2
/dev/sde1 3.6T 583G 2.9T 17% /srv/HDD4
/dev/sdc1 220G 88G 122G 42% /srv/HDD3
/dev/sdd2 721M 190M 495M 28% /boot
/dev/sdd1 511M 3.6M 508M 1% /boot/efi

[HaoyangLiu]

Try these steps:

• Delete addrbook.json
• Empty seed and persistent_peers in config.toml

# Comma separated list of seed nodes to connect to
seeds = ""

# Comma separated list of nodes to keep persistent connections to
persistent_peers = ""

• ./state_recover 90913098
• Start node again, without --pruning flag
• Execute this command and share the response

curl localhost:26657/block_results?height=90913098

[IvanDedov]

bnb@bcs:/srv/HDD4/home# curl localhost:26657/block_results?height=90913098
{
"jsonrpc": "2.0",
"id": "",
"error": {
"code": -32603,
"message": "Internal error",
"data": "Height must be less than or equal to the current blockchain height"
}

[IvanDedov]

bnb@bcs:/srv/HDD4/home# tail -f /srv/HDD4/home/bnb/.bnbchaind/bnc.log
I[2020-05-30|19:16:43.368] ABCI Handshake App Info module=consensus height=90913098 hash=A74D98958D7A6D4EDFFBBF760EDE9A27DF9141A379DBB1E55F5A6AF71C5D3A2C software-version= protocol-version=0
I[2020-05-30|19:16:43.368] ABCI Replay Blocks module=consensus appHeight=90913098 storeHeight=90913098 stateHeight=90913098
I[2020-05-30|19:16:43.368] Completed ABCI Handshake - Tendermint and App are synced module=consensus appHeight=90913098 appHash=A74D98958D7A6D4EDFFBBF760EDE9A27DF9141A379DBB1E55F5A6AF71C5D3A2C
I[2020-05-30|19:16:43.368] This node is not a validator module=consensus addr=B4F6003531C336F2EED86DDD668D4A8B2C31B9F2 pubKey=PubKeyEd25519{C1B26FF8FB267F8DD40BB9CE07570CF52E1DDA7DF1470ED42B466780057888A3}
I[2020-05-30|19:16:43.410] Starting BlockchainReactor module=blockchain impl=BlockchainReactor
I[2020-05-30|19:16:43.410] Starting BlockPool module=blockchain impl=BlockPool
I[2020-05-30|19:16:43.410] Starting ConsensusReactor module=consensus impl=ConsensusReactor
I[2020-05-30|19:16:43.410] ConsensusReactor module=consensus fastSync=true
I[2020-05-30|19:16:43.410] Starting StateSyncReactor module=statesync impl=StateSyncReactor
I[2020-05-30|19:16:43.410] This node does not need state sync, it might has already state synced, please check module=statesync lockfile=STATESYNC.LOCK

[IvanDedov]

Try these steps:

• Delete addrbook.json
• Empty seed and persistent_peers in config.toml

# Comma separated list of seed nodes to connect to
seeds = ""

# Comma separated list of nodes to keep persistent connections to
persistent_peers = ""

• ./state_recover 90913098
• Start node again, without --pruning flag
• Execute this command and share the response

curl localhost:26657/block_results?height=90913098

It's all done
What should I do next?

[chainwhisper]

bnb@bcs:/srv/HDD4/home# curl localhost:26657/block_results?height=90913098
{
"jsonrpc": "2.0",
"id": "",
"error": {
"code": -32603,
"message": "Internal error",
"data": "Height must be less than or equal to the current blockchain height"
}

I think the default port is 27147 for mainnet.

could you please check curl localhost:27147/status

[IvanDedov]

bnb@bcs:/srv/HDD4/home# curl localhost:26657/block_results?height=90913098
{
"jsonrpc": "2.0",
"id": "",
"error": {
"code": -32603,
"message": "Internal error",
"data": "Height must be less than or equal to the current blockchain height"
}

I think the default port is 27147 for mainnet.

could you please check curl localhost:27147/status

bnb@bcs:/srv/HDD4/home# curl localhost:27147/status
{
"jsonrpc": "2.0",
"id": "",
"result": {
"node_info": {
"protocol_version": {
"p2p": "7",
"block": "10",
"app": "0"
},
"id": "1969aa323aa02f990cf934a7a7905456ab533b5d",
"listen_addr": "tcp://0.0.0.0:27146",
"network": "Binance-Chain-Tigris",
"version": "0.32.3",
"channels": "36402021222330380041",
"moniker": "xxx",
"other": {
"tx_index": "on",
"rpc_address": "tcp://0.0.0.0:27147"
}
},
"sync_info": {
"latest_block_hash": "B6562C89B41CD77A86D36C7C8CADFB3A5A70D86E5D51A99B6D57CE9A0EFF53D7",
"latest_app_hash": "28E0E03608674241985DACA5E05266EA7BE84CD3A6CD36C2961B88A9140E6762",
"latest_block_height": "90913098",
"latest_block_time": "2020-05-30T02:22:59.967462567Z",
"catching_up": true,
"index_height": "90913098"
},
"validator_info": {
"address": "B4F6003531C336F2EED86DDD668D4A8B2C31B9F2",
"pub_key": {
"type": "tendermint/PubKeyEd25519",
"value": "wbJv+Psmf43UC7nOB1cM9S4d2n3xRw7UK0ZngAV4iKM="
},
"voting_power": "0"
}
}