Moonriver collator stuck on relay chain block #8949248

[michaelmowry]

Aug 30 21:17:00 vm-ste-binfdot-prd-sfo3-2 moonriver[53842]: 2021-08-30 21:17:00 [Relaychain] 💤 Idle (41 peers), best: #8949573 (0x1fbd…efce), finalized #8949248 (0x102e…0032), ⬇ 75.3kiB/s ⬆ 30.4kiB/s

A Moonriver collator we are setting up has been stuck on block #8949573 during the synch process (see sample above). I have tried rebooting the machine and restarting the service but the problem persists. Any idea on why this would happen for this particular block? Other collators in the discord have reported the same issue.

OS: Ubuntu 20.04 LTS x64
Moonbeam version: 0.11.2

[crystalin]

@michaelmowry, We haven't observed this behavior internally. Also the block #8949573 doesn't seem special in any way.
If restarting is not solving the issue, I think only restoring a previous backup would solve this.

[michaelmowry]

I will try to resynch from scratch and let you know the results.

[ioannist]

We have tried syncing twice and got stuck at the same relay block

First attempt #8949573 , finalized #8949248
Second attempt #8949633 , finalized #8949248

Restarting does not help. We are syncing from scratch with 0.11.2, binary file, systemd, Ubuntu 20

[ioannist]

@michaelmowry, We haven't observed this behavior internally. Also the block #8949573 doesn't seem special in any way.
If restarting is not solving the issue, I think only restoring a previous backup would solve this.

We have a running synced node but we cannot stop it to back the DB up. Are there any public images that we can use?

[ioannist]

I will try to resynch from scratch and let you know the results.

Perhaps you should edit the ticket issue to mention block #8949248 as this is the finalized block that gets stuck in all three attempts

[xlc]

This is most likely Polkadot issue. Some new Karura nodes are also having this issue with the same Kusama block.

[michaelmowry]

I will try to resynch from scratch and let you know the results.

Perhaps you should edit the ticket issue to mention block #8949248 as this is the finalized block that gets stuck in all three attempts

I opened the ticket and for me the issue was with 8949573. It seems that there is nothing special about any of these blocks but it is happening consistently. What is the best was to troubleshoot this?

[ioannist]

I will try to resynch from scratch and let you know the results.

Perhaps you should edit the ticket issue to mention block #8949248 as this is the finalized block that gets stuck in all three attempts

I opened the ticket and for me the issue was with 8949573. It seems that there is nothing special about any of these blocks but it is happening consistently. What is the best was to troubleshoot this?

Your issue is with #8949248 (look at the finalized block of your log). It is basically the last block you were able to finalize. Other validators are also having problems with #8949248, myself included.

[michaelmowry]

After trying to resynch from scratch it failed again at the same block. I then switched to moonbeam 0.9.6 and it continues to fail.

[crystalin]

The bug has been found and a fix is currently being prepared: paritytech/substrate#9692
We will perform a client release early next week.

In the meantime, you can also use kusama backup with block over #8949248 (https://ksm-rocksdb.polkashots.io)

[ioannist]

The bug has been found and a fix is currently being prepared: paritytech/substrate#9692
We will perform a client release early next week.

In the meantime, you can also use kusama backup with block over #8949248 (https://ksm-rocksdb.polkashots.io)

Awesome. Note that these backups are not in archive mode so they won't do for Moonriver or any collator that requires archive

[crystalin]

Sorry, I thought they also had archived ones too

[atenjin]

yes we also meet the same case.

[Relaychain] 💤 Idle (22 peers), best: #8949641 (0x135a…dd8f), finalized #8949248 (0x102e…0032), ⬇ 451.9kiB/s ⬆ 1.6MiB/s

our node's collator stuck on 8949641

and the moonriver best block stuch on some block too.

substrate: [🌗] 💤 Idle (23 peers), best: #479380 (0x6475…16cc), finalized #399756 (0xaad0…f354), ⬇ 13.2kiB/s ⬆ 8.5kiB/s

when I restart the node, it will sync some block, then stuck on some height and do not sync block anymore.