Occasional Invalid justification provided error

[maxsam4]

We are seeing occasional INFO 💔 Invalid justification provided by NODE_ID for #BLOCK_HASH messages on our nodes after an era switch. The NODE_ID is then kicked from the peers list. Validators that are connected to just one sentry become stuck when this happens. This error is likely being caused by unfortunately timed reorgs when the era is changing.

I have seen this a couple of times on Kusama nodes as well.

[andresilva]

If you could provide some -lafg=trace,sync=trace logs that would be helpful, this will generate a lot of data though. Is this using a recent substrate master or v3.0?

[maxsam4]

Substrate v2.0.1

I will run one node with those flags but this error is hard to produce on a production network. A lot of stars have to align for this to happen xD. We've got this only a handful times in last 6 months across ~50 nodes.

[nuke-web3]

@maxsam4 checking in here: any updates and are you still seeing this behavior?

[maxsam4]

I haven't managed to reproduce this since but it's still an open issue. Will post logs when this happens again.

[nuke-web3]

@maxsam4 any fresh logs to inspect? If not, I would propose we close this issue for the time being.

[maxsam4]

Perhaps move this to the Substrate repo for tracking? There is a bug here probably. Just not sure exactly what that is yet :)

Anyway, I think we can close it from here.

[nuke-web3]

Please do open this back up when this happens again with the full logs to inspect 🙏🏼 we can push to substrate if this is a bug for sure 😀

[yashbhavsar007]

My node on westend chain has similar logs here is the log

Idle (0 peers), best: #5432095 (0x8e44…99df), finalized #5337990 (0x8d33…9dba), ⬇ 7.7kiB/s ⬆ 3.7kiB/s    
2021-04-30 10:40:21  ✨ Imported #5432096 (0x51c5…035b)    
2021-04-30 10:40:21  💔 Invalid justification provided by 12D3KooWLmEf7AfaWne7oDg2eMm5RPPe2HnRv8MLc2s1KZAV4WFp for #0xa251…024e    
2021-04-30 10:40:21  💔 Invalid justification provided by 12D3KooWBSMcGrNrdZmQtU9AKZowEeC2uQCwcyzR691z7JeuHGJc for #0xa251…024e  

[andresilva]

paritytech/substrate#8698 is open which should fix this. It's possible that the above could be caused by another bug but with just the logs above there's not much that can be done.

[nuke-web3]

Got it. Subscribing to that PR would be best to do and upgrade asap for those looking.

Closing this in favor of tracking in paritytech/substrate#4678