Dispute storms on validators running v.16 or lower on kusama after runtime upgrade

[alexggh]

After the runtime upgrade and the setting of nodeFeatures to 0b11010000, validators running polkadot that is v1.16 or lower are disputing candidates.

Runtime upgrade happened here: https://kusama.subsquare.io/referenda/519?tab=metadata

Issue started happening after https://polkadot.js.org/apps/?rpc=wss%3A%2F%2Fkusama-rpc.dwellir.com#/explorer/query/0x160b877451077c23c58163fe00af21e6d0d06bfc7da2cb82eaa432a409a10656, when the parachain host configuration was applied.

Identifying the nodes producing disputes: https://apps.turboflakes.io/?chain=kusama#/insights?mode=history, click on Show only disputes

[sandreim]

Block times are also degraded, similarly to #8414

[pascuin]

Just to flag this here for the sake of completion, this has effects the block finalization on Acurast Canary.

[florentina57]

One of my Kusama validator nodes, running on 1.18.2 started initiating disputes on session 47735 and continues at 2025-05-09 18:57:01 (UTC) New dispute initiated for candidate. I am attaching some logs:

validandina-lim.log.tgz

[tdimitrov]

New dispute initiated for candidate. only means that there is a dispute. It's not initiated by your node.

'Detected invalid candidate as an approval checker' or something related to 'invalid candidate' is an indication of a dispute triggered by the local node.

[tugytur]

We didn't have any 'invalid candidate related logs. But here are the logs for the last 12 hours from our active Amforc validators

validator1.tar.gz
validator2.tar.gz
validator3.tar.gz
validator4.tar.gz
validator5.tar.gz
validator6.tar.gz
validator7.tar.gz

[florentina57]

New dispute initiated for candidate. only means that there is a dispute. It's not initiated by your node.

'Detected invalid candidate as an approval checker' or something related to 'invalid candidate' is an indication of a dispute triggered by the local node.

Thanks for clarifying that. My bad regarding the log message.

However, I actually noticed the dispute on turboflakes. I saw that Paulo, from turboflakes, just posted a list 11 DN nodes, including mine, who have initiated disputes despite being on version 1.18.2

[paulormart]

@florentina57 could you share logs starting from ~17:00 UTC? The block number doesn't seem to be included in the logs you shared earlier.

[alexggh]

However, I actually noticed the dispute on turboflakes. I saw that Paulo, from turboflakes, just posted a list 11 DN nodes, including mine, who have initiated disputes despite being on version 1.18.2

If you look here https://apps.turboflakes.io/?chain=kusama#/insights?mode=history, between session 47732 and session 47734, you are going to see that we have disputes only from nodes that have v1.16 or lower, after that there seems to be some from 1.18.2, but at that point finality is already lagging hundreds of blocks, so it might be just a result of nodes being overloaded.

[florentina57]

@florentina57 could you share logs starting from ~17:00 UTC? The block number doesn't seem to be included in the logs you shared earlier.

Thanks @paulormart I just tried getting those logs, but I have so many logs that the oldest I have as of now is 18:36 (UTC) so I guess I lost all logs prior to this time :/

validandina-lim-dispute.log.tgz

[sandreim]

Quick update, a temporary fix is ready: #8483

[tugytur]

Quick update, a temporary fix is ready: #8483

will this release come out tonight?

[sandreim]

We merged the fix.

The release will take a bit more time. In the mean time a binary with the fix can be built from the head of the stable2503 branch: https://github.com/paritytech/polkadot-sdk/tree/stable2503

[lukeisontheroad]

What’s the status now? It’s still not working. Please communicate.

[florentina57]

We merged the fix.

The release will take a bit more time. In the mean time a binary with the fix can be built from the head of the stable2503 branch: https://github.com/paritytech/polkadot-sdk/tree/stable2503

I have compiled the binaries with the fix and upgraded both of my Kusama Validator Nodes. One of the two still saw some error=AuthorityFlooding after restarting the node but stabilized after 10-15 mins. Both nodes seem to have "normal" logs, aside from last finalized block that remains at #28282123.

[sandreim]

Emergency fix release is out: https://github.com/paritytech/polkadot-sdk/releases/tag/polkadot-stable2503-3