polkadot-parachain v1.10.0 breaks collating on polkadot system chains

[stakeworld]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

After updating polkadot-parachain from 1.9.0 to 1.10.0 the collator stops producing blocks and in the logs throws the following error every 6 seconds:
Apr 10 13:35:06 polkadot-parachain[1787724]: 2024-04-10 13:35:06 [Relaychain] failed to handle new activations err=Runtime(NotSupported { runtime_api_name: "para_backing_state" })

Apr 10 13:35:13 polkadot-parachain[1787724]: 2024-04-10 13:35:13 [Relaychain] failed to handle new activations err=Runtime(NotSupported { runtime_api_name: "para_backing_state" })

Reverting the relay chain (polkadot) to 1.9.0 didn't fix, reverting polkadot-parachain to 1.9.0 fixed the problem.

Steps to reproduce

Updating polkadot and polkadot-parachain to 1.10.0 on an active collator

[stakeworld]

The invulnerable parity collators are probably already updated and are not producing blocks. Also I notice some strange double last blocks in the collators list were two collators have the same last block? Not sure if it is related.

[PierreBesson]

I confirm this is happening to Parity collators running 1.10.

[sandreim]

#3795 broke it. The missing API is introduced by async backing which is unfortunately not present on Polkadot. I will work on a fix. In the mean time, it should be fine to switch back to polkadot-parachain 1.9.0

[eskimor]

I believe integration tests are in the works somewhere? Would they cover things like this in the future?

[sandreim]

I believe integration tests are in the works somewhere? Would they cover things like this in the future?

We'd need to pull polkadot runtime from fellowship repo and have a zombienet test in this repo.

[eskimor]

yeah, but the problem is actually that the network is still running the old runtime.

So we need to have an integration test in polkadot-sdk to test with the currently live runtime (can be fetched from chain) (and ideally with any released, but not yet launched one). And in the fellowship, we should do the same prior to release with the currently released node.

I think we need it both ways, otherwise issues will slip. E.g. having that integration test in the fellowship runtime repo would not have caught this issue.

[sandreim]

Fix is in hotfix release v1.10.1