Description
Issue
After having installed the 2304.0-prerelease distribution on their signer node running on the pre-release-preview network, almost all the SPOs received error messages preventing them to contribute to the creation of the multi-signatures.
They started receiving a ProtocolInitializerNotRegistered(CoreRegister(UnregisteredInitializer)) error at the epoch after installing the new version of the node. The following epoch the error message received is the epoch just after installing the new version of the signer. Only 2 nodes are still able to sign after updating the version, one of them is maintained by the Mithril team and is running on the same computer as the aggregator of the network.
Analysis
After investigation of the logs and of the data stores provided by the SPOs it appears that the verification keys used by the aggregator are associated to a protocol initializer with a +1 offset in the signer stores from what is expected. This must probably explain why the verification key retrieved from the pending certificate does not match the verification key generated from the protocol initializer.
This problem seems to have been introduced in the distribution 2302.0 and is tightly coupled to the time needed for the aggregator node to retrieve the stake distribution from the Cardano node to which it is attached.
Solution
Provide a closing of the registration round for a new epoch until the stake distribution is computed: this will avoid the creation of this offset (signer registration epoch = aggregator registration round epoch + 1). This will introduce error message when signers attempt to register when a round is not opened yet for the new epoch, but the signer state machine is resilient to these errors
⚠ After deploying the fix on the signer node we will need a 2 epochs delay to get signer contributing to signatures