ChatGPT wrote this up, I understood it but I am not very technical in how to explain the issue that I noticed/observedduring DevNet testing of our SCC fork.
MNAUTH can remain permanently asymmetric when received before MN sync, causing quorum DKG relay failure
Summary
We observed a connection state in which two active masternodes share the same TCP connection, but the connection is MNAUTH-verified on only one side.
The sequence appears to be:
- Masternode B establishes an ordinary
outbound-full-relay connection to masternode A while B's masternode/blockchain sync is not yet complete.
- Both sides exchange MNAUTH.
- A is sufficiently synced and successfully verifies B's MNAUTH.
- B receives A's MNAUTH while
mn_sync.IsBlockchainSynced() is false.
CMNAuth::ProcessMessage() returns without processing the MNAUTH.
- The MNAUTH is not replayed after synchronization.
- A therefore considers B an authenticated masternode peer, while B permanently considers A an ordinary peer.
- If the deterministic quorum topology expects A to initiate toward B, B never considers A one of its own pending outbound quorum connections.
- The existing 5-second
slow_handshake cleanup in ThreadOpenMasternodeConnections() therefore does not inspect or disconnect this connection on B.
- A can subsequently relay DKG inventory over the connection because A believes B is MNAUTH-verified, while B rejects/refuses the corresponding DKG object because A is not verified from B's perspective.
This produced an actual lost DKG contribution in our test.
Where this was observed
The reproduction occurred on StakeCubeCoin, a Dash-derived network currently migrating onto a Dash 23.1.x code base.
The relevant connection-management and MNAUTH logic was checked against Dash v23.1.7/v23.1.8 and is unchanged in the areas involved in this issue, including:
- the
CMNAuth::ProcessMessage() synchronization guard;
ThreadOpenMasternodeConnections();
connectedNodes;
connectedProRegTxHashes;
- the 5-second non-MNAUTH
slow_handshake cleanup;
- directional deterministic quorum connections.
The SCC build used for the reproduction reports:
SCC Core version v5.0.1-5f5135fa86
protocol version 70240
This report is therefore not claiming that the failure has already been reproduced on Dash mainnet/testnet. It is a reproducible failure on a Dash-derived implementation carrying the relevant Dash 23.1.x networking logic.
Relevant upstream behavior
CMNAuth::ProcessMessage() currently contains:
if (msg_type != NetMsgType::MNAUTH || !mn_sync.IsBlockchainSynced()) {
// we can't verify MNAUTH messages when we don't have the latest MN list
return {};
}
Thus an MNAUTH received before masternode blockchain synchronization is complete is consumed without authenticating the connection.
There does not appear to be a mechanism that subsequently:
- reprocesses that MNAUTH;
- requests another MNAUTH;
- resends MNAUTH after synchronization; or
- necessarily tears down the still-unverified connection.
There is recovery logic in ThreadOpenMasternodeConnections():
if (connectedNodes.count(addr2)) {
// we probably connected to it before it became a masternode
// or maybe we are still waiting for mnauth
bool slow_handshake =
pnode &&
pnode->nTimeFirstMessageReceived.load() != 0s &&
GetTime<std::chrono::seconds>() -
pnode->nTimeFirstMessageReceived.load() > 5s;
if (slow_handshake) {
LogPrint(..., "dropping non-mnauth connection ...");
pnode->fDisconnect = true;
}
continue;
}
However, this check occurs while iterating the local node's masternodeQuorumNodes / pending outbound quorum relationships.
That does not cover the reverse-direction case described below.
Reproduced topology
The quorum contained 12 deterministic members.
For clarity:
A = node 109
proTxHash =
a72037d07fb4c0e3cd21f69986a970021454697b5034651b3fb6a13b4c80c68a
B = node 107
proTxHash =
4b4e5314c68749049139fa58f20b4e335569a0507e986eeaf572355871b3f5b0
A was quorum member index 7.
B was quorum member index 11.
With the sparse deterministic relay topology, the outbound offsets for a 12-member quorum are effectively:
Therefore A's relay/outbound set was:
index 8 -> member 905
index 9 -> member 904
index 11 -> B / node 107
The runtime DKG log confirmed:
forMember[a720]
relayMembers[30ff | 3696 | 4b4e]
where:
30ff = node 905
3696 = node 904
4b4e = node 107
B's outbound relationship was different and did not include A.
For B/index 11:
+1 -> index 0
+2 -> index 1
+4 -> index 3
So A was not one of the members B was responsible for connecting outbound to.
This directional relationship is important to the failure.
Connection timeline
At 02:45:18, B established an ordinary full-relay connection to A:
CConnman::OpenNetworkConnection -- connecting to [...:109]:20000
...
Added connection to [...:109]:20000 peer=10
...
New outbound-full-relay v1 peer connected:
version: 70240,
peer=10,
peeraddr=[...:109]:20000
B sent its own MNAUTH:
CMNAuth::PushMNAUTH -- Sending MNAUTH, peer=10
B also received an MNAUTH from A:
received: mnauth (128 bytes) peer=10
However, there was no corresponding:
CMNAuth::ProcessMessage -- constructed signHash ...
CMNAuth::ProcessMessage -- Valid MNAUTH ...
for peer 10.
The synchronization log explains why.
Immediately before this:
02:45:13
Sync Tick -- asset_id 1
nSyncProgress 0.000000
Immediately after:
02:45:19
Sync Tick -- asset_id 1
nSyncProgress 0.000000
Masternode blockchain synchronization did not complete until:
02:45:25
CMasternodeSync::SwitchToNextAsset --
Completed MASTERNODE_SYNC_BLOCKCHAIN in 19s
Thus A's MNAUTH reached B approximately seven seconds before B completed MASTERNODE_SYNC_BLOCKCHAIN.
This is consistent with the early return in:
CMNAuth::ProcessMessage()
when:
!mn_sync.IsBlockchainSynced()
Opposite side successfully authenticated
On A, the same connection was accepted inbound.
A received B's MNAUTH and successfully verified it:
CMNAuth::ProcessMessage -- constructed signHash
for nVersion 70240, legacy_bls=0, peer=12
CMNAuth::ProcessMessage --
Valid MNAUTH for
4b4e5314c68749049139fa58f20b4e335569a0507e986eeaf572355871b3f5b0,
peer=12
Therefore the TCP relationship became asymmetric:
A's view of connection:
B = verified masternode
verifiedProRegTxHash = 4b4e...
usable quorum relay peer
B's view of the same relationship:
A = ordinary outbound-full-relay peer
verifiedProRegTxHash = null
not a masternode/quorum peer
This state persisted.
Why the existing 5-second cleanup did not repair it
Current Dash has logic in ThreadOpenMasternodeConnections() intended to handle an existing non-MNAUTH connection to a desired quorum member.
If a desired outbound quorum member already has a connection but has not authenticated after approximately five seconds, the connection is disconnected so that a proper masternode connection can subsequently be made.
That works if the affected peer is in this node's pending outbound quorum set.
It does not appear to cover this case because the connection relationship is directional.
A was supposed to connect to B.
B was not supposed to connect to A.
Therefore on B:
A ∉ B's outbound masternodeQuorumNodes
so B's getPendingQuorumNodes() does not iterate A.
Consequently B never executes the existing slow_handshake cleanup against its ordinary connection to A.
On A, no cleanup is needed from its perspective because A successfully authenticated B.
The asymmetric state can therefore persist indefinitely.
DKG failure
At 02:49:12, several minutes after both nodes were synchronized, A entered the DKG contribution phase normally:
generating contributions
generated contributions. time=1
sending contributions
encrypted contributions. time=4
A successfully created its contribution:
qcontrib
9d6477b217b9060d2f5cb659066b8b60f80cd6e7b5b7390baf1c4be5a1c5d77f
The relay topology selected:
relayMembers[30ff | 3696 | 4b4e]
Two of these members were deliberately offline/incompatible in this migration test:
30ff / node 905 = unavailable
3696 / node 904 = unavailable
Therefore B/node 107 was the sole live deterministic relay path for A's contribution.
A sent the INV over its authenticated connection to B:
PushInv -- adding new inv:
qcontrib 9d6477...
peer=12
SendMessages -- queued inv:
qcontrib 9d6477...
peer=12
sending inv
peer=12
On A, peer 12 was authenticated as B (4b4e...).
B immediately received the inventory over peer 10:
received: inv (37 bytes) peer=10
got inv:
qcontrib
9d6477b217b9060d2f5cb659066b8b60f80cd6e7b5b7390baf1c4be5a1c5d77f
new peer=10
But B's DKG connection diagnostics still classified A as:
node[10:[...:109]:20000] not mn
The contribution therefore could not be accepted through this connection as an authenticated DKG source.
In our fork we have an additional diagnostic/workaround which defers GETDATA for DKG inventory announced by a non-MNAUTH peer. This made the condition explicit:
GETDATA deferring DKG inv from non-verified peer
inv=(qcontrib 9d6477...)
peer=10
That fork-specific GETDATA behavior is not the root cause being reported here.
The root cause is that the connection was permanently MNAUTH-verified on A but permanently unverified on B.
Without our diagnostic patch, the underlying asymmetric connection state still exists; the downstream handling of the DKG object may simply fail differently.
Evidence that the contribution itself was valid
A did not fail to participate.
It generated the contribution immediately upon entering the correct DKG phase.
The contribution hash existed locally on A.
A relayed the inventory according to its deterministic relay set.
Across the other quorum-node logs, only B observed this contribution hash.
That matches the deterministic topology exactly: the other two relay targets were unavailable, leaving B as the only live path.
Thus this was not:
- a BLS generation failure;
- a late contribution;
- a DKG worker stall;
- an invalid contribution;
- a slow propagation event.
It was a connection/authentication liveness failure.
Result inside the DKG
Other healthy quorum members did not receive A's contribution and therefore voted A bad during the complaint phase.
For example, complaints contained:
... voted for
a72037d07fb4c0e3cd21f69986a970021454697b5034651b3fb6a13b4c80c68a
to be bad
even though A was online and had successfully generated its contribution.
This means the issue can make a healthy masternode indistinguishable from a non-participating masternode at the DKG layer.
Depending on quorum size, topology and PoSe configuration, this could potentially:
- reduce the valid-member count;
- cause quorum formation failure if enough contributions become unreachable;
- generate bad-member votes against healthy nodes;
- contribute to false PoSe punishment when DKG PoSe is enabled.
Why sparse topology made the issue easy to reproduce
Our reproduction used the sparse deterministic connection topology.
That is not necessarily required for the underlying MNAUTH asymmetry.
The underlying problem is simply:
one TCP connection
+
MNAUTH succeeds only one direction
+
no subsequent recovery
Sparse topology made the problem deterministic and visible because A had three relay targets and only one of them was live.
Therefore there was no alternate verified source from which the missing DKG object could be recovered.
With all-connected quorum topology, redundant announcements from other authenticated members might mask the DKG loss, but the one-way MNAUTH connection state can still exist.
Historical connection with existing Dash behavior
This appears related to assumptions from earlier Dash quorum networking work.
PR #3380 established deterministic connection direction so that only one member of a pair needs to initiate a connection while the other can wait for the inbound connection.
PR #2798 subsequently changed DKG/sig-share relay to use MNAUTHed connections, including inbound connections.
Those behaviors assume that after connection establishment the MNAUTH state will become usable on both sides.
The case above violates that assumption:
initiating side sends MNAUTH while receiver is unsynced
↓
receiver silently ignores MNAUTH
↓
other side successfully authenticates receiver
↓
one side believes the required quorum relationship exists
↓
other side does not
The current 5-second cleanup handles this if the unverified remote member is in the local node's required outbound connection set.
It appears not to handle the reverse-direction case where the remote member is expected to initiate toward this node.
Expected behavior
A connection used as an intra-quorum connection should eventually converge to one of two states:
1. both sides have successfully authenticated the opposite masternode
or
2. the connection is torn down and a fresh MNAUTH-capable connection is established
A persistent state where:
A considers B authenticated
B considers A unauthenticated
should not survive indefinitely.
Possible fixes
Several approaches appear possible.
Option 1: retain/reprocess pre-sync MNAUTH
If a validly framed MNAUTH is received while:
!mn_sync.IsBlockchainSynced()
retain one pending MNAUTH per connection and process it after synchronization completes.
Since MNAUTH is tied to the current connection challenge, this should only be retained for the lifetime of that connection.
Option 2: force a fresh handshake after synchronization
Record that MNAUTH was received while it could not yet be verified.
When masternode blockchain synchronization completes, disconnect such still-unverified connections.
The remote side can then reconnect and exchange MNAUTH while both sides are ready.
This may be simpler than retaining and replaying the message.
Option 3: expand stale non-MNAUTH quorum-peer cleanup
The existing 5-second cleanup could consider every deterministic peer with which the local node is expected to have an intra-quorum relationship in either direction, rather than only peers in the local node's outbound masternodeQuorumNodes set.
Conceptually, the cleanup set would include:
members we should connect to
UNION
members that should connect to us
If such a member already has an ordinary connection that remains non-MNAUTH after the grace period, disconnect it.
In the reproduced case, B would then recognize A as a required reverse-direction quorum peer, drop the ordinary B→A connection, and A would subsequently establish the proper A→B quorum connection.
Suggested functional regression test
A focused functional test could reproduce the state without requiring a failed DKG first.
- Create active masternodes A and B.
- Choose topology/order such that deterministic quorum connectivity requires A to initiate toward B, not B toward A.
- Start or restart B such that
mn_sync.IsBlockchainSynced() is still false.
- Before B completes masternode blockchain sync, force B to establish an ordinary non-masternode connection to A.
- Verify A receives and accepts B's MNAUTH.
- Verify B receives A's MNAUTH while unsynced and does not authenticate A.
- Allow B to finish synchronization.
- Wait significantly longer than the existing 5-second
slow_handshake threshold.
- Assert that the connection cannot remain indefinitely asymmetric.
Before a fix, expected failure state:
A:
peer B has verified_proregtx_hash == B
B:
peer A has verified_proregtx_hash == null
connection remains ordinary
After a fix, require convergence to either:
or:
old connection disconnected
fresh quorum connection established
both sides usable for DKG
An extended version of the test could start a DKG round and verify that a contribution originating on A is accepted by B over the resulting quorum connection.
Additional note
Our test environment deliberately included unavailable quorum members while exercising a protocol migration. That made the propagation consequence severe because the asymmetric A→B relationship became the only live route for A's contribution.
The unavailable members are not required for the underlying MNAUTH bug. They only removed relay redundancy and made the failure observable.
The condition worth investigating upstream is therefore:
Can an MNAUTH message received before mn_sync.IsBlockchainSynced() become permanently lost on one side of a connection when deterministic quorum connection direction means the unverified side never runs the existing slow_handshake cleanup for that peer?
System information
Ubuntu 24.04
Github 23.1.7 / 23.1.8 (self compiled)
Ubuntu 24.04 - Server class machine ( Poweredge T630 )
Attached are the two full debug logs (with debug=1) from our block 480 V19 cutover test with 12 v5 nodes (70240 version) and 5 of our older v3 nodes (70224 version)
These two nodes are version 70240.
debug007-107.log
debug109.log
ChatGPT wrote this up, I understood it but I am not very technical in how to explain the issue that I noticed/observedduring DevNet testing of our SCC fork.
MNAUTH can remain permanently asymmetric when received before MN sync, causing quorum DKG relay failure
Summary
We observed a connection state in which two active masternodes share the same TCP connection, but the connection is MNAUTH-verified on only one side.
The sequence appears to be:
outbound-full-relayconnection to masternode A while B's masternode/blockchain sync is not yet complete.mn_sync.IsBlockchainSynced()is false.CMNAuth::ProcessMessage()returns without processing the MNAUTH.slow_handshakecleanup inThreadOpenMasternodeConnections()therefore does not inspect or disconnect this connection on B.This produced an actual lost DKG contribution in our test.
Where this was observed
The reproduction occurred on StakeCubeCoin, a Dash-derived network currently migrating onto a Dash 23.1.x code base.
The relevant connection-management and MNAUTH logic was checked against Dash v23.1.7/v23.1.8 and is unchanged in the areas involved in this issue, including:
CMNAuth::ProcessMessage()synchronization guard;ThreadOpenMasternodeConnections();connectedNodes;connectedProRegTxHashes;slow_handshakecleanup;The SCC build used for the reproduction reports:
This report is therefore not claiming that the failure has already been reproduced on Dash mainnet/testnet. It is a reproducible failure on a Dash-derived implementation carrying the relevant Dash 23.1.x networking logic.
Relevant upstream behavior
CMNAuth::ProcessMessage()currently contains:Thus an MNAUTH received before masternode blockchain synchronization is complete is consumed without authenticating the connection.
There does not appear to be a mechanism that subsequently:
There is recovery logic in
ThreadOpenMasternodeConnections():However, this check occurs while iterating the local node's
masternodeQuorumNodes/ pending outbound quorum relationships.That does not cover the reverse-direction case described below.
Reproduced topology
The quorum contained 12 deterministic members.
For clarity:
A was quorum member index 7.
B was quorum member index 11.
With the sparse deterministic relay topology, the outbound offsets for a 12-member quorum are effectively:
Therefore A's relay/outbound set was:
The runtime DKG log confirmed:
where:
B's outbound relationship was different and did not include A.
For B/index 11:
So A was not one of the members B was responsible for connecting outbound to.
This directional relationship is important to the failure.
Connection timeline
At
02:45:18, B established an ordinary full-relay connection to A:B sent its own MNAUTH:
B also received an MNAUTH from A:
However, there was no corresponding:
for peer 10.
The synchronization log explains why.
Immediately before this:
Immediately after:
Masternode blockchain synchronization did not complete until:
Thus A's MNAUTH reached B approximately seven seconds before B completed
MASTERNODE_SYNC_BLOCKCHAIN.This is consistent with the early return in:
CMNAuth::ProcessMessage()when:
Opposite side successfully authenticated
On A, the same connection was accepted inbound.
A received B's MNAUTH and successfully verified it:
Therefore the TCP relationship became asymmetric:
This state persisted.
Why the existing 5-second cleanup did not repair it
Current Dash has logic in
ThreadOpenMasternodeConnections()intended to handle an existing non-MNAUTH connection to a desired quorum member.If a desired outbound quorum member already has a connection but has not authenticated after approximately five seconds, the connection is disconnected so that a proper masternode connection can subsequently be made.
That works if the affected peer is in this node's pending outbound quorum set.
It does not appear to cover this case because the connection relationship is directional.
A was supposed to connect to B.
B was not supposed to connect to A.
Therefore on B:
so B's
getPendingQuorumNodes()does not iterate A.Consequently B never executes the existing
slow_handshakecleanup against its ordinary connection to A.On A, no cleanup is needed from its perspective because A successfully authenticated B.
The asymmetric state can therefore persist indefinitely.
DKG failure
At
02:49:12, several minutes after both nodes were synchronized, A entered the DKG contribution phase normally:A successfully created its contribution:
The relay topology selected:
Two of these members were deliberately offline/incompatible in this migration test:
Therefore B/node 107 was the sole live deterministic relay path for A's contribution.
A sent the INV over its authenticated connection to B:
On A, peer 12 was authenticated as B (
4b4e...).B immediately received the inventory over peer 10:
But B's DKG connection diagnostics still classified A as:
The contribution therefore could not be accepted through this connection as an authenticated DKG source.
In our fork we have an additional diagnostic/workaround which defers GETDATA for DKG inventory announced by a non-MNAUTH peer. This made the condition explicit:
That fork-specific GETDATA behavior is not the root cause being reported here.
The root cause is that the connection was permanently MNAUTH-verified on A but permanently unverified on B.
Without our diagnostic patch, the underlying asymmetric connection state still exists; the downstream handling of the DKG object may simply fail differently.
Evidence that the contribution itself was valid
A did not fail to participate.
It generated the contribution immediately upon entering the correct DKG phase.
The contribution hash existed locally on A.
A relayed the inventory according to its deterministic relay set.
Across the other quorum-node logs, only B observed this contribution hash.
That matches the deterministic topology exactly: the other two relay targets were unavailable, leaving B as the only live path.
Thus this was not:
It was a connection/authentication liveness failure.
Result inside the DKG
Other healthy quorum members did not receive A's contribution and therefore voted A bad during the complaint phase.
For example, complaints contained:
even though A was online and had successfully generated its contribution.
This means the issue can make a healthy masternode indistinguishable from a non-participating masternode at the DKG layer.
Depending on quorum size, topology and PoSe configuration, this could potentially:
Why sparse topology made the issue easy to reproduce
Our reproduction used the sparse deterministic connection topology.
That is not necessarily required for the underlying MNAUTH asymmetry.
The underlying problem is simply:
Sparse topology made the problem deterministic and visible because A had three relay targets and only one of them was live.
Therefore there was no alternate verified source from which the missing DKG object could be recovered.
With all-connected quorum topology, redundant announcements from other authenticated members might mask the DKG loss, but the one-way MNAUTH connection state can still exist.
Historical connection with existing Dash behavior
This appears related to assumptions from earlier Dash quorum networking work.
PR #3380 established deterministic connection direction so that only one member of a pair needs to initiate a connection while the other can wait for the inbound connection.
PR #2798 subsequently changed DKG/sig-share relay to use MNAUTHed connections, including inbound connections.
Those behaviors assume that after connection establishment the MNAUTH state will become usable on both sides.
The case above violates that assumption:
The current 5-second cleanup handles this if the unverified remote member is in the local node's required outbound connection set.
It appears not to handle the reverse-direction case where the remote member is expected to initiate toward this node.
Expected behavior
A connection used as an intra-quorum connection should eventually converge to one of two states:
A persistent state where:
should not survive indefinitely.
Possible fixes
Several approaches appear possible.
Option 1: retain/reprocess pre-sync MNAUTH
If a validly framed MNAUTH is received while:
retain one pending MNAUTH per connection and process it after synchronization completes.
Since MNAUTH is tied to the current connection challenge, this should only be retained for the lifetime of that connection.
Option 2: force a fresh handshake after synchronization
Record that MNAUTH was received while it could not yet be verified.
When masternode blockchain synchronization completes, disconnect such still-unverified connections.
The remote side can then reconnect and exchange MNAUTH while both sides are ready.
This may be simpler than retaining and replaying the message.
Option 3: expand stale non-MNAUTH quorum-peer cleanup
The existing 5-second cleanup could consider every deterministic peer with which the local node is expected to have an intra-quorum relationship in either direction, rather than only peers in the local node's outbound
masternodeQuorumNodesset.Conceptually, the cleanup set would include:
If such a member already has an ordinary connection that remains non-MNAUTH after the grace period, disconnect it.
In the reproduced case, B would then recognize A as a required reverse-direction quorum peer, drop the ordinary B→A connection, and A would subsequently establish the proper A→B quorum connection.
Suggested functional regression test
A focused functional test could reproduce the state without requiring a failed DKG first.
mn_sync.IsBlockchainSynced()is still false.slow_handshakethreshold.Before a fix, expected failure state:
After a fix, require convergence to either:
or:
An extended version of the test could start a DKG round and verify that a contribution originating on A is accepted by B over the resulting quorum connection.
Additional note
Our test environment deliberately included unavailable quorum members while exercising a protocol migration. That made the propagation consequence severe because the asymmetric A→B relationship became the only live route for A's contribution.
The unavailable members are not required for the underlying MNAUTH bug. They only removed relay redundancy and made the failure observable.
The condition worth investigating upstream is therefore:
System information
Ubuntu 24.04
Github 23.1.7 / 23.1.8 (self compiled)
Ubuntu 24.04 - Server class machine ( Poweredge T630 )
Attached are the two full debug logs (with debug=1) from our block 480 V19 cutover test with 12 v5 nodes (70240 version) and 5 of our older v3 nodes (70224 version)
These two nodes are version 70240.
debug007-107.log
debug109.log