Fix chained replica crash when doing dual channel replication#2983
Merged
enjoy-binbin merged 3 commits intovalkey-io:unstablefrom Dec 29, 2025
Merged
Fix chained replica crash when doing dual channel replication#2983enjoy-binbin merged 3 commits intovalkey-io:unstablefrom
enjoy-binbin merged 3 commits intovalkey-io:unstablefrom
Conversation
There is a crash in freeReplicationBacklog:
```
Discarding previously cached primary state.
ASSERTION FAILED
'listLength(server.replicas) == 0' is not true
freeReplicationBacklog
```
The reason is that during dual channel operation, the RDB channel is protected.
In the chained replica case, `disconnectReplicas` is called to disconnect all
replica clients, but since the RDB channel is protected, `freeClient` does not
actually free the replica client. Later, we encounter an assertion failure in
`freeReplicationBacklog`.
```
void replicationAttachToNewPrimary(void) {
/* Replica starts to apply data from new primary, we must discard the cached
* primary structure. */
serverAssert(server.primary == NULL);
replicationDiscardCachedPrimary();
/* Cancel any in progress imports (we will now use the primary's) */
clusterCleanSlotImportsOnFullSync();
disconnectReplicas(); /* Force our replicas to resync with us as well. */
freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */
}
```
Dual channel replication was introduced in valkey-io#60.
Signed-off-by: Binbin <binloveplay1314@qq.com>
enjoy-binbin
commented
Dec 26, 2025
Member
Author
|
The crash report: |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## unstable #2983 +/- ##
============================================
+ Coverage 73.79% 73.91% +0.12%
============================================
Files 125 125
Lines 69345 69349 +4
============================================
+ Hits 51173 51260 +87
+ Misses 18172 18089 -83
🚀 New features to boost your workflow:
|
zuiderkwast
approved these changes
Dec 27, 2025
Contributor
zuiderkwast
left a comment
zuiderkwast
left a comment
There was a problem hiding this comment.
The fix looks safe to me.
Signed-off-by: Binbin <binloveplay1314@qq.com>
naglera
approved these changes
Dec 28, 2025
Contributor
naglera
left a comment
naglera
left a comment
There was a problem hiding this comment.
LGTM thanks for the fix!
Signed-off-by: Binbin <binloveplay1314@qq.com>
enjoy-binbin
added
the
run-extra-tests
github-actions
bot
removed
the
run-extra-tests
enjoy-binbin
added
bug
jdheyburn
pushed a commit
to jdheyburn/valkey
that referenced
this pull request
Jan 8, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com>
zuiderkwast
pushed a commit
to zuiderkwast/placeholderkv
that referenced
this pull request
Jan 29, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com>
roshkhatri
pushed a commit
to roshkhatri/valkey
that referenced
this pull request
Jan 29, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com>
roshkhatri
pushed a commit
to roshkhatri/valkey
that referenced
this pull request
Jan 29, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
roshkhatri
pushed a commit
to roshkhatri/valkey
that referenced
this pull request
Jan 29, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
roshkhatri
pushed a commit
to roshkhatri/valkey
that referenced
this pull request
Jan 29, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com>
roshkhatri
pushed a commit
to roshkhatri/valkey
that referenced
this pull request
Jan 29, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
roshkhatri
pushed a commit
to roshkhatri/valkey
that referenced
this pull request
Jan 30, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
zuiderkwast
pushed a commit
to zuiderkwast/placeholderkv
that referenced
this pull request
Jan 30, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com>
zuiderkwast
pushed a commit
that referenced
this pull request
Feb 3, 2026
There is a crash in freeReplicationBacklog:
```
Discarding previously cached primary state.
ASSERTION FAILED
'listLength(server.replicas) == 0' is not true
freeReplicationBacklog
```
The reason is that during dual channel operation, the RDB channel is protected.
In the chained replica case, `disconnectReplicas` is called to disconnect all
replica clients, but since the RDB channel is protected, `freeClient` does not
actually free the replica client. Later, we encounter an assertion failure in
`freeReplicationBacklog`.
```
void replicationAttachToNewPrimary(void) {
/* Replica starts to apply data from new primary, we must discard the cached
* primary structure. */
serverAssert(server.primary == NULL);
replicationDiscardCachedPrimary();
/* Cancel any in progress imports (we will now use the primary's) */
clusterCleanSlotImportsOnFullSync();
disconnectReplicas(); /* Force our replicas to resync with us as well. */
freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */
}
```
Dual channel replication was introduced in #60.
Signed-off-by: Binbin <binloveplay1314@qq.com>
roshkhatri
pushed a commit
to roshkhatri/valkey
that referenced
this pull request
Feb 4, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
roshkhatri
pushed a commit
to roshkhatri/valkey
that referenced
this pull request
Feb 18, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
roshkhatri
pushed a commit
to roshkhatri/valkey
that referenced
this pull request
Feb 20, 2026
…-io#2983) There is a crash in freeReplicationBacklog: ``` Discarding previously cached primary state. ASSERTION FAILED 'listLength(server.replicas) == 0' is not true freeReplicationBacklog ``` The reason is that during dual channel operation, the RDB channel is protected. In the chained replica case, `disconnectReplicas` is called to disconnect all replica clients, but since the RDB channel is protected, `freeClient` does not actually free the replica client. Later, we encounter an assertion failure in `freeReplicationBacklog`. ``` void replicationAttachToNewPrimary(void) { /* Replica starts to apply data from new primary, we must discard the cached * primary structure. */ serverAssert(server.primary == NULL); replicationDiscardCachedPrimary(); /* Cancel any in progress imports (we will now use the primary's) */ clusterCleanSlotImportsOnFullSync(); disconnectReplicas(); /* Force our replicas to resync with us as well. */ freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */ } ``` Dual channel replication was introduced in valkey-io#60. Signed-off-by: Binbin <binloveplay1314@qq.com> Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
madolson
pushed a commit
that referenced
this pull request
Feb 24, 2026
There is a crash in freeReplicationBacklog:
```
Discarding previously cached primary state.
ASSERTION FAILED
'listLength(server.replicas) == 0' is not true
freeReplicationBacklog
```
The reason is that during dual channel operation, the RDB channel is protected.
In the chained replica case, `disconnectReplicas` is called to disconnect all
replica clients, but since the RDB channel is protected, `freeClient` does not
actually free the replica client. Later, we encounter an assertion failure in
`freeReplicationBacklog`.
```
void replicationAttachToNewPrimary(void) {
/* Replica starts to apply data from new primary, we must discard the cached
* primary structure. */
serverAssert(server.primary == NULL);
replicationDiscardCachedPrimary();
/* Cancel any in progress imports (we will now use the primary's) */
clusterCleanSlotImportsOnFullSync();
disconnectReplicas(); /* Force our replicas to resync with us as well. */
freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */
}
```
Dual channel replication was introduced in #60.
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
madolson
pushed a commit
that referenced
this pull request
Feb 24, 2026
There is a crash in freeReplicationBacklog:
```
Discarding previously cached primary state.
ASSERTION FAILED
'listLength(server.replicas) == 0' is not true
freeReplicationBacklog
```
The reason is that during dual channel operation, the RDB channel is protected.
In the chained replica case, `disconnectReplicas` is called to disconnect all
replica clients, but since the RDB channel is protected, `freeClient` does not
actually free the replica client. Later, we encounter an assertion failure in
`freeReplicationBacklog`.
```
void replicationAttachToNewPrimary(void) {
/* Replica starts to apply data from new primary, we must discard the cached
* primary structure. */
serverAssert(server.primary == NULL);
replicationDiscardCachedPrimary();
/* Cancel any in progress imports (we will now use the primary's) */
clusterCleanSlotImportsOnFullSync();
disconnectReplicas(); /* Force our replicas to resync with us as well. */
freeReplicationBacklog(); /* Don't allow our chained replicas to PSYNC. */
}
```
Dual channel replication was introduced in #60.
Signed-off-by: Binbin <binloveplay1314@qq.com>
Signed-off-by: Roshan Khatri <rvkhatri@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
There is a crash in freeReplicationBacklog:
The reason is that during dual channel operation, the RDB channel is protected.
In the chained replica case,
disconnectReplicasis called to disconnect allreplica clients, but since the RDB channel is protected,
freeClientdoes notactually free the replica client. Later, we encounter an assertion failure in
freeReplicationBacklog.Dual channel replication was introduced in #60.