Skip to content

[#949] Report a ReplicaOfflineMsg the broker refused as not sent - #976

Merged
vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:issues/949-report-a-refused-replica-offline-msg
Sep 11, 2026
Merged

vharseko merged 1 commit into
OpenIdentityPlatform:masterfrom
vharseko:issues/949-report-a-refused-replica-offline-msg

Conversation

@vharseko

@vharseko vharseko commented Sep 9, 2026 •

Copy link
Copy Markdown
Member

Fixes #949

ReplicationDomain.publish() discarded the outcome of broker.publish(), so a
ReplicaOfflineMsg the broker refused was still recorded as sent:

public void publish(UpdateMsg msg)
{
  broker.publish(msg);
  ...
}

The path

PendingChanges.pushCommittedChanges() publishes the announcement through the domain, and the
void overload of ReplicationBroker.publish() returns without touching any session when the
replica has no usable session (connectionError) or when the changes which come before this one
still have to be republished by the recovery (connectRequiresRecovery). Its publish loop also
ends when the broker is stopped in between, and it reported that message as published too.

The connectionError path is the reachable one: a replica whose replication server went away
first announces itself offline into a broken session.

What it costs

The same as #918: DSRSShutdownSync holds an announcement which is not on the wire, and
ReplicationServer.shutdown() spends the whole grace period in
awaitReplicaOfflineMsgsForwarded() waiting for it to be forwarded. The announcement itself is
lost as well, so notifyReplicaOffline() never reaches the changelog of the peer RSs and their
medium consistency point keeps waiting for changes from a replica which is gone - which is what
OPENDJ-1453 added DSRSShutdownSync for.

The change

ReplicationBroker.publish() returns done rather than true: its loop also ends when the
broker is stopped, and then nothing was written to any session. Four callers already act on that
answer - the export of a total update, and the two initialization requests #861 made check it -
and they now fail on a broker stopped mid-flight instead of waiting for an answer which cannot
come.

ReplicationDomain.publish() reports whether the broker wrote the message. Its other effects stay
unconditional on purpose: the domain state says which changes the replica has done, and it is by
finding it ahead of the state its replication server reports that the next session knows which
changes to republish from the historical information of their entries. A change the broker refuses
is recovered that way - the offline announcement is stored nowhere, so it is the one message whose
delivery the caller must know about.

PendingChanges.pushCommittedChanges() therefore reports the CSN of the offline message the
replication service accepted, and putReplicaOfflineMsg() returns it only when it is the message
it has just queued - the message carries the newest CSN of the replica, so anything else means a
change before it held it back. Either way the message leaves the queue, which is what #946 made
putReplicaOfflineMsg() do: there is nobody left to publish it afterwards, and it must not
surface on the session which follows. publishReplicaOfflineMsg() records only what was really
sent, and traces the case where the replica could not announce itself: an operator looking at a
peer RS which never got the offline CSN otherwise has nothing to go on.

Rebased on #946

#946 has landed (bb6e7c5816), and this branch is rebased on it - the "whichever lands first"
of the overlap this PR described. The two verdicts became one: the answer now comes from
pushCommittedChanges(), so the message the broker refused is reported as not sent as well as
the message a change in flight held back, and the unconditional pendingChanges.remove() of #946
still drops what was not published. A message which could not be published is both dropped and
reported as not sent, and the trace of publishReplicaOfflineMsg() names both reasons. The test
class is the union of the two.

Not fixed here

Tests

PendingChangesTest, the class #946 added, grows to five cases: the announcement the broker
published is reported as sent, the one it refused is not, the one a change in flight holds back is
not even attempted, a message which could not be sent is not published later on the session which
follows, and a change the broker refused still leaves the pending changes - it is the recovery,
not the queue, which publishes it again.

@vharseko vharseko added bug replication java tests Test suites: fixing, enabling, un-disabling labels Sep 9, 2026
@vharseko
vharseko requested a review from maximthomas September 9, 2026 05:46
@vharseko vharseko removed the java label Sep 9, 2026
…sed as not sent

ReplicationDomain.publish() discarded the outcome of broker.publish(), so an
announcement the broker never wrote to a session - it has no usable session, the
recovery still has to republish the changes which come before it, or it was
stopped in between - was still recorded as sent, and the shutdown of a collocated
replication server then spent its whole grace period waiting for it to be
forwarded.

ReplicationBroker.publish() now reports whether the message was written rather
than true when its loop ends on the shutdown, ReplicationDomain.publish() passes
that answer on, and PendingChanges reports the CSN of the offline message the
replication service accepted, so publishReplicaOfflineMsg() announces only what
really was sent.
@vharseko
vharseko force-pushed the issues/949-report-a-refused-replica-offline-msg branch from 1288ead to a2dd8e4 Compare September 9, 2026 08:51
@vharseko

vharseko commented Sep 9, 2026

Copy link
Copy Markdown
Member Author

Rebased on master, where #946 has landed - the "whichever lands first" of the overlap this PR
described. The two changes touch the same lines of putReplicaOfflineMsg(),
publishReplicaOfflineMsg() and the same test class, and the conflict is resolved as their union:

Checked on the rebased branch with -Pprecommit verify: PendingChangesTest 5/5, and
DSRSShutdownSyncTest, ReplicationServerShutdownSyncTest, ReplicaCursorTest 25/25.

The description is updated to record the result rather than the plan. One stale reference went
with it: the last bullet of "Not fixed here" pointed the recovery-guard question at #949, which is
the issue this PR fixes; the question stands on its own without the number.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug replication tests Test suites: fixing, enabling, un-disabling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A ReplicaOfflineMsg the broker refuses is still recorded as sent: ReplicationDomain.publish() discards the outcome

2 participants