dsreplication enable hangs initializing registration information when adding a third replica; CI Test replication step has no timeout #861

Activity

1. vharseko

   added

   CI

   replication

   bug

   concurrencyThread-safety / race-condition bugsThread-safety / race-condition bugs

   on Aug 12, 2026
2. vharseko commented on Aug 12, 2026

   

   vharseko

   on Aug 12, 2026

   MemberAuthor

   More actions

   Root-cause analysis

   How this phase works

   1. CLI (ReplicationCliMain.initializeSuffix) creates a ds-task-initialize-from-remote-replica task on the destination server and polls its ds-task-state every 500 ms with no timeout (Installer.initializeSuffix, the while (!isOver) loop).
   2. The task on the destination (InitializeTask.runTask) calls initializeFromRemote and then waits in while (initState == RUNNING) wait(1000) — also without a timeout. The final state is only set once InitializeTargetMsg + entries + DoneMsg have been processed, or once an ErrorMsg is accepted.
   3. ReplicationDomain.initializeFromRemote publishes an InitializeRequestMsg to the broker and returns immediately. From then on the fate of the task depends entirely on a response arriving.

   The window "request published → InitializeTargetMsg not yet received" has no protection at all, and there are two real holes in it. (The import phase itself is protected: broker disconnection and exporter disappearance are detected in receiveEntryBytes — which is why the hang always happens at the start, never mid-transfer.)

   Hole 1: the request can be lost silently

   ReplicationBroker.publish() (the void variant) silently returns false when connectionError is set or when connectRequiresRecovery is true after a reconnect, and initializeFromRemote ignores the return value. The "resend procedure of the connect phase" that the comment relies upon only replays UpdateMsgs — an InitializeRequestMsg is simply gone. The re-send in ReplicationDomain.initialize() (attemptCnt < 2) only fires after a failed import, i.e. after the request has already reached the source.

   The CI timing sits exactly in this window: at 17:33:22 dsreplication had just re-configured the cn=admin data domain on all three servers ("Updating registration configuration … Done") and created the task in the same instant — the domain brokers are restarting their RS sessions at that very moment, and connectRequiresRecovery silently swallows non-recovery messages after a reconnect.

   Hole 2: a legitimate ErrorMsg is discarded by a millisecond-granularity filter

   Even when the topology does answer with an error (RS finds no route → replyWithUnreachablePeerMsg; the source refuses the export → the ErrorMsg added by #841 — that fix was in the failing build, verified by commit ancestry), the requester only accepts it when errorMsg.getCreationTime() > ieCtx.startTime (two call sites: the listener loop and receiveEntryBytes). ErrorMsg carries no correlation id, so this timestamp comparison is the only staleness filter. In CI all three servers run on the same host: the full round-trip "request → rejection → ErrorMsg" easily fits inside one millisecond, the strict > fails, the error is logged as "relates to a previous attempt" — and the task waits forever.

   Observed consequence

   The task stays RUNNING forever, the CLI polls it every 500 ms forever, and the job dies only at the GitHub Actions 6-hour default limit (mitigated by #862). The orphan-process list of the cancelled job (3 × java + dsreplication + _server-script.sh) matches this state exactly.

   Planned fixes

   1. Do not lose the request: in initializeFromRemote, use the result-returning publish(msg, retryOnFailure) variant and fail the task immediately when the request could not be sent — the CLI already has a 5-attempt retry loop for exactly this kind of failure.
   2. Start watchdog: if neither InitializeTargetMsg nor ErrorMsg arrives within a bounded time after the request was published, fail the task instead of waiting forever, turning the infinite hang into a retryable error.
   3. ErrorMsg gate: accept errors created in the same millisecond as the context (>= instead of >).
3. vharseko commented on Aug 12, 2026

   

   vharseko

   on Aug 12, 2026

   MemberAuthor

   More actions

   Addendum: a fourth bug uncovered while writing regression tests

   The regression tests for the two holes above kept failing in an unexpected way — the initialize task completed successfully instead of failing — which exposed one more, and arguably the worst, bug in this code path:

   // ReplicationDomain.ImportExportContext
   public void setExceptionIfNoneSet(DirectoryException exception)
   {
     if (exception == null)        // checks the ARGUMENT instead of the field
     {
       this.exception = exception; // i.e. only ever assigns null
     }
   }

   setExceptionIfNoneSet never records anything: the null check was written against the argument instead of the field. The bug is inherited from upstream — the same code is present in the "Recreate OpenDJ 3 development branch" commit (2014).

   Consequences, since every error path of the total update funnels through this method:

   • processErrorMsg — an ErrorMsg that does pass the timestamp gate sets no exception, so updateTaskCompletionState(ieCtx.getException()) is called with null and the task ends COMPLETED_SUCCESSFULLY even though nothing was initialized.
   • LDAPReplicationDomain.importBackend — a failed LDIF import records no error: the generation id is not recomputed despite partial data in the backend, importBackend does not throw, and the initialize task again reports success.
   • receiveEntryBytes — losing the RS connection or receiving out-of-order entry messages mid-import is likewise swallowed.

   In other words: whenever a total update failed with a properly delivered error, dsreplication initialize (and the ADS initialization inside dsreplication enable) reported success over partially imported data.

   The fix is a one-liner (if (this.exception == null)), included in the fix PR together with the request-loss, timestamp-gate and watchdog changes; the new ReplicationDomainTest regression tests cover the error-propagation end to end.
4. vharseko

   added

   data-lossData integrity / loss of entriesData integrity / loss of entries

   on Aug 12, 2026
5. vharseko

   added 3 commits that reference this issue on Aug 12, 2026

   [#861] Fail fast when the CI Test replication step hangs (#862)

   207c349

   [OpenIdentityPlatform#861] Keep the import/export context non-null th…

   ...

   88b14db

   [#861] Fail fast when a total update request gets no answer (#864)

   641ff2c
6. vharseko

   linked a pull request that will close this issue[#861] Fail fast when a total update request gets no answer #864on Aug 13, 2026
7. vharseko

   closed this as completedon Aug 13, 2026