An exception from a state write kills the checkpointer and hangs server shutdown #952

Symptom

If writing the replication state ever throws, the domain's state checkpointer thread dies silently and server shutdown hangs forever. The domain also stops checkpointing for the rest of the server's life, with nothing in the log to say so.

Found while working on #916; it is a separate robustness gap, unchanged by that fix.

The mechanism

ServerStateFlush.run() guards only against interruption:

// LDAPReplicationDomain.java:554-579
public void run()
{
  done = false;
  while (!isShutdownInitiated())
  {
    try
    {
      synchronized (this)
      {
        wait(1000);
        if (!disabled && !ieRunning())
        {
          state.save();
        }
      }
    }
    catch (InterruptedException e)
    {
      // Thread interrupted: check for shutdown.
      Thread.currentThread().interrupt();
    }
  }
  state.save();

  done = true;
}

state.save() runs a modify against the backend (PersistentServerState.runUpdateStateEntry()); ModifyOperationBasis.run() catches only CanceledOperationException, so any RuntimeException from the backend, a workflow element or a plugin propagates out of save(). Nothing in the loop catches it, so the thread terminates with done still false (it is set at :555 and only reaches true at :578).

shutdown() then waits for that flag with no bound:

// LDAPReplicationDomain.java:2393-2399
// wait for completion of the ServerStateFlush thread.
try
{
  while (!done)
  {
    Thread.sleep(50);
  }
}

so shutting the server down never returns. The same throw also silently ends checkpointing for the domain: nothing restarts the thread, and the only trace is whatever the default uncaught-exception handler prints.

Impact

One unexpected exception on the state-write path — a backend in an error state, a plugin failing on an internal operation, an OutOfMemoryError in the entry lock manager — turns into (a) a domain whose ds-sync-state is never written again, so a later crash replays from an arbitrarily old point, and (b) a server that cannot be shut down other than by killing it.

Suggested direction

• Catch RuntimeException (and arguably Throwable) around the state.save() calls in run(), log it, and keep the loop going: a failed checkpoint is not a reason to stop checkpointing, and the state stays marked as unsaved so the next tick retries it.
• Set done = true from a finally around the whole body, so the thread cannot exit without releasing shutdown().
• Give shutdown()'s while (!done) a deadline, and log if the flush thread does not finish within it — hanging without a word is the worst of the options available at that point.