Repository navigation
A ServerState update landing during a save is marked saved and never written to disk #916
Description
Activity
Walked the code for this. The diagnosis holds, the failure data pins the race window tighter than the description does, and one of the arguments in the description does not survive contact with the modify pipeline.
Confirmed against the code
- The published CSN reaches the shared
ServerStatesynchronously:processSchemaChange→synchronizeSchemaModifications(LDAPReplicationDomain.java:4485) →synchronize(:2116, publish at:2170) →PendingChanges.pushCommittedChanges(PendingChanges.java:166) →ReplicationDomain.publish→state.update(csn)(ReplicationDomain.java:3659-3666), andModifyMsgdoes contribute to the domain state (UpdateMsg.java:293). PersistentServerStatewraps the sameServerStateinstance the domain publishes into (LDAPReplicationDomain.java:695passesgetServerState()), so thesavedflag is shared between the publisher and the checkpointer.setSaved(true)has exactly one origin —save()(PersistentServerState.java:112). There is no other way for the flag to be lost, which rules out the alternative "something else marked it clean" explanations.- The
ds-sync-stateREPLACE does rewrite99-user.ldifinside the modify:SchemaBackend.java:577-593(non-schema attribute →extraAttributes+FILE_USER_SCHEMA_ELEMENTS) →SchemaHandler.updateSchemaAndSchemaFilesunderexclusiveLock(SchemaHandler.java:434-448).
The failure data pins the window to 401 ms
Decoding the CSNs in the dumped file, and the one being awaited:
CSN time what it is 000001a063a1509400010000000219:38:31.444 the local DELETEfrompushSchemaChange000001a063a1522600020000000119:38:31.846 the remote change replayed by replaySchemaChange(serverId 2)000001a063a153b700010000000319:38:32.247 the change awaited by pushSchemaFilesChangeA replayed CSN enters the state only in the post-operation commit (
RemotePendingChanges.java:238-240), i.e. after the replay itself has rewritten the schema files. So the file that carriesserverId 2, seq 1can only have been written by a state save whose snapshot was taken between 19:38:31.846 and 19:38:32.247 — and whosesetSaved(true)necessarily landed after 19:38:32.247, otherwise thestate.update()of that publish would have left the flag dirty and the next tick (≤ 1 s) would have written it. That is exactly the interleaving described, with the snapshot-to-completion window bounded by 401 ms.Why the write was slow enough to straddle the publish: the checkpointer's modify waits for
SchemaHandler'sexclusiveLock(SchemaHandler.java:437), which the concurrent replay was holding while rewriting the schema files. As a second-order effect,state.save()runs insidesynchronized (this)next towait(1000)(LDAPReplicationDomain.java:558-568), so a slow save also stretches the following tick.Correction: the
modifyTimestampargument does not holdits
modifyTimestamp— 19:38:31 — predates the start of the test method … Over the whole 10 s window the file was not rewritten oncemodifyTimestampis put into the entry byLastModPlugin(LastModPlugin.java:179-186), and pre-operation plugins are skipped for synchronization operations (LocalBackendModifyOperation.java:323-334) — which is whatrunUpdateStateEntrymarks its modify as (PersistentServerState.java:263). Neither a state save nor a replay refreshes that attribute; the value in the file is left over from the last client modify (theDELETEat 19:38:31.444). The file was rewritten inside the window — it could not otherwise containserverId 2, seq 1— it just kept the stale timestamp. The conclusion is unchanged: the file's contents prove the lost flag more strongly than its timestamp ever could.The suggested patch needs a companion change
Clearing the flag before the snapshot is not sufficient on its own, because
ServerState.update()publishessaved = falsebefore mutating the map (ServerState.java:85, with theputIfAbsent/replaceat:93/:102). The interleavingsaved = false→ [saver:setSaved(true), snapshot] →putstill loses the CSN — a narrower window, same bug.The dirty marker has to be published after the mutation it advertises. With both halves in place the invariant holds for every interleaving: if the
putprecedes the snapshot the CSN is in the write; if it follows the snapshot then itssaved = falsenecessarily followssetSaved(true), so the next tick writes it. The cost is at most one redundant write per race. Moving the assignment also stops a non-meaningfulupdate()(an older or duplicate CSN) from marking the state dirty. A version counter buys nothing extra here — it needs the same "increment after the mutation" ordering.One more point, independent of the race above:
save()has no mutual exclusion while being called from three different threads —ServerStateFlush(LDAPReplicationDomain.java:566,:576),disable()(:3871) andbackupStart()(:4113). Two overlapping saves can complete out of order, leaving the older snapshot on disk and the flag clean.Regression test
Deterministic and thread-free: extract the modify execution of
runUpdateStateEntrybehind a package-private seam, and inPersistentServerStateTest(same package, already runs against a started server) override it so that astate.update(lateCSN)lands inside the write — i.e. after the snapshot has been taken. Thensave()must leave the state dirty (fails on current code), and the followingsave()must putlateCSNon disk, verified by reloading through a freshPersistentServerState.SchemaReplicationTest.pushSchemaFilesChangeneeds no change: with the fix the worst case is a write one tick later, well inside its 10 s budget.- The published CSN reaches the shared
- added a commit that references this issue
on Sep 10, 2026
Symptom
SchemaReplicationTest.pushSchemaFilesChangefails intermittently in CI. Seen on build-maven (ubuntu-latest, 21) of run 33668788856 (PR #893, whose change touches only the JDBC backend); the other eleven jobs of the same run, running the same 32152 tests, passed.The test published a schema change, received the
ModifyMsgback from the broker, and then waited 10 s for the CSN of that change to reach99-user.ldif(SchemaReplicationTest.java:229-241). It never did: the file still carriedserverId 1, seq 2whileserverId 1, seq 3was awaited, and itsmodifyTimestamp— 19:38:31 — predates the start of the test method (19:38:32.3). Over the whole 10 s window the file was not rewritten once, although the checkpointer ticks every second.What the run rules out
"Replica DS(1) state checkpointer for domain "cn=schema"" daemon Id=94 TIMED_WAITING on LDAPReplicationDomain$ServerStateFlush.runUpdateStateEntry()logsDEBUG_ERROR_UPDATING_RUVon any result code other than success, and the server error log is silent between 19:38:32 and the timer giving up at 19:38:42.ReplicationDomain.publish()callsstate.update(csn)synchronously (ReplicationDomain.java:3659-3667), and the test had already received the message that publish sent.REPLACEofds-sync-statedoes rewrite the file synchronously inside the modify —SchemaBackendputs a non-schema attribute intoextraAttributesand addsFILE_USER_SCHEMA_ELEMENTStomodifiedSchemaFiles(SchemaBackend.java:577-592).So the modify that would have written the awaited CSN was never issued.
The race
PersistentServerState.save()decides whether to write from a flag that the write itself clears:updateStateEntry()serialises the state once, at the top ofrunUpdateStateEntry()—new LDAPAttribute(REPLICATION_STATE, state.toASN1ArrayList())— and only then runs the modify, which rewrites99-user.ldifand takes hundreds of milliseconds.ServerState.update()setssaved = falseunconditionally (ServerState.java:78-85).A
publish()landing after the snapshot is taken and beforesetSaved(true)is therefore lost: its CSN is not in the attribute being written, and thesaved = falseit set is overwritten by thesetSaved(true)of the write that did not carry it. Nothing sets the flag again, so the state is not written until the next meaningful update on that domain.On a busy domain the next update arrives shortly and the loss heals itself, which is why this is not seen elsewhere.
cn=schemais the domain where changes are rare, so a lost flag can leave the persisted state stale indefinitely — and that is exactly what the test observes: one save at 19:38:31, and then silence.Impact beyond the test
ds-sync-stateon disk can lag the replica's real state by an unbounded amount, until some later change happens to arrive. After a crash or restart the domain resumes from that stale point, and for the schema domain the window can be the whole life of the server. It is the safe direction (changes are replayed rather than skipped), but it defeats the checkpointing the flush thread exists to provide, and it makes any test — or operator — that reads the persisted state see something arbitrarily old.Suggested fix
Clear the flag before the snapshot rather than after the write, so that an update racing with the write leaves the flag set for the next tick:
The cost is at most one redundant write per race. A version counter compared across the write would achieve the same without that redundancy.
A regression test can drive it directly: hold the modify (or simply publish a change while a save is in flight) and assert that the following tick writes the newer CSN.