Failure
Seen on master CI (build-maven (ubuntu-latest, 11)): https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/29511617846/job/87666303869
org.junit.ComparisonFailure:
[In entry dn:
firstchangenumber: 1
lastchangenumber: 4
changelog: cn=changelog
lastExternalChangelogCookie: o=test:0000019f6b9b363604b100000004;
incorrect value for attr 'lastchangenumber']
expected: "8"
but was: "4"
at org.opends.server.backends.ChangelogBackendTestCase.assertAttributeValue(ChangelogBackendTestCase.java:1511)
at org.opends.server.backends.ChangelogBackendTestCase.assertChangelogAttributesInRootDSE(ChangelogBackendTestCase.java:941)
at org.opends.server.backends.ChangelogBackendTestCase.searchInChangeNumberModeOnOneSuffixMultipleTimes(ChangelogBackendTestCase.java:598)
The test publishes 4 changes, waits for them to be indexed, searches them, then publishes 4 more and waits for lastchangenumber to reach 8. It stays at 4 until the 30 s repeatUntilSuccess timer expires.
Root cause
The replica DB silently drops the entire second batch — the server log shows it for all 4 Multiple/5 records:
category=SYNC severity=INFORMATION msgID=296 msg=Filtering out from log file '.../1.dom/1201.server/head.log'
the record 'Record [0000019f6b9b363304b100000001:DeleteMsg ... dn: uid=Multiple/51,o=test ...]'
because it would break ordering. Last key appended is '0000019f6b9b363604b100000004'.
The second batch's CSNs are identical to the first batch's (...3633–...3636, seqnums 1–4), so the file-based changelog correctly refuses them as ≤ its last appended key, and change numbers 5–8 are never created.
The duplicate CSNs come from the test helper generateCSNs (ChangelogBackendTestCase.java:1253):
- it builds CSNs as
new CSN(TimeThread.getTime() + i, i + 1, serverId) — the seqnum restarts at 1 on every call;
TimeThread refreshes its cached clock only every 200 ms (TimeThread.java:437).
So whenever the first batch's publish → index → search cycle completes within a single TimeThread tick (fast CI runner), the second generateCSNs call gets the same start time and produces the exact same CSNs as the first call. Any advance of ≤ 3 ms is enough to lose part of the batch.
The test already guards against this for its 9th change (new CSN(lastCsn.getTime() + 1, 9, ...) at line 613), but not between the two 4-change batches.
Proposed fix
Make generateCSNs monotonic across calls: remember the last generated CSN time in an instance field and start at max(TimeThread.getTime(), lastTime + 1). This removes the flake for every caller in the test class without touching production code.
Failure
Seen on master CI (
build-maven (ubuntu-latest, 11)): https://github.com/OpenIdentityPlatform/OpenDJ/actions/runs/29511617846/job/87666303869The test publishes 4 changes, waits for them to be indexed, searches them, then publishes 4 more and waits for
lastchangenumberto reach 8. It stays at 4 until the 30 srepeatUntilSuccesstimer expires.Root cause
The replica DB silently drops the entire second batch — the server log shows it for all 4
Multiple/5records:The second batch's CSNs are identical to the first batch's (
...3633–...3636, seqnums 1–4), so the file-based changelog correctly refuses them as ≤ its last appended key, and change numbers 5–8 are never created.The duplicate CSNs come from the test helper
generateCSNs(ChangelogBackendTestCase.java:1253):new CSN(TimeThread.getTime() + i, i + 1, serverId)— the seqnum restarts at 1 on every call;TimeThreadrefreshes its cached clock only every 200 ms (TimeThread.java:437).So whenever the first batch's publish → index → search cycle completes within a single
TimeThreadtick (fast CI runner), the secondgenerateCSNscall gets the same start time and produces the exact same CSNs as the first call. Any advance of ≤ 3 ms is enough to lose part of the batch.The test already guards against this for its 9th change (
new CSN(lastCsn.getTime() + 1, 9, ...)at line 613), but not between the two 4-change batches.Proposed fix
Make
generateCSNsmonotonic across calls: remember the last generated CSN time in an instance field and start atmax(TimeThread.getTime(), lastTime + 1). This removes the flake for every caller in the test class without touching production code.