Skip to content

Config change paths do not flag an admin action when a bounded storage write gives up #962

Closed

Description

@maximthomas

Storage.write() is bounded on every pluggable engine that replays a rolled-back transaction:
JDBCStorage by MAX_RETRIES and MAX_RETRY_WINDOW_NANOS, and PDBStorage since #921. When the
bound is spent the write throws. Four configuration-change paths of the pluggable backend mutate
in-memory state around that write and none of them tells the operator that the two have diverged.

The four sites

  • EntryContainer.java:255 — attrIndexMap.remove(...) / attrCryptoMap.remove(...) run before
    the locked write, deliberately, so that a replay still finds the deletion to perform. On a spent
    bound the index is out of the map and its trees are still on disk.
  • EntryContainer.java:357 — the same shape for vlvIndexMap.remove(...).
  • AttributeIndex.java:940-942 — config, indexingOptions and indexIdToIndexes are swapped
    before the locked write at :948, and the third write at :965 never runs once it throws.
  • EntryContainer.java:2544-2552 — here the mutation is inside the WriteOperation:
    id2entry.setDataConfig(newDataConfig(cfg)) and EntryContainer.this.config = cfg. On a spent
    bound the loop at :2553-2556 (indexCrypto.newParameters(...)) is skipped, leaving id2entry on
    the new cipher while every attribute index's CryptoSuite still holds the old parameters.

What the operator sees

None of the four catches — EntryContainer:268, :371, :2558, AttributeIndex:977 — calls
ccr.setAdminActionRequired(true). Each reports the failure as
LocalizableMessage.raw(StaticUtils.stackTraceToSingleLineString(e)), a raw stack trace with no
message id, and sets only the server error result code. dsconfig prints that blob and never says
the backend now holds index trees no configuration points at, or an id2entry encrypted under
parameters its indexes do not share.

Ask

  1. Set ccr.setAdminActionRequired(true) on the four catches.
  2. Give the four sites a message id instead of a raw stack trace.
  3. Where it can be done without breaking the replay-idempotence the current ordering exists for,
    move the in-memory swap to after the write has been applied — AttributeIndex.applyConfigurationChange
    and EntryContainer.applyConfigurationChange are the two candidates.

Origin

Raised in review of #937 (bound the transaction replay of PDBStorage.write(), fixes #921). The
exposure pre-dates that PR: it already exists on a JDBC backend, which has been bounded all along,
and the "the write may be replayed, so the maps are updated outside of it" ordering came in with
0b9c0f6 (#867). #937 extends the exposure to PDB rather than creating it, which is why this is
tracked on its own.

Activity

  1. added a commit that references this issue on Sep 8, 2026
  2. self-assigned this
    on Sep 8, 2026
  3. added
    concurrencyThread-safety / race-condition bugs
    and removed on Sep 8, 2026
  4. added 2 commits that reference this issue on Sep 19, 2026
  5. added a commit that references this issue on Sep 19, 2026
  6. added a commit that references this issue on Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugconcurrencyThread-safety / race-condition bugs

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions