Skip to content

restore_backup 409s when a RocksDB database has another loaded alias — closeDatabase only closes handles under the restored name #2802

Description

@kriszyp

Summary

restore_backup can 409 on a RocksDB database that has no component holding it open — a second
database name (alias) pointing at the same on-disk directory is enough, because closeDatabase
only releases the RocksDB column-family handles registered under the name being restored, not
every alias of the same physical store.

Details

  • Two database names can resolve to the same on-disk RocksDB directory (normal scan
    resources/databases.ts:665, configured scan :734, path-keyed store reuse :1029 LMDB /
    :1061 RocksDB) — that's what "alias" means here.
  • closeDatabase(databaseName) (resources/databases.ts:2223) walks only databaseName's own
    tables' root stores. For LMDB it now also closes every other alias sharing the same environment
    (Close a shared LMDB environment once instead of closing its dbis after another alias freed it #2766's fix — the recursive closeDatabase(aliasName) call at the bottom of the function, and
    resources/DESIGN.md's "Closing an LMDB database closes its environment, never its dbis" note).
    RocksDB column families are deliberately left "closed one by one" (same DESIGN.md note) — so
    when the restore ITC handler (server/itc/serverHandlers.js:53,
    closeDatabase(event.message.schema)) closes only the name being restored, an unrestored alias's
    handles are untouched.
  • verifyDatabaseClosed (dataLayer/rocksdbBackup.ts:624) checks rocksdb-js's process-global
    registry by resolved path, not by name, so the surviving alias's still-open handle is exactly
    what it sees as "still open," and throws the 409 (dataLayer/rocksdbBackup.ts:636) after the 3s
    grace window — even though the operator correctly closed the name they're restoring.

Impact

Online restore_backup on a RocksDB database with a second configured/discovered alias 409s
regardless of what the operator does, until they also unload the other alias, or fall back to the
offline CLI the 409 message already points at. Bounded (loud failure, documented workaround),
narrow trigger (requires an alias configuration + online restore), but a real functional gap with
no online-only fix today.

Proposed fix

closeDatabase's RocksDB branch needs the same "close every alias of this root store" treatment
its LMDB branch got from #2766 — or the restore handler needs to resolve and close every alias
name sharing the target's path before calling closeDatabase. PR #2721 (closed, superseded by
#2766 for the LMDB crash) built a closeDatabaseWithAliases(name) helper for exactly this; its
awaited-runtime-stop half is tracked separately (#1811), but its alias-discovery approach is a
reasonable starting point — see branch fix/alias-close-shared-root-store at 840d14f26
(unmerged).

References

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:storageStorage engine, LMDB/RocksDB, compactionbugSomething isn't working

    Type

    Fields

    Priority

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions