You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
restore_backup 409s when a RocksDB database has another loaded alias — closeDatabase only closes handles under the restored name #2802
restore_backup can 409 on a RocksDB database that has no component holding it open — a second
database name (alias) pointing at the same on-disk directory is enough, because closeDatabase
only releases the RocksDB column-family handles registered under the name being restored, not
every alias of the same physical store.
Details
Two database names can resolve to the same on-disk RocksDB directory (normal scan resources/databases.ts:665, configured scan :734, path-keyed store reuse :1029 LMDB / :1061 RocksDB) — that's what "alias" means here.
closeDatabase(databaseName) (resources/databases.ts:2223) walks only databaseName's own
tables' root stores. For LMDB it now also closes every other alias sharing the same environment
(Close a shared LMDB environment once instead of closing its dbis after another alias freed it #2766's fix — the recursive closeDatabase(aliasName) call at the bottom of the function, and resources/DESIGN.md's "Closing an LMDB database closes its environment, never its dbis" note).
RocksDB column families are deliberately left "closed one by one" (same DESIGN.md note) — so
when the restore ITC handler (server/itc/serverHandlers.js:53, closeDatabase(event.message.schema)) closes only the name being restored, an unrestored alias's
handles are untouched.
verifyDatabaseClosed (dataLayer/rocksdbBackup.ts:624) checks rocksdb-js's process-global
registry by resolved path, not by name, so the surviving alias's still-open handle is exactly
what it sees as "still open," and throws the 409 (dataLayer/rocksdbBackup.ts:636) after the 3s
grace window — even though the operator correctly closed the name they're restoring.
Impact
Online restore_backup on a RocksDB database with a second configured/discovered alias 409s
regardless of what the operator does, until they also unload the other alias, or fall back to the
offline CLI the 409 message already points at. Bounded (loud failure, documented workaround),
narrow trigger (requires an alias configuration + online restore), but a real functional gap with
no online-only fix today.
Proposed fix
closeDatabase's RocksDB branch needs the same "close every alias of this root store" treatment
its LMDB branch got from #2766 — or the restore handler needs to resolve and close every alias
name sharing the target's path before calling closeDatabase. PR #2721 (closed, superseded by #2766 for the LMDB crash) built a closeDatabaseWithAliases(name) helper for exactly this; its
awaited-runtime-stop half is tracked separately (#1811), but its alias-discovery approach is a
reasonable starting point — see branch fix/alias-close-shared-root-store at 840d14f26
(unmerged).
Summary
restore_backupcan 409 on a RocksDB database that has no component holding it open — a seconddatabase name (alias) pointing at the same on-disk directory is enough, because
closeDatabaseonly releases the RocksDB column-family handles registered under the name being restored, not
every alias of the same physical store.
Details
resources/databases.ts:665, configured scan:734, path-keyed store reuse:1029LMDB /:1061RocksDB) — that's what "alias" means here.closeDatabase(databaseName)(resources/databases.ts:2223) walks onlydatabaseName's owntables' root stores. For LMDB it now also closes every other alias sharing the same environment
(Close a shared LMDB environment once instead of closing its dbis after another alias freed it #2766's fix — the recursive
closeDatabase(aliasName)call at the bottom of the function, andresources/DESIGN.md's "Closing an LMDB database closes its environment, never its dbis" note).RocksDB column families are deliberately left "closed one by one" (same DESIGN.md note) — so
when the restore ITC handler (
server/itc/serverHandlers.js:53,closeDatabase(event.message.schema)) closes only the name being restored, an unrestored alias'shandles are untouched.
verifyDatabaseClosed(dataLayer/rocksdbBackup.ts:624) checks rocksdb-js's process-globalregistry by resolved path, not by name, so the surviving alias's still-open handle is exactly
what it sees as "still open," and throws the 409 (
dataLayer/rocksdbBackup.ts:636) after the 3sgrace window — even though the operator correctly closed the name they're restoring.
Impact
Online
restore_backupon a RocksDB database with a second configured/discovered alias 409sregardless of what the operator does, until they also unload the other alias, or fall back to the
offline CLI the 409 message already points at. Bounded (loud failure, documented workaround),
narrow trigger (requires an alias configuration + online restore), but a real functional gap with
no online-only fix today.
Proposed fix
closeDatabase's RocksDB branch needs the same "close every alias of this root store" treatmentits LMDB branch got from #2766 — or the restore handler needs to resolve and close every alias
name sharing the target's path before calling
closeDatabase. PR #2721 (closed, superseded by#2766 for the LMDB crash) built a
closeDatabaseWithAliases(name)helper for exactly this; itsawaited-runtime-stop half is tracked separately (#1811), but its alias-discovery approach is a
reasonable starting point — see branch
fix/alias-close-shared-root-storeat840d14f26(unmerged).
References
resources/databases.ts:2223—closeDatabaseserver/itc/serverHandlers.js:53— restore ITC handlerdataLayer/rocksdbBackup.ts:624—verifyDatabaseClosedresources/DESIGN.md— "Closing an LMDB database closes its environment, never its dbis"still open); Close a shared LMDB environment once instead of closing its dbis after another alias freed it #2766 (merged LMDB fix; this issue is the RocksDB-side gap it left open by design)