Skip to content

503 mapping for DatabaseIsClosedException is broader than the resync race it fixes (#6770 follow-up) #6778

Description

@robfrank

Follow-up from the review loop on #6770 / PR #6776.

AbstractServerHttpHandler.sendMappedErrorResponse maps every DatabaseIsClosedException to a retryable 503 (added in #6776 to fix the HA snapshot-reinstall resync race, issue #6770). LocalDatabase.checkDatabaseIsOpen() throws that same exception type for any operation on a closed database, not only the transient resync race:

  • PostServerCommandHandler.dropDatabase() (non-HA branch) and closeDatabase() call database.getEmbedded().drop()/close() followed by server.removeDatabase() without holding databasesLock across the gap - unlike SnapshotInstaller.swapAndReopen, which deliberately brackets close→swap→reopen with that lock.
  • An in-flight request on the same database can hit DatabaseIsClosedException during a concurrent DROP DATABASE/close, which is a permanent close, not a transient one.
  • The client's automatic retry (RemoteHttpComponent, driven by the new 503) re-resolves the database with allowLoad=false (DatabaseAbstractHandler.execute). Once the registry entry is gone this throws DatabaseOperationException("Database '...' is not available"), which has no arm in sendMappedErrorResponse and falls through to the generic 500.

Net effect for a dropped/closed database raced with an in-flight request: client gets 503 ("retry me") -> retries -> generic 500, instead of an immediate, accurate 404/410. This is a narrow edge case (requires DROP/CLOSE racing an in-flight request on the same database) and not a regression severe enough to block #6776, but the mapping should eventually be scoped to the resync case specifically rather than the exception type alone.

Possible approaches (from the PR #6776 review, unverified in depth - pick after investigation):

  1. Scope the 503 mapping to the resync case specifically - e.g. check databasesLock/registry state, or a dedicated resync-in-progress marker, before mapping to 503. Note: ArcadeDBServer.isSnapshotInstallInProgress() exists but its usage at AbstractServerHttpHandler's request-entry check (line ~252) covers a different, whole-server snapshot-install condition unrelated to a single-database resync-reinstall (HA integration tests repeatedly trip a DatabaseIsClosedException race right after a snapshot-reinstall resync #5977 pattern) - reusing it as-is may not be semantically correct, needs verification.
  2. Handle DatabaseOperationException's "is not available" case explicitly in sendMappedErrorResponse (404/410) so a retry after a permanent close at least lands on an accurate status instead of the generic 500, even though the initial 503 would still be a wasted round trip.

Pointer comment added at the new 503 arm in AbstractServerHttpHandler.sendMappedErrorResponse (server module) referencing this issue.

Metadata

Metadata

Assignees

Labels

in progressThe team is actively working on this issue

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions