Document the noms LOCK contract (<data_dir>/<db>/.dolt/noms/LOCK): acquire timeout, lifetime/staleness, external-waiter flock safety

[sjarmak]

Background — an orchestrator's need. We manage the dolt sql-server lifecycle (start/stop/restart) from a supervisor that also coordinates other processes against the same data dir. To restart safely we need to know when the previous server has actually released its hold on a database and when a new server can take it — without guessing. Today we wait on the noms LOCK file with an arbitrary timeout (60s) because the lock's contract isn't documented anywhere.

What's unclear today (and what the code actually does). From reading go/store/nbs:

• LOCK is an advisory fslock at <data_dir>/<db>/.dolt/noms/LOCK (journal.go:576; lockFileName, file_manifest.go:45), held from DB open (newJournalManifest) until Close() — i.e. for the entire server lifetime, not a short critical section.
• On acquire, dolt waits only lockFileTimeout = 100ms (file_manifest.go:46); on timeout it either errors with ErrDatabaseLocked or silently falls back to read-only. This appears to be the root of the recurring "database is read only" / "locked by another process" reports (CLI writes fail with 'database is read only' due to stale journal lock after process exit #10852, Dolt Fails to Start After Restart in Kubernetes STS with ProxySQL – Reports Database Locked by Another Process #11036, Feature request: The error message from Dolt when failing a write because the database is read-only should be better. #10858).
• It's unspecified whether the lock survives a hard crash as a stale lock, how an operator clears it, and whether an external process may itself flock() the file to coordinate.

Proposal. Document the LOCK contract in the dolt docs (and/or a noms/LOCK README), covering: (a) it is an advisory flock held for the server's lifetime — there is no "max hold time," the correct wait is "until the holding process exits / closes"; (b) dolt's own 100ms acquire timeout and the read-only fallback, so operators understand why a too-fast restart yields a read-only or "locked" server; (c) crash/staleness behavior and the safe way to clear a stale lock; (d) explicitly, whether an external supervisor may flock() these files to coordinate hand-off, or whether that risks corrupting dolt's own locking.

Design options (≥2):

1. Docs-only. A "Storage locking" page in dolthub/docs describing the above; zero code change. Resolves the CLI writes fail with 'database is read only' due to stale journal lock after process exit #10852/Dolt Fails to Start After Restart in Kubernetes STS with ProxySQL – Reports Database Locked by Another Process #11036 confusion at the source.
2. Docs + a clearer error. Same docs, plus replace the silent read-only fallback / generic "locked by another process" with a message that names the LOCK path and points at the doc (overlaps Feature request: The error message from Dolt when failing a write because the database is read-only should be better. #10858's ask). Lets operators self-diagnose.
3. Docs + a supported readiness/lock-state query. Expose lock-held state (e.g. via the readiness signal in the companion request, or a dolt subcommand) so supervisors coordinate on a supported signal instead of poking the LOCK file directly — removes the need for external flock entirely.

Filing as a documentation/clarity request; option 1 alone would unblock us. Thanks.

---

Filed from a downstream orchestrator (Gas City) that manages the dolt sql-server lifecycle; verified against current dolthub/dolt source.

[reltuk]

Why is documenting the internal disk structure of Dolt's storage necessary for the service monitor? What actual problem is the service monitor trying to solve that makes it think it needs to understand how Dolt uses this specific file?

[sjarmak]

The issue framing reached too far into Dolt's internals. The real problem is narrower, and it isn't about the lock file.

Gas City supervises a long-lived dolt sql-server and restarts it during normal lifecycle management: config changes, upgrades, host maintenance. The failure I keep hitting is a restart race. When Gas City stops a server and immediately starts a new one against the same data dir, the new server intermittently comes up read-only or reports "database locked by another process", because the previous process hasn't finished releasing the database. Gas City works around it today with a fixed 60s wait, both too slow for the common case and unreliable under load.

Tracing the cause led me to the LOCK acquire timeout and the read-only fallback, which is where the original framing came from. What I actually want is a supported signal that a stopped server has fully released its databases and a restart is safe: the shutdown-side complement to the readiness signal in #11261. With that, Gas City never looks at internal storage at all.

So the reframed ask: can an external supervisor get a supported way to know "this server has shut down cleanly and the data dir is free to re-open"? The recurring reports of the same restart race (#10852, #11036, #10858) suggest this isn't unique to Gas City. Happy to retitle this around that, or fold it into #11261 if you'd rather keep one lifecycle-observability thread.

[reltuk]

The most reliable shutdown signal is that the prior server process is no longer running. Is there an observed problem with something like kill() -> wait() -> fork()/exec(). systemctl restart semantics should work as expected. If they don't, that is a bug that needs to be investigated and driven to resolution.

The option to launch and be willing to wait for the lock, instead of open in read-only or fail, is potentially a reasonable way forward for launching the server in the face of uncontrolled concurrency on Dolt database Open attempts.

In almost all of these cases of confusion about lock and database access, there is something else going on. The fundamental problem is mixing dolt CLI access with dolt sql-server. When operating in that mode, dolt CLI will be trying to open the databases and if the server is restarting it might accidentally open them in read/write mode and end up with exclusive write access, whereas the intent was for the server to always have exclusive access. Is that what's happening here? If the agent needs access to the dolt databases running behind the sql-server, and it needs it through dolt CLI, maybe a more reliable pattern would be to run the client dolt processes in a separate, unrelated directory, and have them specify --host, --port, -u, -p, and --use-db, so they always connect to the running server?

We could also investigate an approach that uses filesystem permissions. The database directories are only writable by the user that the sql-server process runs as, and the Dolt CLI processes run as users with read-only access to those directories. I'm not sure this works as desired today --- we would need to test it and add the use case to our integration tests.