feat(ops): add verified SQLite backup and recovery rehearsal

CHANGELOG.md

@@ -9,6 +9,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Added a verified SQLite backup and recovery-rehearsal operator boundary that
+  creates live `VACUUM INTO` snapshots, fails closed on source/backup integrity
+  or foreign-key defects, preserves schema/version metadata, never overwrites an
+  existing destination, publishes owner-only backup files, and keeps destructive
+  restore outside the automated interface.
 - Added deterministic PM analysis for requirements/RFI/RFP readiness, WBS
   estimation coverage, dependency risk, and procurement package section checks.
 - Preserved PM-analysis research papers, NASA WBS handbook, BCP 14, and JSON

README.md

@@ -96,7 +96,7 @@ Docker: set a **persistent** `SCOPEWEAVE_JWT_SECRET` first, then run `docker com
 | Var | Purpose |
 | --- | --- |
 | `SCOPEWEAVE_JWT_SECRET` | **Required** — JWT signing secret (at least 32 non-whitespace characters; startup fails closed otherwise) |
-| `SCOPEWEAVE_DB` | SQLite path (default `data.db`; `:memory:` for tests) |
+| `SCOPEWEAVE_DB` | SQLite path (default `data.db`; `:memory:` for tests). Back up or verify it with `npm run ops:sqlite-backup`; see [`docs/operations/sqlite-backup-recovery.md`](docs/operations/sqlite-backup-recovery.md). |
 | `PORT` | API port (default 8787) |
 | `OIDC_ISSUER/CLIENT_ID/CLIENT_SECRET/REDIRECT_URI` | Real SSO IdP (mock when unset) |
 | `STRIPE_SECRET_KEY` | Real checkout (mock URL when unset) |

docs/deploy.md

@@ -122,6 +122,9 @@ logs, traces, or alert annotations.
 ## Data & scale path
 
 - **Dev / single node**: `node:sqlite` on a persistent volume (this setup). Simple, no external DB.
+  Use `npm run ops:sqlite-backup` for verified live backup or read-only verification,
+  and follow [`docs/operations/sqlite-backup-recovery.md`](operations/sqlite-backup-recovery.md)
+  for command syntax, recovery rehearsal, sidecar preservation, and restore safety.
 - **Production / multi-instance**: swap the SQLite driver in `server/db.mjs` for
   managed **Postgres** (the schema is Postgres-portable) and run several stateless
   backend replicas behind a load balancer. `node:sqlite` is a single-writer,
@@ -145,4 +148,4 @@ stays focused on the container + compose path.)
 ## Health
 
 `GET /api/health` → `{"ok":true}`. Wired as the container `HEALTHCHECK` and the
-compose healthcheck.
+compose healthcheck.

docs/doctoring/sqlite-backup-recovery.md

@@ -0,0 +1,68 @@
+# Doctoring: verified SQLite backup and recovery
+
+Status: **implemented on active PR only** until merged into protected `develop`.
+
+## Decision
+
+ScopeWeave's supported live SQLite backup path uses SQLite `VACUUM INTO` through the repository's existing Node `DatabaseSync` boundary. It does not raw-copy the live main database file, does not require a journal-mode transition, and does not silently raise the repository's Node runtime floor.
+
+The operator boundary is intentionally narrow:
+
+- backup and read-only verification only;
+- the source backup connection itself is opened read-only while retaining live-WAL `VACUUM INTO` support;
+- no automated destructive restore;
+- destination never overwritten;
+- destination publication remains bound to the canonically resolved directory even if a caller-visible parent symlink is retargeted during snapshot creation;
+- a concurrent process that wins the destination name remains authoritative and is never deleted by the losing backup attempt;
+- secure temporary output reserved with owner-only permissions before snapshot work begins;
+- source and backup integrity plus foreign-key checks;
+- exact `application_id`, `user_version`, and canonical `sqlite_schema` comparison;
+- schema inspection limits SQLite's result set to 100,001 rows before materialization and rejects more than 100,000 non-internal schema objects;
+- stable non-secret JSON error codes;
+- only uniquely owned temporary output is cleaned up after pre-publication failure, without replacing the causal failure.
+
+## Primary-source basis
+
+SQLite's current backup documentation identifies the Online Backup API as the original live-backup mechanism and `VACUUM INTO` as an alternative that creates a copy of a live database. The current `VACUUM` documentation likewise defines `VACUUM ... INTO` as a backup-copy mechanism and permits the output path to be absent or an empty file. SQLite's corruption guidance warns that a live database and its rollback journal or WAL represent one logical state; copying only the main file while transactions are active can therefore produce an inconsistent backup. These properties justify letting SQLite materialize the snapshot rather than implementing a filesystem-level copy routine.
+
+ScopeWeave currently supports Node `^22.13.0 || >=23.4.0`. Node's `node:sqlite` `DatabaseSync` API is available in the supported Node 22 line, including read-only database opens and prepared statements. The implementation therefore reuses `DatabaseSync` and parameterized SQL rather than adopting a newer helper that would silently change the runtime contract. The live-WAL regression keeps a separate writer connection open while the backup connection runs `VACUUM INTO`, so changing the backup connection to `readOnly: true` retains the buyer-visible live-backup behavior while reducing source-side write authority.
+
+## Requirement-to-evidence traceability
+
+| Requirement | Implementation evidence | Regression evidence |
+| --- | --- | --- |
+| Consistent live backup | read-only source connection plus parameterized `VACUUM INTO` | open-writer WAL fixture verifies committed content in the snapshot |
+| Bounded schema inspection | `sqlite_schema` query uses `LIMIT 100001` before mapping rows | fake-database contract asserts the SQL limit and rejects a 100,001-row schema |
+| No destination overwrite | secure temp + atomic no-overwrite hard-link publication | existing destination and source-alias tests fail closed |
+| Canonical destination authority | publication uses the destination path resolved under the canonical parent directory | parent-symlink swap during snapshot cannot redirect the final backup |
+| Preserve concurrent winner | an `EEXIST` publication failure cleans only the unique temporary path, never the destination | competing publisher fixture remains byte-for-byte intact after the losing attempt fails |
+| Owner-only output | temporary file reserved at `0600`; published hard link references the same verified inode | final-mode assertion |
+| Source corruption/FK rejection | source `integrity_check` and `foreign_key_check` | corrupt and invalid-FK fixtures |
+| Backup corruption/FK rejection | independent read-only verification | verify failure fixtures |
+| Schema/version fidelity | canonical `sqlite_schema`, `application_id`, `user_version` comparison | injected metadata-mismatch snapshot |
+| Stable operator output | `runSqliteBackupCli` emits bounded JSON fields | success, usage, missing-file, output-sink, and direct-process tests |
+| Portable direct-process test path | Node `fileURLToPath` converts the module URL to a filesystem path | backup, verify, and usage subprocess cases share the converted path |
+| No destructive restore | no restore operation exported or accepted by CLI | operator runbook defines manual stopped-writer recovery rehearsal |
+| CI/coverage retention | package scripts instrument and execute the module | coverage-script contract locks both the include and regression case |
+
+The first branch commit, `3a04516d8cf3ff7336c47de911daf02faf496ef2`, intentionally added the backup contract before the production module existed. Production implementation followed on the same bounded branch. Later tests strengthened live-WAL and metadata-mismatch evidence. Security review then added a parent-symlink retarget regression at `3f55dafba40ace2a2d0192e893d961262878b8e5` and the canonical-directory repair at `1b31a58be5cfca0877252da79980d315369edf3e`. Concurrency review added a competing-publisher regression at `91f0cc411fcd8cd96be2538c8c99836ae4d76f81`; `b7083c6b8e038d1e772a2d43dfae6f804472edbb` corrected ownership so a losing publish attempt cannot delete another process's destination. Review follow-up `037fea3081774d78a80fcf04cab7fe2c1f98de76` added the pre-materialization schema-bound contract, and `2f7339486226730dd2103079b5312e437fcd4794` implemented the bounded query and read-only source connection.
+
+## Security and privacy analysis
+
+The backup may contain all persisted tenant/customer data present in the source database. The command therefore treats the destination as operator-controlled sensitive storage and never uploads it, logs row contents, or prints schema SQL. Stable CLI output is restricted to operation state, file size, SQLite application ID, user version, schema-object count, and error code. Encryption-at-rest, remote replication, key custody, retention, deletion, and geographic residency belong to the deployment/storage layer and must be configured there rather than simulated by this module.
+
+The module rejects source/destination aliasing after canonical path resolution and refuses pre-existing destinations. Temporary snapshot files are created with exclusive creation and `0600` permissions before SQLite writes into them. Publication uses one no-overwrite hard-link operation into the already resolved canonical destination directory. A caller-visible parent-symlink change therefore cannot redirect publication after validation, and an `EEXIST` race means another process owns the destination; the losing attempt removes only its unique temporary file. This keeps path authority and cleanup ownership explicit rather than relying on a vulnerable check-then-delete sequence.
+
+## Recovery boundary
+
+A backup is evidence only after verification succeeds. Restoration is deliberately excluded from the executable API because replacing a live SQLite database is destructive and requires writer shutdown, preservation of the original database plus any `-wal`, `-shm`, and `-journal` sidecars as one evidence set, deliberate removal of those sidecars from the configured active path before replacement, and post-start application acceptance. `docs/operations/sqlite-backup-recovery.md` defines that rehearsal without claiming an unmeasured RPO/RTO.
+
+## References
+
+Node.js contributors. (n.d.). *SQLite: Node.js v22 documentation*. Node.js. https://nodejs.org/download/release/latest-v22.x/docs/api/sqlite.html
+
+SQLite Consortium. (n.d.). *How to corrupt an SQLite database file*. SQLite. https://www.sqlite.org/howtocorrupt.html
+
+SQLite Consortium. (n.d.). *SQLite backup API*. SQLite. https://www.sqlite.org/backup.html
+
+SQLite Consortium. (n.d.). *VACUUM*. SQLite. https://www.sqlite.org/lang_vacuum.html

docs/operations/sqlite-backup-recovery.md

@@ -0,0 +1,71 @@
+# Verified SQLite backup and recovery
+
+Status: **implemented on active PR only** until the owning branch is merged into protected `develop`.
+
+ScopeWeave persists the self-hosted/cloud runtime in SQLite with WAL enabled. A raw copy of only the main database file is not an acceptable live-backup procedure because committed state can still reside in the WAL. The supported operator boundary therefore asks SQLite itself to create a consistent snapshot with `VACUUM INTO`, verifies the snapshot before publication, and never performs an automatic restore.
+
+## Create a backup
+
+Choose a destination on trusted storage that is not the live database path and does not already exist:
+
+```bash
+npm run ops:sqlite-backup -- backup "$SCOPEWEAVE_DB" "/secure/backups/scopeweave-$(date +%Y%m%d-%H%M%S).db"
+```
+
+The command performs the following fail-closed sequence:
+
+1. Resolve the source and destination parent to canonical filesystem paths and reject source/destination aliasing.
+2. Require the source to be a regular file and the destination parent to be an existing directory.
+3. Open the source connection read-only, then run `PRAGMA integrity_check` and `PRAGMA foreign_key_check` against the source.
+4. Capture `application_id`, `user_version`, and a canonical ordered `sqlite_schema` snapshot while bounding schema materialization to 100,001 rows and rejecting more than 100,000 non-internal schema objects.
+5. Reserve a unique temporary destination with owner-only `0600` permissions.
+6. Execute parameterized `VACUUM INTO` against that temporary path while SQLite owns consistency across the live database/WAL state.
+7. Verify the produced database independently with integrity and foreign-key checks, positive file size, and exact metadata/schema comparison.
+8. Publish the verified inode into the already resolved canonical destination directory with one no-overwrite hard-link operation. If another process wins the destination name first, the backup attempt fails with `destination_exists` and leaves the winner untouched.
+9. Remove only the uniquely owned temporary path best-effort; never treat a pre-existing or concurrently created destination as cleanup owned by the losing attempt.
+
+Success and failure output is stable JSON. Failure output contains only a machine-readable error code; raw SQLite/network/path exception text is not emitted by the operator boundary.
+
+## Verify an existing backup
+
+Verification is read-only:
+
+```bash
+npm run ops:sqlite-backup -- verify "/secure/backups/scopeweave-20260816-220000.db"
+```
+
+A successful response reports file size, SQLite application ID, user version, and schema-object count. It deliberately does not dump schema SQL into operator output.
+
+A backup is not considered usable merely because the file exists. Verification must succeed before the backup enters a retention tier or a recovery rehearsal.
+
+## Recovery rehearsal
+
+ScopeWeave intentionally does **not** provide an automatic destructive restore command. Recovery remains an explicit operator-controlled procedure:
+
+1. Put the service in maintenance mode or stop every ScopeWeave process that can write the configured database. Keep writers stopped until the recovered database has been placed and the service is deliberately restarted.
+2. Preserve the failed state as **one SQLite evidence set**. Move the original database together with every existing same-basename `-wal`, `-shm`, and `-journal` sidecar into the incident-evidence location while preserving their filename relationship. Do not discard, separately rename, or pair any sidecar with another database.
+3. Verify that the configured active database path and its `-wal`, `-shm`, and `-journal` sidecar paths are now absent. If any remain, stop the rehearsal and investigate before placing a backup.
+4. Run the read-only `verify` command against the selected backup and record the stable verification result. Do not continue with a backup that fails verification.
+5. Place only the verified backup at the configured database path with owner-only permissions. Do not copy any sidecar from the failed evidence set beside the restored snapshot.
+6. Start ScopeWeave and run application acceptance checks covering authentication, organization/project reads, permissions, and representative writes. SQLite may create fresh sidecars for the recovered database according to the configured journal mode; those new sidecars are not evidence from the failed state.
+7. Run backup verification/integrity checks again against the recovered database after the acceptance pass.
+8. Retain the complete failed-state evidence set and recovery evidence according to incident and privacy-retention policy.
+
+Do not claim a recovery-time objective, recovery-point objective, or disaster-recovery SLA until repeated protected-environment rehearsals establish measured evidence.
+
+## Operational constraints
+
+- The destination filesystem must support the file operations used by the verified publication path; unsupported filesystems fail closed rather than falling back to an overwriting copy.
+- The destination parent is resolved once before snapshot creation. Later caller-visible symlink retargeting cannot redirect the verified publication into another directory.
+- The destination name is never reclaimed from another process. Operators should choose unique backup names; a collision fails closed and preserves the existing entry.
+- Backup destinations are operator-selected trusted storage. This command does not upload, encrypt, rotate, or retain backups on behalf of the operator.
+- `VACUUM INTO` captures committed state. Uncommitted transactions are intentionally absent.
+- The backup connection opens the source read-only and does not mutate application schema, change `journal_mode`, or checkpoint WAL as a prerequisite.
+- Backup creation should be scheduled according to the buyer's measured recovery-point requirement once that requirement is formally defined; the repository does not invent an RPO.
+- Restore remains separate from backup so a typo or compromised automation cannot overwrite the live database through this interface.
+
+## Verification evidence
+
+The owning regression suite covers a populated relational database, live WAL with an open writer connection, source and backup integrity/FK checks, bounded schema inspection, schema/version matching, destination collisions and aliases, caller-visible parent-symlink retargeting, a competing process that wins the destination name, malformed/corrupt inputs, invalid foreign keys, metadata mismatch, incomplete-snapshot cleanup, owner-only permissions, stable CLI output, and direct CLI invocation. The new module is registered in normal unit CI and the repository's instrumented coverage command.
+
+See `docs/doctoring/sqlite-backup-recovery.md` for the primary-source basis and requirement-to-test traceability.