feat(backup): scope a backup to one or more databases (#1084)

[xe-nvdk]

Closes #1084. Stage A of the internal backup series (#1083–#1087); community contributions are not accepted for this series.

Summary

• POST /api/v1/backup takes databases: ["audit", ...]. A scoped backup copies those databases and nothing else: their data files, their _schema/ anchors and their _compaction_state/ manifests (parked .quarantined ones included). The manifest records scope; the list and the status endpoint show it; the 202 echoes databases. backup_type stays full. Empty or absent is today's whole-instance backup, byte for byte (unscoped manifests have no scope key).
• include_metadata/include_config default to false when scoped; explicit include_metadata: true → 400 (the SQLite is every database's state). Names use the storage-segment rule (isSafeStoragePathSegment), no reserved roots, no duplicates, ≤ 256; unknown names → 400 naming them. "Known" = tier rows (one indexed query, first), or a listable object under <db>/, or under _schema/<db>/, or, only after those three said no, hidden keys under <db>/ (so the all-unaddressable fatal still names the real problem). The two prefix tests go through a new bounded storage.PrefixProber (local walk with early stop, S3/Azure first accepted key per page); backends without it fall back to ListObjects.
• Scope is the storage-root segment: ["prod"] does not include edge-sync spoke1/prod, ["spoke1"] takes the spoke with its anchors and compaction state (_schema/<spoke>/<db>/ is an exact second-segment rule, _compaction_state/<tier>/<spoke>/<db>/ a boundary-prefix rule on the database part; both agree).
• Iceberg: the scoped databases' namespace directories are excluded and counted (iceberg_namespace_files_excluded, iceberg_namespaces_excluded), at the root or under a subdirectory warehouse (fix(iceberg): warehouseRelKey must trim the storage root, not the warehouse #534 shape) or in an outside-root warehouse; an object-store warehouse is excluded but not counted (documented).
• Cluster: the cross-check and the late-arrivals recheck consider only manifest entries whose PATH first segment is in scope (not Database, which is canonical for spokes). findUnaddressable enumerates only the scope's prefixes.
• Restore: backup.ResolveRestoreMode(raw, scope, clustered) is the one resolver; a scoped backup with no mode restores in replace on a node whose Raft manifest is wired (handler and manager agree via Manager.ClusterManifestWired(), not the coordinator's presence); explicit merge honoured; standalone stays merge; the 202 echoes the effective mode and restore progress carries scope. Replace selects the entries to remove by path first segment when scoped. Replace is refused (400 / failed) when the scoped backup holds no data files for one of its databases (fully cold, or dropped and re-created with stale tier rows): it would only delete. restart_required/staged only when the backup actually holds metadata/config.
• A non-empty request body must be JSON-typed: curl's -d default (form) was parsed by Fiber into an empty request and ran a whole-instance backup (seen live); now 400. Empty body still means the defaults. arcli already sends application/json.
• Wiring: backupManager.SetTierLookup(tieringManager) next to SetTierRecorder; tiering.Manager.DatabaseHasTierRows (nil-receiver safe).
• Not in this stage: remote targets (backup (stage B): named remote targets (local, S3, Azure) with per-database routing #1085), cold tier objects and the INCOMPLETE marker (backup (stage C, enterprise): include cold-tier objects in backups and restores #1086/backup (stage B): named remote targets (local, S3, Azure) with per-database routing #1085), arcli backup create --database (arcli#44), scoping below the storage-root segment.

Review

Configuration matrix + one deep reviewer + a cluster-ops reviewer (CLAUDE.md). Folded in: the pure-delete replace of a zero-file scoped backup (Blocker, both reviewers → refusal); form-typed bodies running whole-instance backups (High, also seen live); whole-root hidden-key walk on scoped runs (High); manager belt for include_metadata; Iceberg exclusion count moved before the data copy; replace admission keyed on the manifest hook; restart_required gating; restore progress scope; doc fixes.

Test plan

• gofmt -l, go build, go vet clean (-tags=duckdb_arrow, plus objectstore for storage)
• go test -race: internal/storage, internal/backup, internal/tiering, internal/api (duckdb_arrow, ~2 min), cmd/arc
• 25 new tests (plus new cases in three existing ones), each shown failing with the covered line neutralised (go test -overlay), each alone with -race -count=20
• One existing test adapted on purpose: TestRestoreBackupModeValidation/cluster encoded the old keying (coordinator present ⇒ replace admitted); replace admission is now keyed on the Raft manifest hook, and the coordinator-without-Raft cell moved to TestRestoreBackupEchoesTheResolvedMode
• S3 contract tests (-tags=objectstore) including the new TestS3HasObjectsUnderPrefix against a live SeaweedFS 4.47
• Standalone live run (OSS binary, local primary, SeaweedFS cold tier, tiering on): scoped backup holds only a/ + _schema/a/, manifest scope set and backup_type full, metadata refusal, unknown/traversal/duplicate/empty/reserved names → 400, form/text/no-Content-Type/malformed bodies → 400, fully cold database (no hot files, no anchor, tier rows only) accepted and completed with 0 files, unscoped manifest has no scope, scoped merge restore touched only a, replace on standalone → 400
• Cluster live run (Pattern 1: 3 writers + reader + compactor): unscoped backup reports the one manifest-only entry of b; scoped ["a"] reports none and holds only a; scoped restore with no mode echoes and runs replace under the backup (stage 0 follow-up): a replace restore can race an in-flight compaction job; add a cluster-wide compaction pause #1087 pause (5 acks), b untouched on every node; explicit merge honoured; no warn/error lines
• Standalone with an S3 primary (SeaweedFS) and with an Azure primary (real storage account via az): unknown name → 400 through the real probers (0.03 s on S3, 0.35 s on Azure, three probes plus the hidden-key enumeration), scoped backups held only the scope, scoped merge restore wrote the files back into the object store and the query saw them; no warn/error lines
• Cluster with tiering on every node and a SeaweedFS cold tier: a database made fully cold (daily file migrated to the cold bucket, hot copies dropped, anchor removed) was accepted scoped and backed up with 0 files; the restore with no mode (cluster default replace) and an explicit replace both answered 400 with the refusal, an explicit merge completed with 0 files under the pause, and the other database was untouched

Filed from the live runs: #1091 (a dropped database keeps its tier rows and cold objects) and #1094 (DELETE /api/v1/databases/:name on a cluster node removes only that node's files; the manifest entries and the replicas on the other nodes remain, which the cluster run showed as manifest_only_files: 2 on the scoped backup).