import_backup: convert a get_backup archive (upload or pull) into a managed backup #2639

Part of #2632 (item 7). Design note §5.2–§5.5 and §8 in docs/proposals/archive-restore.md on branch design/archive-restore. Depends on #2633, #2636, #2640. This is the operation #995 asked for; it is filed fresh rather than reopening #995 because the design has changed shape since.

Problem

Nothing consumes a get_backup archive. The only route is the README's "stop Harper and hand-copy files into the database directory and each blob root", which requires reproducing the blob root-index mapping by hand (records persist the root index, not the path) — a silent-mis-addressing failure mode — and is impossible on Fabric.

What to build

import_backup converts an archive into a verified managed backup id in the database's repository. No restore, no restart. Two sources, one pipeline after the bytes are on disk:

• Push: multipart streaming upload (the deploy_component machinery; server/serverHelpers/multipartParser.ts). The declared uncompressed size is a field part and is advisory (early reject only — it is client-supplied). The HTTP worker streams to staging; when the upload is complete the main thread adopts the staging area (owner record rewritten to the import id, fsynced) before the job row is inserted, so a worker that dies between enqueue and a post-enqueue rewrite cannot have its staging swept from under the job.
• Pull: url (https) or the s3 object import_from_s3 takes. A standard job (scalar request); resumable with Range; never crosses the Fabric ingress. Primary Fabric form.

Staging is one area beside the repository (<backupsRoot>/<db>/.import/<importId>/) so the blob snapshot can hard-link; entry filtering is explicit (reject .., absolute paths, symlinks, hardlinks, device nodes) and bounded by entry count, metered uncompressed bytes, per-entry size, path length, duplicate names — the multipart parser deliberately enforces no size cap (multipartParser.ts:15-17), so inode exhaustion is otherwise unbounded. Failure removes staging eagerly; the main thread sweeps worker-owned staging on worker exit; a boot sweep removes staging no live job owns.

Disk gate: one ledger across staging, conversion and any later restore, grouped by the quota or filesystem each path lands on, debited by every byte written, re-based only on a sample newer than the last debit. getStorageSpaceStats() under basis: 'quota' returns the same host-manager snapshot for ~90 s (server/storageReclamation.ts:19-22), so re-reading it is not a gate; EDQUOT/ENOSPC is the authoritative stop. Blob bytes count unless the hard-link actually succeeded (EMLINK/EPERM fall back to copy, dataLayer/blobBackup.ts:59-80).

Validation and conversion, in a purpose-built child process (engine level only, no resource layer; bounded by wall-clock, output size, heap, block cache, file descriptors, and oom_score_adj on Linux):

1. RocksDatabase.open on the staged directory.
2. validateTransactionLogStore(<staged>/transaction_logs, { strict }) — the check backups.verify runs.
3. Walk __dbis__ for the expected table set and read the physical column-family inventory from OPTIONS-*; a catalogue row whose family is missing fails here.
4. db.backup(<backupsRoot>/<db>, { transactionLogs: true }) — the conversion (createBackupOffline already does this against a closed directory, dataLayer/rocksdbBackup.ts:959-990).
5. Opt-in: checksum walk; blob-reference scan.

Then, back in Harper: snapshotBlobs from the staged blobs/<rootIndex>/ trees, the manifest (blobs, expected table set, inventory, archive digest, producer fields from #2633, imported_from), directory-only verify_backup (#2636), remove staging. Signal death, timeout, malformed output and non-zero exit are all failures; the serving process survives each. The strict Harper replay is not here (a replay writes and needs the resource layer); it belongs to the publication protocol in the restore routes.

Pull-source security (see the note's §5.2 for the full list):

• Never ambient credentials: the S3 client is built with explicit credentials only (as utility/AWS/AWSConnector.js does today); the url client carries no Harper TLS client certs and no inherited headers; GET, https only; other schemes rejected.
• No secrets in the job row: inline long-lived keys are rejected; accept a secrets-store reference (as deploy_component resolves registry/git credentials, components/secretOperations.ts:337) or a presigned URL, redacted to origin and path in every log, error and result.
• One fetch helper shared with csv_url_load and deploy_component package=<url>, so the SSRF policy tracked for csv_url_load applies here too; the default allow/deny list must keep today's private-endpoint pulls working.
• Optional sha256 pin; digest and imported_from always recorded.
• Idle and total download timeouts; one import per database and per instance, with the lock owned by the main thread (a process-owned flock taken on an HTTP worker strands on the routine worker restarts deploy_component/restart_service perform — dataLayer/restoreMarker.ts:41-46).

Engine-only archives (exclude_blobs) import fine; their manifest records blobs: false and the single rule in #2640 governs restoring them in place.

Acceptance

• CLI get_backup → import_backup (push, and pull from S3 and https) → verify_backup on the imported id, including into a database that does not exist yet.
• Malformed, truncated, legacy and pre-manifest archives; inode exhaustion; a gzip bomb; a stale quota sample with an undeclared upload (ledger holds); EDQUOT mid-write cleans up.
• A worker killed between enqueue and adoption (staging survives for the job) and between adoption and enqueue (staging swept); an HTTP worker restart mid-upload releases the lock and sweeps staging.
• Inline S3 keys rejected; a presigned URL never appears un-redacted in logs, errors or results; the S3 client never falls back to the default provider chain (test with ambient env credentials present).
• Unrelated request latency measured during extraction and conversion.