[sinashamsizadeh]

Root cause

No copy of the data leaves the server. OFFSITE_BACKUP_TARGET is not set on the production server, and the only
target types were rsync: (needs a full shell on the destination) and dir: (a mounted path). The owner chose an
SFTP account as the destination (Google Drive was dropped: Google restricts business services in Iran). Off-site
destinations were also never pruned. Today, losing the server loses everything. Part of Daneshbonyan re-evaluation
track 7 (evaluator reason 4: database maturity).

Design doc

docs/features/db-maturity/RESEARCH.md section 6 item 10 and work-breakdown row H, ADR-028 in
docs/engineering/DECISIONS.md. This PR is the DUMP half of row H. The WAL half (pgBackRest repo2 over SFTP) comes
with the pgBackRest unit.

What it does:

• New sftp:user@host:/path target in app/services/backup_offsite.py: system sftp in batch mode, fixed argv, no
  shell, key login only, strict host-key checking against a known_hosts file.
• The dump is encrypted to a gpg PUBLIC key before upload (fingerprint checked; a private key file is refused). Only
  the .gpg file leaves the server. The private key exists only on paper, so the server cannot read its own copies.
• The manifest offsite block records the sha256 of the encrypted file; the remote size is checked.
• Remote files are named <database>.<backup id>.dump.gpg; prune keeps the newest OFFSITE_REMOTE_KEEP and only
  ever deletes this install's own files (also for dir:), so two installs sharing a path cannot delete each other's
  copies. Leftover .part files of this install are cleaned too. rsync: is documented as not pruned.
• Failure stays non-fatal, as before.
• Admin page: "هیچ نسخه‌ای بیرون از سرور نیست" when no target is set; otherwise the last off-site result.
• Five settings (OFFSITE_SFTP_IDENTITY_FILE, OFFSITE_SFTP_KNOWN_HOSTS, OFFSITE_GPG_PUBLIC_KEY,
  OFFSITE_GPG_FINGERPRINT, OFFSITE_REMOTE_KEEP), all empty by default: nothing changes on an install that sets nothing.
• Runbook (Persian): make the gpg key pair off the server, print the private key to paper, set up the SFTP account
  and host key, one path per install, and restore after server loss. It says plainly that no destination exists yet.

Admin-panel destination (H2, product owner decision 2026-10-08)

The operator now sets the SFTP destination in the admin panel (Infrastructure > Backups), not only by env:
host, port, username, remote path, a password OR a private key, and the server's host-key fingerprint (pinned,
never auto-accepted). Settings table -> env -> default, the same pattern as app/services/sms.py. The password and
key are stored with secure_store (enc: tokens) and never returned to the browser, logs or audit entries; the page
shows only "saved", with replace or clear. A "Test connection" button checks the pinned host key, writes and removes a
small test file, and answers in plain Persian (one reason per failure, including a short refusal after wrong tries).
A destination counts as ready only when encryption is ready too, so the page never says "ready" while copies would
fail. Admin-only (verify_admin), CSRF prefix, per-admin rate limit on the test, audit-logged. The password reaches
ssh through SSH_ASKPASS, the key through a 0600 temp file that is always removed: nothing secret on argv.
The off-site copy now also writes its manifest key under the same lock as verify() and the restore drill.
This branch also merges main after #179, so the drill section and the SFTP card share the Backups page.

Tests

• Oracle written by the reviewing agent from the task contract before it saw the code
  (test_offsite_sftp_oracle.py, not committed): on the base 6 failed + 32 errors / 4 passed; on this branch 42/42.
  It caught 3 deliberate mutations (a broad prune pattern, StrictHostKeyChecking=no, a missing fingerprint check).
• Committed: tests/test_backup_offsite.py, tests/test_backup_offsite_admin.py, and tests/test_offsite_sftp_live.py,
  which runs against a throwaway SFTP-only container (atmoz/sftp) and PostgreSQL 16: the upload is encrypted
  (pg_restore --list fails on it), the private key decrypts it to the exact original, a wrong fingerprint uploads
  nothing, prune keeps exactly N, a wrong host key is refused, and two installs on one path keep each other's files.
• New CI job offsite-sftp runs the live test on GitHub.
• H2 oracle (written first by the reviewing agent, 110 cases incl. live password/key/host-key-mismatch): on the H1
  base 104 failed / 6 passed; on H2 110/110. Live tests against a real SFTP container (password, key, pinned and
  wrong host key, a real sshd temporary refusal): 19 passed. Real browser on the app with PostgreSQL: both page
  sections render, no console errors.
• Not run: anything against a real SFTP destination (none exists yet).

Security

No plain dump ever leaves the server. sftp and gpg run with a fixed argv and no shell; the sftp batch text only
holds safe characters; host keys are checked strictly; login is key-only; the server holds only the public gpg key.
The database name used in remote file names is validated (unsafe names fail the copy). The admin line is behind the
existing verify_admin door. Throwaway keys in tests are generated in a temp dir and never committed.

AI assistance

Written by Claude agents in a Herdr team: implementer ops-t1-impl (Opus 5.5) wrote the code, tests, CI job and
docs; reviewer ops-t1-rev (Opus 5.5) wrote the oracle first and reviewed two rounds (round 1: 1 medium, two
installs on one path could delete each other's copies, fixed); final reviewer drill-t1-final (Opus 5.5, xhigh)
reviewed H1, then the whole PR after H2 and the merge (accepted; its lows fixed, including moving the off-site
manifest write under the lock); H2 implementer and reviewer ran at xhigh effort (auth and secrets); lead foreman7 (Opus 5.5)
re-ran the oracle on the base and on each patch. Log entry in docs/engineering/AI_ASSISTANCE_LOG.md.

Human review