feat(backup): record the cold-tier files a backup does not carry (#1085)

[xe-nvdk]

Closes #1085.

A backup copies hot storage. On a deployment with tiered storage on, the files that have been
migrated to the cold tier are not in it, and until now nothing said so: a manifest reading
total_files: 1400 was describing 1400 hot files and however many cold ones you had, silently.

Every backup manifest now carries cold_files_excluded, with cold_files_excluded_databases
breaking it down per database — because a single total does not tell an operator which database
has data the backup is missing, and that is the whole point of the marker.

{
  "backup_id": "backup-20261007-141500",
  "total_files": 1400,
  "cold_files_excluded": 412,
  "cold_files_excluded_databases": { "audit": 400, "metrics": 12 }
}

The figure also appears in the backup listing, per target in GET /api/v1/backup/:id, and in a
warning the restore logs.

This is the last stage of #1085. B1 made the destination configurable, B2a added the Azure key
prefix, B2b-1 allowed a remote destination, B2b-2 added per-database routing; this adds the marker
that says what a backup of a tiered deployment is not carrying. Backing up the cold tier itself is
#1086.

The decision this stage turns on

cold_files_excluded deliberately does NOT join the replace-mode incompleteness refusal.

replaceDatabases refuses a replace-mode restore from an incomplete backup, because replace
deletes the current files of every database it restores and then puts the backup back — so the
difference is lost. A cold-tier file is a different population: no backup carries it yet, and
replace cannot delete it either. Three independent mechanisms say so:

• the delete set is built only from the cluster manifest entries passed in, and tiering removes a
  migrated file from that manifest as phase 2 of the migration (releaseHotCopies, manifest first);
• the shared-backend object delete runs against the hot backend and cannot address the cold
  store at all;
• the tier rows survive — recordHotFileIfNotCold binds its ON CONFLICT update to tier = 'hot'.

So refusing on this count would prevent nothing, and would refuse a replace restore for every
partly-cold database — the normal state of any deployment with tiering on. It would make tiered
storage and replace-mode restore mutually exclusive. There is a comment at that refusal saying
exactly this, and a regression test that fails if the count is ever added to it.

The comment is deliberately narrow about what it claims. It says the refusal would prevent nothing;
it does not say a partly-cold backup restores leaving tiering consistent, because that is false
in a neighbouring case (see Internal changes).

Shape

• backup.ColdCounter — one method, CountColdFilesByDatabase, wired from the tiering manager in
  cmd/arc/main.go beside SetTierLookup, nil when tiering is off. The same pattern backup (stage A): scope a backup to one or more databases #1084 used, so
  the backup package still does not import tiering.
• One grouped query, not one per database:
  SELECT database, COUNT(*) FROM tier_files WHERE tier = ? AND quarantined_at IS NULL GROUP BY database.
  The set that matters is "every database with cold rows", and that is not the backup inventory — a
  fully cold database has no hot files, so it is absent from the listing and from
  manifest.Databases while still holding the rows this counts. Grouping covers that by
  construction, and cannot half-fail the way N point queries can.
• Per leg, by the same rule the data follows (run.legFor), since B2b-2 made backups fan out:
  the audit target's manifest reports the audit database's gap and nothing else. Summed into the
  run-level view by mergeRunManifests.
• Informational. A counting failure warns and the backup completes: the count describes data the
  backup was never going to carry, so failing the run would trade a healthy backup for no backup
  over a diagnostic.
• No new config key.

Test plan

• 20 new tests (internal/backup/cold_marker_1085b3_test.go, internal/tiering/cold_count_1085b3_test.go)
• 12 pre-fix proofs — each mutation applied and its test confirmed to fail: merge sum
  removed · count written to every leg · filtered by the inventory · scope filter dropped · count
  failure fatal · cold count added to the replace refusal · restore Progress mirror dropped ·
  TargetSlice loses the field · SummaryFromManifest loses the field · quarantined_at filter
  removed · nil-receiver guard removed · breakdown map not merged
• go test -race green on internal/backup, internal/tiering, internal/config,
  internal/api, cmd/arc
• gofmt -l ./internal ./cmd empty; go vet clean
• Configuration matrix written first, 17 rows, every row traced
• One deep reviewer + a second on the SQLite Review Checklist
• Live run, standalone, real SeaweedFS cold tier: the audit leg carries {audit:1} only; the
  default leg carries {cold:1} for a fully cold database absent from the inventory; the
  merged listing is the two legs summed; a merge restore leaves the cold objects at 2→2; tiering
  off → the field is absent
• Live run, 5-node cluster, real cold tier: a partly-cold database (3 cold rows, 2 hot files
  left) backed up at total_files=6, cold_files_excluded=1, and the replace-mode restore
  completed (files_registered=4, no error) with cold objects 1→1 and cold tier rows 3→3 —
  the decision-1 proof, run on a cluster because mode=replace is refused outright off-cluster
• Release notes: ## New features + ## Internal changes; arc.toml operator note
• Rebased onto current main (it moved 4 commits mid-run); re-ran build, gofmt, vet and the
  race suite on the combined tree, plus a binary run confirming cold_counter: true and no
  panics with tiered_storage.enabled=true and cold.enabled=false

Review

Two passes, 15 pre-fix proofs. Neither found a blocker. What they did find, and what changed:

Deep reviewer, one High — a licence claim that was simply wrong. I had written that the counter
is wired "whenever tiering is CONFIGURED, including without the tiering licence". It is not: the
tiering block in main.go is gated on the licence, so an unlicensed node has a nil
tieringManager, never reaches SetColdCounter, and reports nothing. That matters because it is
the deployment where the marker would help most — migrations stopped, the files already out of hot
storage — and an operator reading my docs would have taken an absent field as proof nothing was ever
migrated. The behaviour stays (the alternative is running a tiering schema on an unlicensed node,
which is the boundary the licence pattern exists to hold); the prose is corrected in all four places
it appeared, and now says plainly that an absent field is not that evidence.

Four Mediums, all taken: the one-leg merge clone aliased the breakdown map — the manifest's first
map field, where in-place writes are natural — so it is now copied, with a test; the matrix row for
"a fully cold database routed to a non-default target" had no test, so it has two now; the
BackupSummary doc claimed the breakdown reaches the per-target detail, which it does not; and the
Progress field now records that the count survives a run that later fails.

SQLite checklist reviewer, no highs — all eight items pass. Its one Medium was a performance
claim of mine that measurement contradicted, and I reproduced it before acting: I had written that
the chosen plan "needs no new index", but a covering index on (tier, database, quarantined_at) is
~6x faster (21 ms → 3 ms) and is picked without ANALYZE. The decision not to add it stands, on
cost rather than optimality — ~7% of the database file and a sixth b-tree on a table the ingest
flush path writes to, to save ~18 ms once per backup run — and the comment now records the scale at
which that trade flips (~1M cold rows). It also caught that the quarantine tests got the production
time domain by coincidence; they now go through QuarantineFile and get it by construction.

Internal changes worth knowing

A pre-existing defect this work surfaced but does not fix. A restore of a file that was hot when
the backup was taken and migrated to cold afterwards writes a hot copy back and registers it, while
the tier report that should flip the row back to hot is silently refused —
recordHotFileIfNotCold's ON CONFLICT update is excluded for a cold row, so RowsAffected() is 0,
wrote is false, and the switch in tiering/replicated.go hits neither the error branch nor the
case wrote branch: no counter, no log. The query path then unions the hot and cold globs and reads
the file twice. This concerns files the backup does carry, happens with this marker at 0, and
happens in merge mode too, so gating on this count would not address it. Filed separately.

A low count is reachable with no error. The marker is the reporting node's view of its own tier
metadata. On a gated cluster only the primary migrates, the others learn through
syncColdTierMetadata, and CreateBackup is not primary-writer gated — so a reader whose sync has
not run yet answers a number that is present and too low. The field doc and the log line say so, and
the log names the node.