Problem
CreateBackup (internal/backup/backup.go, classification loop around line 126) copies only .parquet objects and Iceberg metadata. Compaction's crash-recovery manifests under _compaction_state/{tier}/{database}/{job}.json are never copied. (One accidental exception: a database literally named metadata matches the /metadata/ Iceberg catch-all.)
A compaction job writes its manifest before uploading the output and deletes it only after the inputs are deleted (internal/compaction/job.go around lines 385-410). A backup whose listing lands between the upload and the input deletion therefore holds both the compacted output and its inputs. Restoring that backup brings back both, and without the manifest nothing ever completes the deletion: recovery (internal/compaction/manifest.go) is the only path that removes consumed inputs. The restored node serves those rows twice, permanently.
Proposal
- Back up
_compaction_state/**/*.json as auxiliary state alongside the data files (counted for progress, kept out of the database inventory), and restore it with everything else so the next compaction cycle's recovery completes the interrupted job.
- Think through replaying a manifest onto a store that has moved on since the backup: recovery already tolerates a missing output (deletes the manifest and retries) and already-deleted inputs, which is the state a restored manifest lands in; confirm with a test rather than assume.
- Alternatively, or additionally, make the backup refuse to snapshot while a compaction cycle is running, or record the in-flight set in the manifest so a restore can warn.
Acceptance
- A backup taken with a pending manifest restores it; the next compaction cycle deletes the consumed inputs; a query over the partition returns each row once.
- Backups without manifests are unchanged.
- Release notes entry.
Found while reviewing #927.
Problem
CreateBackup(internal/backup/backup.go, classification loop around line 126) copies only.parquetobjects and Iceberg metadata. Compaction's crash-recovery manifests under_compaction_state/{tier}/{database}/{job}.jsonare never copied. (One accidental exception: a database literally namedmetadatamatches the/metadata/Iceberg catch-all.)A compaction job writes its manifest before uploading the output and deletes it only after the inputs are deleted (internal/compaction/job.go around lines 385-410). A backup whose listing lands between the upload and the input deletion therefore holds both the compacted output and its inputs. Restoring that backup brings back both, and without the manifest nothing ever completes the deletion: recovery (internal/compaction/manifest.go) is the only path that removes consumed inputs. The restored node serves those rows twice, permanently.
Proposal
_compaction_state/**/*.jsonas auxiliary state alongside the data files (counted for progress, kept out of the database inventory), and restore it with everything else so the next compaction cycle's recovery completes the interrupted job.Acceptance
Found while reviewing #927.