Skip to content

backup: compaction recovery manifests are not backed up, so a restore can serve compacted rows twice #930

Description

@xe-nvdk

Problem

CreateBackup (internal/backup/backup.go, classification loop around line 126) copies only .parquet objects and Iceberg metadata. Compaction's crash-recovery manifests under _compaction_state/{tier}/{database}/{job}.json are never copied. (One accidental exception: a database literally named metadata matches the /metadata/ Iceberg catch-all.)

A compaction job writes its manifest before uploading the output and deletes it only after the inputs are deleted (internal/compaction/job.go around lines 385-410). A backup whose listing lands between the upload and the input deletion therefore holds both the compacted output and its inputs. Restoring that backup brings back both, and without the manifest nothing ever completes the deletion: recovery (internal/compaction/manifest.go) is the only path that removes consumed inputs. The restored node serves those rows twice, permanently.

Proposal

  • Back up _compaction_state/**/*.json as auxiliary state alongside the data files (counted for progress, kept out of the database inventory), and restore it with everything else so the next compaction cycle's recovery completes the interrupted job.
  • Think through replaying a manifest onto a store that has moved on since the backup: recovery already tolerates a missing output (deletes the manifest and retries) and already-deleted inputs, which is the state a restored manifest lands in; confirm with a test rather than assume.
  • Alternatively, or additionally, make the backup refuse to snapshot while a compaction cycle is running, or record the in-flight set in the manifest so a restore can warn.

Acceptance

  • A backup taken with a pending manifest restores it; the next compaction cycle deletes the consumed inputs; a query over the partition returns each row once.
  • Backups without manifests are unchanged.
  • Release notes entry.

Found while reviewing #927.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions