Found by Fable 5:
Concurrent creates produce duplicate chunks that later crash borg compact (and leak space).
Two clients backing up overlapping data concurrently will both see a new chunk id as "not seen" and each writes its own copy into its own pack. Both incremental index fragments get written; when merged, the ChunkIndex can point to only one copy — the other pack now contains an object no index entry covers.
compact_packs() calls compact_pack(pid, keep_ids=..., drop_ids=...) with the default complete=True (compact_cmd.py:283), and compact_pack then asserts the indexed objects tile the pack with no gaps (repository.py:936 and the covered == pack_size assert at line 943). A pack holding a superseded, unindexed object has a gap, so as soon as that pack qualifies for a threshold rewrite, compact dies with AssertionError — after delete_chunkindex_from_repo() already ran, and it recurs on every subsequent run (the slow index rebuild can't help: a dict can index only one copy per chunk id). Repository.delete's docstring already acknowledges superseded bytes and passes complete=False; compact assumes the opposite.
Two related accounting gaps from the same root cause (pack usage computed only from index entries, compact_cmd.py:219-225):
superseded bytes are invisible to pack_unused, so a pack full of them but with all indexed objects used is "all used → keep" — the duplicate copies are never reclaimed;
a pack with no indexed objects (e.g. a create that crashed after PackWriter.flush() but before the fragment write) is entirely invisible to compact and leaks forever.
Suggested direction: compact should derive per-pack ground truth from the pack headers (it already has PackReader.iter_headers) or at least call compact_pack(..., complete=False) and treat unindexed spans as droppable waste.
Found by Fable 5:
Concurrent creates produce duplicate chunks that later crash
borg compact(and leak space).Two clients backing up overlapping data concurrently will both see a new chunk id as "not seen" and each writes its own copy into its own pack. Both incremental index fragments get written; when merged, the ChunkIndex can point to only one copy — the other pack now contains an object no index entry covers.
compact_packs()callscompact_pack(pid, keep_ids=..., drop_ids=...)with the defaultcomplete=True(compact_cmd.py:283), andcompact_packthen asserts the indexed objects tile the pack with no gaps (repository.py:936 and thecovered == pack_sizeassert at line 943). A pack holding a superseded, unindexed object has a gap, so as soon as that pack qualifies for a threshold rewrite, compact dies withAssertionError— afterdelete_chunkindex_from_repo()already ran, and it recurs on every subsequent run (the slow index rebuild can't help: a dict can index only one copy per chunk id). Repository.delete's docstring already acknowledges superseded bytes and passescomplete=False; compact assumes the opposite.Two related accounting gaps from the same root cause (pack usage computed only from index entries, compact_cmd.py:219-225):
superseded bytes are invisible to
pack_unused, so a pack full of them but with all indexed objects used is "all used → keep" — the duplicate copies are never reclaimed;a pack with no indexed objects (e.g. a create that crashed after
PackWriter.flush()but before the fragment write) is entirely invisible to compact and leaks forever.Suggested direction: compact should derive per-pack ground truth from the pack headers (it already has
PackReader.iter_headers) or at least callcompact_pack(..., complete=False)and treat unindexed spans as droppable waste.