Skip to content

Avoid carrying empty overwrite manifests - #1154

Merged
zeroshade merged 1 commit into
apache:mainfrom
rraulinio:fix_empty_manifests
Jun 8, 2026
Merged

zeroshade merged 1 commit into
apache:mainfrom
rraulinio:fix_empty_manifests

Conversation

@rraulinio

Copy link
Copy Markdown
Contributor

Fixes #1153.

Overwrite commits read existing manifest entries with deleted entries filtered out. If no live entries remain, carrying that manifest forward keeps stale deletion history reachable in the current snapshot and causes repeated replace operations to grow the manifest list.

This change skips existing overwrite manifests that have no live entries after discardDeleted=true filtering. It does not rely on manifest-level count metadata, so zero-count inherited manifests that still contain live entries are still preserved.

A regression test covers repeated ReplaceDataFilesWithDataFiles calls and verifies the current snapshot keeps a bounded manifest list instead of accumulating old delete-only manifests.

@tanmayrauth tanmayrauth left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Walked through the four (foundDeletedCount, len(notDeleted)) cases — only (0, 0) changes behavior, which is exactly the all-tombstone manifest from a prior snapshot, and that's right to drop.
iterManifestEntries(m, true) only filters EntryStatusDELETED (manifest.go:835), so an empty iteration is a reliable signal for "no live data here". The current overwrite's own tombstones are produced separately in overwriteFiles.deletedEntries, so this doesn't lose any deletion record. Older snapshots' manifest lists still reference the skipped manifests, so time-travel and expiration paths are unaffected.

Two small things in the test, both optional:

  • table/table_test.go:1322: the trailing continue is dead since it's the last statement in the loop body.
  • table/table_test.go:1295: for i := range 5 with i+1 inside reads a bit oddly; for i := 1; i <= 5; i++ would let you drop the +1.

@zeroshade zeroshade left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zeroshade

Copy link
Copy Markdown
Member

@rraulinio let's get the suggestions that @tanmayrauth implemented first, then we'll merge

@rraulinio

Copy link
Copy Markdown
Contributor Author

@zeroshade Done.

Signed-off-by: Raul <gfxsraul@gmail.com>
@rraulinio
rraulinio force-pushed the fix_empty_manifests branch from e8748f0 to e11b9f3 Compare June 5, 2026 02:08
@zeroshade
zeroshade merged commit a286d19 into apache:main Jun 8, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Repeated overwrite commits carry delete-only manifests forward

3 participants