Repeated overwrite commits carry delete-only manifests forward

[rraulinio]

Apache Iceberg version

main (development)

Please describe the bug 🐞

Problem

Repeated overwrite commits can keep carrying old manifests that no longer contribute any live entries to the current snapshot.

In overwriteFiles.existingManifests, existing manifest entries are read with deleted entries filtered out:

for entry, err := range of.base.iterManifestEntries(m, true) {
    ...
}

If a carried manifest only contains deleted entries, this leaves notDeleted empty and foundDeletedCount == 0. The current logic then appends the original manifest to existingFiles, so the delete-only manifest remains reachable from the new snapshot.

Why This Matters

For workloads that repeatedly replace the full table contents, every overwrite can create a delete-entry manifest for the previous files. Old delete-only manifests are then carried into later snapshots even though they do not affect current snapshot reads.

Over time, the current snapshot's manifest list grows with stale delete-only manifests and normal orphan cleanup cannot remove those Avro files because they are still reachable.

Expected Behavior

After deleted entries are filtered out, a manifest with zero remaining live entries should not be carried forward into the next overwrite snapshot.

This should not drop zero-count inherited manifests blindly. The check should be based on actually reading entries with discardDeleted=true, not on manifest count metadata, because some valid inherited manifests may have zero or unset counts but still contain live entries.

Proposed Fix

In the overwrite existing-manifest path, skip an existing manifest if iterManifestEntries(m, true) produces no live entries.

Add a regression test with repeated ReplaceDataFilesWithDataFiles calls to verify the current snapshot keeps a bounded manifest list instead of accumulating old delete-only manifests.

[rraulinio]

PR: #1154.