fix(iceberg): reclaim unreachable aged manifest metadata

[efegokdemir]

Summary

Reclaim unreachable Iceberg manifest-list and manifest .avro metadata
older than seven days without granting iceberg-go deletion authority
over Arc-owned Parquet data.

• Collect references from the current catalog metadata and retained
  metadata versions, including directory-reader copies.
• Abort cleanup when metadata or manifest references cannot be read,
  listings change, or the catalog advances during inspection.
• Restrict deletion to aged, unreferenced .avro files in the table's
  metadata directory.
• Run cleanup after successful publication, including converged
  reconciliations where the data-file set has not changed.
• Preserve the existing WithPostCommit(false) behaviour.

Regression

• Reproduced the quiet-table failure before the fix.
• Verified reclamation of an actual Iceberg-generated orphan manifest.
• Verified that reachable metadata and ineligible files survive.
• Added fail-closed and reachability tests.

Local validation

• Issue-specific Iceberg tests: passed.
• Full Iceberg tests: passed.
• Iceberg race tests: passed.
• Build with duckdb_arrow: passed.
• Iceberg vet, gofmt and diff checks: passed.

Scope and limitations

This addresses only orphaned manifest metadata. Growth of the
live manifest list remains unresolved.

Tests use local temporary storage. S3/Azure integration tests,
repository-wide CI and concurrent external-writer scenarios have
not been verified. Cleanup assumes the exporter owns writes to
the table's metadata directory.

ChatGPT assisted with implementation and tests. Local validation
was executed by the contributor.

Refs #835.

[jallegri]

Review note: the current CI run (Oct 6) fails in internal/tiering at TestApplyTierEventBatch_AppliesAMigrationChunkTheProbeLatencyWouldTimeOut: 346 batches applied, 54 failed, 0 dropped, out of 400 expected. This package is outside this PR's diff, so I can't attribute the failure to the manifest cleanup; please rerun or confirm it against the current base before merge. The Enterprise chart and CLA jobs passed. GitHub also currently marks the branch as having conflicts that must be resolved.

[xe-nvdk]

Thanks for this, and an apology owed first: this PR has been open since 19 September and nobody
reviewed it. In the meantime #835 was implemented internally and merged yesterday as #1107, which
closed the issue your PR targets. That is our process failure, not a judgement on the work — we did
not check whether an open PR already existed before starting, and we should have.

What shipped in #1107, for comparison with what you built here:

	this PR	shipped
reachability source	current catalog metadata + retained versions + directory-reader copies	every on-disk metadata.json
eligible for deletion	aged, unreferenced .avro in the table's metadata dir	same, .avro only, directly in the metadata dir
fail-closed	unreadable refs, changed listings, catalog advancing mid-scan	any reachability failure
re-check before deleting	yes — catalog identity, re-listing, second pass	no
grace	7 days	1 h, or 2x the reconcile interval when that is longer; not configurable
off switch	—	iceberg.orphan_sweep_enabled, default true

The two designs converged on the same core independently, which is some evidence the approach was
the right one.

One thing your PR has that ours does not, and it is the part worth keeping: the re-check
immediately before deleting — reload the table, abort if MetadataLocation() moved, re-list, and run
reachability a second time so a retained version rewritten in place is caught. The shipped sweep
resolves reachability once and then deletes, which can act on a stale scan. Filed as #1125 with your
code quoted and credit assigned to you when it lands.

We are also crediting you for this fix in the contributors list, on the basis that you implemented
it first here.

Closing as superseded by #1107. One process change on our side, so this does not happen to you
again: before we start work on an issue, we check for an open PR against it, and if there is one we
review it instead of reimplementing.

For transparency on what comes next in this area: Arc's Iceberg exporter is now maintained
internally and not open to community contributions, so #1125 and the other open Iceberg items are
tracking entries rather than invitations. That decision is about the subsystem, not about this PR.