chore(iceberg): move Iceberg export to iceberg-go v0.7.0

[xe-nvdk]

Summary

Moves Arc's Iceberg exporter from iceberg-go v0.6.0 to v0.7.0. No Arc source changes were needed for the bump itself — it is source-compatible. What this PR carries is the behavioural fallout, which is where the risk in a bump like this lives.

Why

v0.7.0 drops a manifest left with no surviving entries instead of carrying it into every later snapshot, and parallelises the manifest scan. That is the upstream half of #1106 — and it is already fixed upstream, so no new issue was filed: apache/iceberg-go#1153 and #1393 cover it and are closed. Measured on the same 300-file rig, one file replaced per pass:

removal passes	manifests	DELETED-only	scan plan
	v0.6.0 → v0.7.0	v0.6.0 → v0.7.0	v0.6.0 → v0.7.0
16	33 → 18	16 → 1	6.87 → 2.99 ms
64	129 → 66	64 → 1	18.62 → 7.12 ms

Per accumulated manifest, a reconcile pass costs ~0.24 ms instead of ~1.6 ms. The .avro count on disk is unchanged (258 after 64 passes on both), so #835's orphan sweep is still the only thing reclaiming metadata files.

Blocker found and mitigated: a dotted namespace becomes unaddressable

Arc builds one namespace component per database, nsPrefix + "_" + sanitizeNamespaceDB(database). v0.7.0's SQL catalog addresses a namespace whose component contains a dot by a JSON-encoded key rather than the plain dotted string, and deliberately refuses the legacy key:

// catalog/sql/sql.go, namespaceStorageKeys
// Only marker collisions get a legacy fallback. Dotted components must
// never fall back because their legacy key belongs to a different namespace.

Measured:

tableIdent("rocket.01", "cpu") = []string{"arc_rocket.01", "cpu"}
catalog row, v0.6.0            = arc_rocket.01
catalog row, v0.7.0            = __iceberg_namespace_v1__:["arc_rocket.01"]

Three failures stack, none of them loud: the v0.6.0 table and its whole history are orphaned while EnsureTable correctly creates a new one; the new directory __iceberg_namespace_v1__:[...].db is not matched by isWarehouseDir, so Measurements() walks the exporter's own metadata back in as a user database (the exact harm the /→. mapping exists to prevent, #634); and MetadataLocation() is percent-encoded while the directory is not, so no version-hint.text is published and the declined fingerprint cache re-reconciles that measurement every tick forever.

Mitigation here — two refusals, at the two scopes the problem has:

• iceberg.namespace_prefix containing a dot is refused at config load, since it would affect every database on the node.
• A database whose namespace component would contain a dot is refused per measurement (checkNamespaceAddressable, called from EnsureTable), logged and skipped, so the rest of the node keeps exporting — matching how a column-type conflict is already handled. This covers what a config check structurally cannot: an edge-sync spoke registering after startup.

I deliberately did not add a node-wide startup refusal: one dotted spoke should not stop a hub from booting, and the per-measurement guard already prevents every new broken table.

Reachability. Arc database names are dot-free by their create-time rule. The one source that permits a dot is an edge-sync spoke ID — validateSpokeID rejects /, \, : and .., but not ., so rocket.01 is valid today, and the Iceberg source receives spoke namespaces unconditionally when both features are on.

What is still unfixed: a table already published under a dotted namespace is now refused rather than served, and there is no migration. #1129 covers the permanent fix — emitting an addressable namespace and migrating existing tables, filed as one issue because shipping the first alone would orphan tables the same way the bump does.

Threshold re-derived; it stays at 12

Growth is now one manifest per append pass instead of two per removal pass, and the per-manifest cost fell ~6.7×. T* = sqrt(C(N)/0.12) = 6.5 / 8.2 / 16.6 / 23.5 at 300 / 1000 / 5000 / 10000 live files, so 12 is within ~1.25× across the range — better behaved than on v0.6.0.

Live measurement sharpened what the collapse is actually for. Only append-only passes grow the set: a removal-only pass usually adds nothing (v0.7.0 drops the manifest it empties), and a pass containing an in-place rewrite already merges the whole set as a side effect of #633's re-registration. The collapse is the backstop for append-only growth, and a table both written and deleted from reaches the threshold slowly or never.

Honesty about the tests

Two regression tests from #1124 no longer discriminate and are labelled as such rather than left looking like proofs. EmptyingPassSucceeds and AllSurvivorsSkippedDoesNotCollapse both pass with their guard removed, because v0.7.0 drops the entry-less manifests that made the merge illegal — ErrEmptyManifest still exists and is still returned by ManifestWriter.Close, but reaching it needs ≥2 inherited data manifests whose entries are all foreign-DELETED, and an emptying pass now inherits exactly one, taking mergeBin's single-manifest passthrough. The guards are kept: one comparison, they encode the intent, and reachability returns with a multi-spec table or any change to that drop.

BelowThresholdLeavesManifestsAlone had hard-coded v0.6.0's 1 + 2*3 growth. It now asserts the count never drops, which is what a collapse actually looks like, so it cannot encode a library constant again.

Six comments corrected

v0.7.0 falsified them; all rewritten in this PR rather than left to rot: the #633 tombstone premise, the empty-manifest refusal, collapseManifests' purpose, the RemoveProperties note (v0.7.0 adds it), the snapshot-age default (which was wrong for v0.6.0 — it defaulted to effectively forever, not 5 days — and is now correct), and the threshold derivation.

Also commented: tableDataFiles is correct only because it is a map keyed on path storing the file's size. v0.7.0 can plan several tasks for one Parquet file above read.split.target-size (128 MiB), so len(tasks), a slice, or task.Length would each double-count. Arc's own files are far below that; bulk-imported Parquet need not be.

New dependencies

OpenTelemetry's API, RoaringBitmap v2, and geospatial encoders — for deletion vectors, geometry columns and remote scan planning, none of which Arc uses. No telemetry is registered or emitted: the OTel SDK is not in Arc's module graph, the only instrumented path requires a scan-planning mode Arc never selects, and the metrics reporter defaults to a no-op. No new network traffic, no new log output. Flagging it explicitly because Arc dropped curl from the container to shed 22 CVEs, so added dependency surface is a deliberate trade here, not a free one.

Configuration matrix

Configuration	Reaches new code?	Preconditions established?
iceberg.enabled=false (default)	No — exporter not constructed; the config refusal is inside if cfg.Iceberg.Enabled	n/a
OSS standalone, iceberg.enabled=true	Yes — EnsureTable → checkNamespaceAddressable before tableIdent	strings only, no deref; runs before any catalog or FS call
Cluster, primary writer	Yes, same path	writer-gated per tick (scheduler.go:142); no new cluster field touched
Cluster, reader/follower	No — writer gate returns first	n/a
iceberg.namespace_prefix dotted (NON-DEFAULT)	No — refused at config load, process never starts	verified on the binary
iceberg.namespace_prefix=wh_nondefault (NON-DEFAULT, valid)	Yes — the #534 cell: isWarehouseDir keys off the prefix	live-verified: wh_nondefault_hv.db/cpu, version-hint.text published, DuckDB reads it
Spoke ID containing a dot (NON-DEFAULT)	Yes — that measurement refused, others continue	the dynamic case config load cannot see; fails without the guard
iceberg.retain_snapshots=3 (NON-DEFAULT)	Yes	live-verified; v0.7.0's new 5-day default age is overridden by Arc's explicit WithOlderThan(0)
iceberg.reconcile_interval=5 (NON-DEFAULT)	Yes, more passes	live-verified
Bulk-imported Parquet > 128 MiB	Yes, via tableDataFiles	may split into several tasks; map-keyed-on-path makes it correct; commented

Test plan

• go test -tags=duckdb_arrow -race -count=1 ./... — 39 packages ok, 0 FAIL, 0 DATA RACE (internal/cluster 156s, internal/api 121s, internal/ingest 88s, so the suite really ran)
• -race -count=20 on the new and changed tests: ok, 143s
• New internal/iceberg/namespace_guard_test.go: the guard refuses rocket.01, site.a.b, rocket/telemetry and a dotted prefix, accepts mydb/rocket-01/rocket_01; a reconcile for a dotted database errors, writes no catalog row and no warehouse directory, and a dot-free database on the same exporter still exports. Fails without the guard: reconcile accepted a dotted database: it would publish an unreadable table
• New config test: my.warehouse, .arc, arc. refused; arc, arc_wh, my-warehouse accepted
• go build ./cmd/... ./internal/..., gofmt -l, go vet: clean
• Rebased on current main and re-verified
• Live binary run, iceberg.namespace_prefix=wh_nondefault + retain_snapshots=3 + reconcile_interval=5 (three non-default keys):
  • dotted prefix refuses at startup with the reason
  • warehouse at wh_nondefault_hv.db/cpu; version-hint.text published under the non-default prefix
  • manifest count climbs 2,3,4…12 over twelve append passes, then the next pass collapses to 1, logged once
  • iceberg_scan reads the table before and after; no Iceberg error or warning

Refs #1106, #835. Follow-up: #1129.