fix(compaction): materialize dedup normalization in temp table to stop bind wedge

[xe-nvdk]

Summary

• Fixes a live compaction wedge on the dedup path (measurements with tag metadata): INTERNAL Error: Failed to bind column reference "time": TIMESTAMP WITH TIME ZONE != VARCHAR.
• This is a follow-up to fix(compaction): stop time-type-mismatch wedging partitions; un-wedge existing #493. That fix wrapped the time normalization in a CTE the dedup window reads from. On real partitions (many files at the full column width of a typical measurement, e.g. host, time, value, cpu_idle, cpu_user), DuckDB does not treat the CTE as a materialization barrier — the window's time reference binds against the raw read_parquet(union_by_name=true) scan and fails, even when every file's time is already TIMESTAMP WITH TIME ZONE and no VARCHAR file exists anywhere (the VARCHAR in the error is a binder artifact, not a real column type).
• fix(compaction): stop time-type-mismatch wedging partitions; un-wedge existing #493's regression test used a 3-column fixture below DuckDB's column-count threshold for this binder behavior, so the failure only ever surfaced on production-shape partitions. Affected measurements failed to compact on every cycle and accumulated raw files (no self-heal).

Root cause (verified empirically against Arc's linked DuckDB, v1.5.01, on the real wedged files)

Query form	Binds on real files?	Dedups correctly?
plain time (no normalize)	❌ phantom VARCHAR	—
CTE + REPLACE (what #493 shipped)	❌ phantom VARCHAR	—
CTE + OFFSET 0	❌ (passed on synthetic, failed on real)	—
flat no-CTE * REPLACE … QUALIFY	✅	❌ silently under-dedups (QUALIFY runs over raw time)
CREATE OR REPLACE TEMP TABLE AS + COPY	✅	✅

The fix

• buildCompactionQuery now returns []string: 1 statement (standard path — query text unchanged) or 2 (dedup: temp-table materialization + COPY running the dedup window over the table).
• The caller pins both statements to a single connection via db.Conn (a DuckDB TEMP table is connection-local; database/sql does not guarantee two ExecContext calls share a pooled connection) and DROPs the temp table before releasing the connection (the DuckDB driver implements no session reset, so a leftover temp table would retain a partition's rows on a pooled connection).
• Standard (no-tag) compaction path and dedup semantics (last-write-wins on (tags, time)) are unchanged.

Test plan

• go build ./cmd/... ./internal/..., go vet -tags duckdb_arrow ./internal/compaction/, gofmt -l — all clean
• go test ./internal/compaction/ (default tag) — pass
• go test -tags duckdb_arrow ./internal/compaction/ — pass
• TestDedupCompaction_RealPartition (opt-in via ARC_REPRO_DIR) — the faithful regression against real Arc-written files; verified the old query fails on these exact files and the new one binds + dedups (6000 rows, injected VARCHAR duplicate collapses to 1)
• Synthetic TestBuildCompactionQuery_DedupMixedTimeAtScale kept but honestly documented as a weaker guard (synthetic files don't reproduce the all-TZ binder quirk)
• Live: a manual compaction cycle that previously wedged now reports 269/269 succeeded, 0 failed

Review notes

Internal review found and this PR addresses: (1) the temp-table connection-locality hazard (pinned connection), (2) a retained-memory leak from the temp table persisting on a pooled connection after Close() (explicit DROP), (3) stale OFFSET 0 comments corrected, (4) release-notes time-type entry corrected (the CTE materialization claim did not hold at scale).

🤖 Generated with Claude Code

[xe-nvdk]

@gemini-code-assist please review

Focus areas for this fix:

• The dedup path now runs two statements (CREATE OR REPLACE TEMP TABLE + COPY) pinned to a single connection via db.Conn, with an explicit DROP of the temp table before the connection is released. Please scrutinize the connection lifecycle in job.go compactFiles — temp-table visibility across the two statements, leak-on-error, and that the post-loop rowsAfter probe (which uses the pool, not the pinned conn) is correct.
• Whether CREATE OR REPLACE TEMP TABLE materialization is the right barrier vs alternatives, and whether dedup semantics (last-write-wins on (tags, time)) are preserved.
• The standard (no-tag) branch should be behaviorally unchanged from main.

[gemini-code-assist]

Code Review

This pull request resolves a critical issue where deduplication compaction would wedge on wide, many-file partitions in DuckDB. The fix replaces the previous CTE-based materialization with a temporary table materialization pinned to a single connection, ensuring proper column binding and preventing silent under-deduplication. Additionally, subprocess log lines are now forwarded at their original severity. The reviewer's feedback correctly identifies a potential resource leak where the temporary table might not be dropped if the job's context is cancelled, and suggests using a detached background context for the cleanup query.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Code Review

This pull request refactors the dedup compaction path to use a connection-local DuckDB temporary table instead of a CTE, resolving a plan-binding error on wide, many-file partitions. It also updates subprocess log forwarding to preserve the original log severity. The review feedback correctly identifies a risk where the temporary table cleanup in the defer block could be skipped if the context is cancelled, and suggests using a detached context with a timeout to ensure reliable execution.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[xe-nvdk]

Addressed the review finding (commit 8ba5258).

Temp-table cleanup now runs on a detached context. The deferred DROP TABLE IF EXISTS arc_compaction_staged previously executed on the job's ctx; a cancelled or timed-out job would skip it and leak the temp table on the pooled connection (the DuckDB driver implements no session reset). It now runs on an independent context.Background() with a 5s timeout, so cleanup always executes regardless of the job context's state. The same fix is mirrored in the execCompaction test helper. Build, vet, gofmt, and the full compaction suite (both build tags) plus the opt-in real-partition regression all pass.

@gemini-code-assist please review

[gemini-code-assist]

Code Review

This pull request addresses a DuckDB plan-binding error during dedup compaction on wide, many-file partitions by replacing the CTE-based materialization of the normalized time column with a temporary table materialization. The statements are pinned to a single connection to ensure visibility of the connection-local temp table, which is then explicitly dropped to prevent memory leaks. Additionally, subprocess log forwarding has been improved to preserve and re-emit logs at their original severity level instead of flattening them to INFO. I have no feedback to provide as there are no review comments to assess.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.