feat(compaction): make batch size configurable; remove dead TargetSizeMB

[xe-nvdk]

Summary

• compaction.max_files_per_batch is now configurable (default 30, range [2, 500], env ARC_COMPACTION_MAX_FILES_PER_BATCH). Compaction splits a large partition into batches, each becoming an independent job with its own output file, upload, and manifest entry — that size was a hardcoded const.
• Removed TargetSizeMB, which was plumbed through 7 files and reported on a public API but never read by anything.
• Fixed a pre-existing bug found while testing: a sub-minimum trailing remainder was emitted as its own batch and could never compact.
• Plumbed BatchNumber into job IDs and output filenames to remove a latent collision.

Why make it configurable

It's a file-count bound, not a byte bound — compacted output size tracks input file size, which follows the ingest buffer settings. Lowering it yields smaller, independently-transferable compacted files, which matters when those files are shipped over a constrained or intermittent link (edge/field deployments). The cost is more compaction jobs per partition, and in cluster mode proportionally more Raft manifest entries — at 5, a 600-file partition produces 120 manifest entries instead of 20.

Guards on both ends

1 is rejected and falls back to the default with a startup warning. compactFilesAdaptively rejects any batch below 2 files at depth 0 (manager.go:393), so max_files_per_batch=1 — plausibly the first thing an edge operator tries — would have failed every batch of every partition, with no symptom beyond a per-batch error log.

500 caps the top end. The const exists because DuckDB can abort when one read_parquet() call spans too many files, and the adaptive retry only halves four times (10000 → 625 still crashes). It's a conservative bound, not empirically derived.

Pre-existing bug: stranded remainder

31 files at batch size 30 split into 30 and 1. That 1-file batch hit the same minBatchSize floor and failed every cycle, so the partition's tail file never compacted — silently. Confirmed pre-existing by stashing the diff and re-running. Sub-minimum remainders now fold into the final batch, which overshoots by at most one file.

TargetSizeMB removal

Defaulted to 512 (hourly) / 2048 (daily), copied through five structs and the parent↔subprocess IPC, and reported as target_size_mb on GET /api/v1/compaction/stats. Nothing read it: Candidate carries no file sizes, and the compaction COPY emits no FILE_SIZE_BYTES. There was no config key, so no deployment could have set it — removal cannot change anyone's behavior.

⚠️ API shape change: target_size_mb is gone from the tiers[] entries of GET /api/v1/compaction/stats.

Configuration matrix

Configuration	Reaches new code?	Preconditions established?
OSS, compaction disabled	No — manager.go:663 unreached	n/a
OSS, key at default (30)	Yes — batching identical to today	m.MaxFilesPerBatch set at manager construction
OSS, key at NON-DEFAULT (5)	Yes — 6× more jobs/outputs on large partitions	Each batch is an independent job → independent upload + manifest
OSS, key = 1	Guard forces default	Without it, every batch fails at manager.go:393
OSS, key = 0 / negative	Guard forces default	Guard precedes all arithmetic; else div-by-zero
OSS, key = 10000	Clamped to 500 + Warn	Adaptive retry can't rescue it (10000→625)
Cluster, key at default	Yes — unchanged	CompletionDir != ""; untouched by this diff
Cluster, key at NON-DEFAULT (5)	Yes — 6× more Raft entries	Watcher has no fixed buffer (watcher.go:282-299); batches ops per-manifest via BatchFileOps (watcher.go:371), so no per-file Raft proposals
Hourly + daily both on, key at 5	Yes — compounding	More hourly outputs → more daily inputs, re-split by the same key
Any mode reading /api/v1/compaction/stats	Yes — target_size_mb disappears	No non-Go consumer repo-wide

Deref check: no new pointer dereferences. m.MaxFilesPerBatch is an int on an already-constructed manager.

Invariant preserved: CompactPartition takes a non-blocking, skip-on-contention per-partition lock (manager.go:222-229). Batches are safe only because they run sequentially in one goroutine — if they were ever parallelized, batches 2..N would be silently dropped and reported as success. Do not parallelize batches.

Test plan

• go build ./cmd/... ./internal/...
• go test ./internal/compaction/... ./internal/config/...
• go test -race ./internal/compaction/...
• gofmt -l clean on all changed files
• go vet clean for affected packages
• Binary run at NON-DEFAULT (max_files_per_batch=5) — drove 13 real files through compaction: 3 batches of 5/5/3, succeeded: 3, failed: 0, distinct _b2/_b3 job IDs and output filenames
• Binary run at 1 — clamp warning fired: configured=1 effective=30 min=2 max=500
• Binary run at default — no warning, no errors, clean startup
• Verified target_size_mb absent from the live /api/v1/compaction/stats response
• New tests verified to FAIL pre-fix — reverted the remainder fix, confirmed the odd-file-count cases fail, restored

New test coverage

• HonorsConfiguredSize — same input, different param, different result (12 files → 1 batch at 30, 3 batches at 5)
• ClampsOutOfRange — table-driven over {-1, 0, 1, 2, 10000}
• AtMinimumSize — exact batch sizes at maxFilesPerBatch=2 across file counts 3–7
• EveryBatchIsCompactable — sweeps batch sizes × file counts, asserting every emitted batch clears compactFilesAdaptively's floor
• BatchNumberAtCustomSize — 1-based and distinct, including single-batch-is-1-of-1
• TestLoad_CompactionMaxFilesPerBatch — default resolves to 30; env var lands correctly

Review

Internal deep review found two blockers, both fixed here: the minBatchSize=1 hard-fail, and a collision mitigation that cited a batch suffix which didn't exist. A second pass verified the remainder arithmetic (exhaustive sweep over ~15,600 (batch size, file count) combinations: zero file loss, zero sub-minimum batches) and identified an unreachable disjunct in the split loop, now removed.

Docs

docs.basekick.net updated in b5f4c00, gated behind a v2026.09.1+ admonition.