fix(compaction): preserve S3 prefix in subprocess; drop dead ResilientBackend (#320, #560)

[xe-nvdk]

Closes #320. Closes #560.

Investigating #320 surfaced a live correctness bug (#560) in code that #320's own subject — ConfigJSON() — feeds. Both are here.

#560 — S3 prefix silently dropped in compaction subprocesses (the real bug)

Compaction forks a subprocess for DuckDB memory isolation; the child rebuilds its storage backend from the parent's Type() + ConfigJSON() (manager.go:248-249 → subprocess.go:308).

S3Backend.ConfigJSON() emits prefix (s3.go:712), but the subprocess parse struct had no Prefix field (subprocess.go:320-326) and the reconstructed S3Config never set it. json.Unmarshal discarded it silently.

The prefix is applied via prefixedKey() to every key the backend touches — read, write, delete, list (s3.go:235, 269, 309, 339, 370, 403, 429). So on any deployment with storage.s3_prefix set, compaction subprocesses operated against the bucket root instead of the configured prefix. No error raised.

S3Config.Prefix is a real operator-settable key (config.go:539) documented for multi-tenant use (instances/abc123/).

Why it survived: prefix defaults to empty, so "prefixed" and "unprefixed" are the same string in every test and default deployment. Exactly the shape CLAUDE.md flags after #534 — a config key whose default makes two different values identical.

The failing round-trip shows it directly:

parent:  {"bucket":"arc-data",...,"prefix":"instances/abc123/",...}
rebuilt: {"bucket":"arc-data",...,"prefix":"",...}

Azure and local have no equivalent gap — each emits exactly the fields the subprocess parses. I checked.

#320 — ResilientBackend removed rather than completed

internal/storage/resilient.go (395 lines) wrapped a backend with retry + circuit breaker. The issue asks for pass-through Type()/ConfigJSON() to satisfy Backend.

It's dead code. Every reference lives inside the file itself — zero call sites in cmd/ or internal/, no tests, no type assertions, no doc/config mentions. git log shows one commit: the original Go migration. It was never wired in.

It's missing four methods, not two (ReadToAt, StatFile, Type, ConfigJSON) — it drifted as the interface grew, and nothing regressed precisely because nothing constructs one.

Removed rather than completed: retry/circuit-breaking around cloud storage is worth having, but an unused untested wrapper drifts again at the next interface change. A future implementation should target the interface as it stands then and actually be wired in. Follows the #448 sharding-deletion precedent. The build is unchanged — the code was unreachable.

Configuration matrix

Configuration	Reaches new code?	Preconditions established?
Any deployment (ResilientBackend removal)	No — the type was unreferenced; go build ./cmd/... ./internal/... clean after deletion	n/a
Compaction disabled	No — subprocess never forked	n/a
Compaction + local backend (default)	Yes — createStorageBackendFromConfig "local" branch	Unchanged; single field, parsed
Compaction + S3, s3_prefix unset (default)	Yes	Was already correct — empty prefix round-trips as empty
Compaction + S3, s3_prefix SET (NON-DEFAULT)	Yes — this was the bug	Prefix now parsed and forwarded; round-trip test pins it
Compaction + Azure	Yes	No gap — emits exactly what it parses

Test plan

• TestCreateStorageBackendFromConfig_PreservesS3Prefix (new) — verified it fails against the pre-fix code, printing the exact prefix divergence above
• TestCreateStorageBackendFromConfig_PreservesLocalConfig (new) — same contract for the default backend
• TestCreateStorageBackendFromConfig_RejectsUnknownType (new) — unknown type must error, not fall back to a default backend
• go test ./internal/compaction/... and ./internal/storage/... pass
• go build ./cmd/... ./internal/..., go vet, gofmt -l clean

Binary run — drove the real subprocess path end to end:

• Wrote 3 files into a backdated partition, aged the filenames past hourly_min_age_hours (compaction keys off the filename timestamp, not the data timestamp), triggered compaction
• Compaction cycle complete | succeeded=1, compaction-subprocess ... files_compacted=3, bytes_before=3381, bytes_after=492
• All 3 rows still queryable after compaction (SELECT count(*) → 3)
• Zero unsupported storage type / failed to parse storage config / failed to create storage backend lines
• Clean Graceful shutdown complete

End-to-end against real S3 (MinIO), with the prefix set — the caveat in the original PR description is now resolved. Built two images from this branch, identical except for the four-line fix, and ran the same scenario against deploy/docker-compose/oss-s3-style MinIO with ARC_STORAGE_S3_PREFIX=instances/abc123/:

	pre-fix image	fixed image
Subprocess result	files_compacted=0, bytes_before=0, bytes_after=0	files_compacted=3, bytes_before=3402, bytes_after=606
Reported status	success:true, total_succeeded=1	success:true, total_succeeded=1
*_compacted.parquet produced	none	instances/abc123/.../mem_..._compacted.parquet
Source files after run	all 3 untouched	deleted, replaced by the compacted file
Objects at bucket root	none	none
Query after compaction	—	SELECT count(*) → 9, all rows intact

The pre-fix failure is worse than "compaction targets the wrong location". The subprocess listed the bucket root, found nothing to compact, and reported success — files_compacted=0 with total_succeeded=1 and no error at any log level. So on a prefixed S3 deployment, compaction silently never ran: files accumulated forever while every cycle reported success. That is a silent-no-op, not a loud failure, which is why it could persist unnoticed.

The parent process honors the prefix correctly in both images (writes landed under instances/abc123/ pre-fix too) — confirming the bug was isolated to the subprocess's backend reconstruction.

Also included: .dockerignore

Building the image from this working tree failed — the build context pulled in a 27GB local data/ directory and 137MB of release tarballs, exhausting the builder's disk before the binary compiled. Added a .dockerignore covering gitignored runtime data, packaging artifacts, and docs/deploy trees that aren't needed to compile or run the binary. Unrelated to the bug, but it blocked the end-to-end verification above and will block anyone else building locally.

One process note

My first attempt at a regression guard was a test that compared ConfigJSON() against a struct literal hand-written in the test file rather than the real parse struct. It passed against the buggy code — it could never have caught the bug, since I controlled both sides. I deleted it and replaced it with the round-trip above, which drives the production function. Flagging it because a guard that cannot fail is worse than no guard: it reads as coverage.