chore(cluster): remove unwired sharding subsystem (closes #448)

[xe-nvdk]

Summary

Removes internal/cluster/sharding/ (~5k LOC + tests) — the Phase-4 "multi-writer sharding foundation" (commit c05bdde, 2026-01-13): consistent-hash data partitioning across nodes with scatter-gather queries. It was never wired into the running server:

• RouteShardedWrite/RouteShardedQuery (in internal/api/routing.go) had zero non-test callers.
• No ShardRouter is ever constructed in cmd/arc/main.go or the coordinator.

Arc shipped two different horizontal-scaling models instead, both of which replicate the full dataset per node rather than partitioning it — neither imports sharding:

• Pattern A — shared object storage (cluster.shared_storage_mode)
• Pattern B — per-node local storage + peer file/WAL replication (internal/cluster/replication + filereplication)

Deleting the superseded sharding code removes ~5k LOC of dead code and dead attack surface — including the client-spoofable X-Arc-Shard-Routed loop guard that Gemini flagged on #530 (which was in this unreachable code). Closes #448.

What's removed

• internal/cluster/sharding/ (19 files: shardmap, meta/FSM, router, scatter-gather, shard replication/receiver, failover, aggregation + tests)
• The 4 unused shard helpers in internal/api/routing.go (RouteShardedWrite, RouteShardedQuery, HandleShardRoutingError, ShardRoutedHeader) + their tests
• The now-unnecessary X-Arc-Shard-Routed entry from the clientForwardingHeaders strip list (added in fix(cluster): strip client forwarding headers + harden loop guard (CVE-2026-45045 class) #530 — no shard router to defend anymore)

The live parallel-partition-scan query speedup (internal/query/parallel_executor.go) is independent and untouched.

Test plan

• go build ./cmd/... ./internal/... clean (default + duckdb_arrow tags)
• gofmt -l clean, go vet ./internal/api/... clean
• go test ./internal/api/... ./internal/cluster/... pass; replication suites pass under -race
• Live Pattern B cluster (deploy/docker-compose/enterprise-local, 3 writers + reader, per-node volumes) on the post-deletion binary: write → Parquet flush → peer-replicated to all 4 independent volumes → reader queried the row back (n=1, max(temp)=42.5). Reader log confirms File pulled from peer peer=arc-writer1:9100.
• Confirmed coordinator.go / main.go (Pattern A) and replication/filereplication (Pattern B) never imported sharding — 0 files in those paths changed

🤖 Generated with Claude Code

[xe-nvdk]

@gemini-code-assist please review

This removes the never-wired internal/cluster/sharding subsystem (~5k LOC, closes #448). Key things to verify:

• Nothing live imports it: only internal/api/routing.go (dead helpers, also removed here) referenced the package; coordinator.go/main.go (Pattern A shared-storage) and replication/filereplication (Pattern B per-node replication) never did.
• The parallel-partition-scan query speedup (internal/query/parallel_executor.go) is independent and must remain untouched.
• This moots your fix(cluster): strip client forwarding headers + harden loop guard (CVE-2026-45045 class) #530 High finding on the X-Arc-Shard-Routed loop guard (that code is gone).
• Build passes on both default and duckdb_arrow tags; a live Pattern B cluster replicated + queried correctly on the post-deletion binary.

[gemini-code-assist]

Code Review

This pull request removes the entire sharding implementation from the codebase. It deletes the sharding package, which includes the shard map, shard router, scatter-gather query coordinator, failover manager, meta-cluster coordinator (Raft FSM), shard Raft manager, and shard replication sender/receiver components. Additionally, it removes shard-routing endpoints and error handling from the API routing layer, along with all associated unit and integration tests. I have no feedback to provide as there are no review comments.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Code Review

This pull request completely removes the sharding implementation from the codebase. It deletes the entire internal/cluster/sharding package, which includes the shard map, shard router, failover manager, meta cluster coordination, replication senders/receivers, scatter-gather query logic, and two-stage aggregation rewriter. Additionally, all sharding-related routing functions, headers, and corresponding unit tests have been removed from the internal/api package. As there are no review comments provided, I have no feedback to offer.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.