PG19: support distributed REPACK CONCURRENTLY

[ihalatci]

Part of #8597 (Phase 3 — later feature integration). Depends on #8756.

Current state

PG19 renames ClusterStmt to RepackStmt. #8624 / 6813788ed dispatches T_RepackStmt and propagates plain REPACK to shards through the existing CLUSTER shard-DDL path. REPACK CONCURRENTLY and REPACK ... ANALYZE are rejected with an error (src/backend/distributed/commands/cluster.c, RepackStmtOptionEnabled() guard near line 111).

That behaviour is correct and fail-closed. #8756 covers refining when the rejection happens relative to lock acquisition.

This issue covers the remaining depth gap: actually supporting REPACK CONCURRENTLY on distributed tables.

Why it is not a simple propagation

Unlike plain REPACK, CONCURRENTLY is not a self-contained DDL statement that can be forwarded per shard and forgotten:

• It runs work in a background worker, not in the issuing backend.
• It creates a temporary logical replication slot named pg_repack_<PID> to capture concurrent changes.
• It therefore requires wal_level = logical on every node that runs it.
• Slot count is bounded by a new PGC_POSTMASTER GUC, max_repack_replication_slots (src/include/replication/slot.h:327). Postmaster-level means it cannot be raised without a restart.

What distributed support has to decide

1. Slot budgeting. A distributed REPACK CONCURRENTLY fans out to every shard placement on a node. Concurrent repacks on one worker can exhaust max_repack_replication_slots, and the limit cannot be raised at runtime. Fan-out must be throttled, or the operation must be serialized per node.
2. wal_level precondition. Citus must verify wal_level = logical on all nodes and fail with a clear message rather than surfacing a per-shard slot error.
3. Partial failure. Background-worker execution means shard operations complete asynchronously and independently. Some shards can succeed while others fail, leaving the table repacked unevenly. Reporting and cleanup semantics need defining.
4. Interaction with Citus's own maintenance — rebalancer moves, shard splits, and logical replication already in flight.

Priority

Low. Nothing is silently wrong today; users get an explicit error. This is a capability gap, not a correctness gap, and it should not gate the pg19-support → main merge.

[ihalatci]

Scoping note: distributed REPACK CONCURRENTLY

I traced this against PostgreSQL 19beta2 (~/.pgenv/src/postgresql-19beta2) and Citus at
pg19-support, and verified the load-bearing claims on a live single-node cluster rather
than from the docs. Recommendation: defer. There is a hard upstream blocker that no
amount of Citus-side engineering removes, and the fallback designs all trade the current
fail-closed behaviour for something worse.

The decisive blocker

repack.c:307:

PreventInTransactionBlock(isTopLevel, "REPACK (CONCURRENTLY)");

Citus ships shard DDL through worker_apply_shard_ddl_command inside a distributed
transaction. That is a transaction block on the worker, so REPACK (CONCURRENTLY) on a
shard errors out by construction — it is not a matter of Citus choosing not to
propagate it. cluster.c:129-134 already records this.

Any implementation therefore has to run each shard's REPACK outside the coordinated
transaction, on its own connection, with no 2PC. That is not a variation on the existing
shard-DDL path; it is a second, parallel execution model with its own failure semantics.

Corrections to premises in the issue

wal_level = logical is not required. check_concurrent_repack_requirements
(repack.c:907+) gates only on wal_level < WAL_LEVEL_REPLICA, i.e. the default replica
is sufficient. EnsureLogicalDecodingEnabled() (repack_worker.c:224) turns on logical
decoding on demand. Confirmed live: a core-only REPACK (CONCURRENTLY) on a cluster with
wal_level=replica created slot pg_repack_76251 and completed. So there is no cluster-wide
wal_level precondition to check or document.

The slot budget is per-backend, not per-shard. The slot is named pg_repack_<PID>
(repack_worker.c:221) and is RS_TEMPORARY; one REPACK per backend means N shards on one
node do not consume N slots from a single statement. The pressure is still real, just
differently shaped: max_repack_replication_slots defaults to 5
(src/backend/replication/slot.c:163, postgresql.conf.sample:359) and has context
PGC_POSTMASTER (src/backend/utils/misc/guc_parameters.dat:2108) — it cannot be raised
without a restart — while citus.max_adaptive_executor_pool_size defaults to 16.
Concurrent user REPACKs, not fan-out width, are what exhaust it.

Repack slots come from a dedicated reserved slice, not the general pool.
ReplicationSlotCreate (slot.c:440-441) partitions the shared slot array:

startpoint = !repack ? 0 : max_replication_slots;
endpoint = max_replication_slots + (repack ? max_repack_replication_slots : 0);

so REPACK allocates only from [max_replication_slots, max_replication_slots + max_repack_replication_slots) and everything else only from [0, max_replication_slots).
Two consequences. REPACK cannot starve the shard rebalancer or shard splits of slots,
and vice versa — so slot contention between the two is not a concern. But the slice is a
hard per-node ceiling of 5 concurrent REPACK CONCURRENTLY, not a soft limit that can
be borrowed against: raising max_replication_slots does not move it, only the
PGC_POSTMASTER max_repack_replication_slots does, and that needs a restart. It can also
be set to 0, disabling the feature on a node outright. (An operator who hits the ceiling
is at least pointed at the right knob: slot.c:461-462 emits
errhint("Free one or increase \"%s\"", repack ? "max_repack_replication_slots" : ...).)

This is the load-bearing capacity argument, and it is a firmer one than contention would
have been: any fan-out design must gate on a fixed, dedicated resource it cannot borrow
against — 5 per node, against a citus.max_adaptive_executor_pool_size default of 16 —
rather than compete for an elastic shared pool. Admission control is therefore mandatory,
not an optimisation.

What is actually required, if this is ever built

1. A non-transactional shard execution path. No 2PC, therefore no atomic rollback.
2. Partial-failure semantics. Background-worker execution means shards finish
   asynchronously and independently. A mid-run failure leaves a table repacked unevenly.
   There is no rollback story, so this needs an explicit contract (resumable? best-effort?
   reported how?) before any code.
3. Per-node slot admission control, keyed on concurrent REPACKs per node against the
   PGC_POSTMASTER limit of 5, with a clear error rather than a mid-fan-out failure.
4. Eligibility pre-checks. Core additionally requires a permanent relation, a replica
   identity that is neither NOTHING nor FULL, and a valid identity index or
   non-deferrable PK. Plenty of distributed tables fail these, and the failure would surface
   per shard, late.
5. Interaction with Citus's own logical replication. The shard rebalancer and shard
   splits already use logical replication slots on the same nodes. As noted above they do
   not contend for the same slot budget, so this is not a capacity problem — but REPACK and a
   concurrent shard move do both need AccessExclusiveLock on the same relation at their
   final swap, so this still needs a mutual-exclusion story with the rebalancer.

Verdict

Defer. Item 1 alone means a new execution model, and items 2 and 5 are design questions
about correctness guarantees, not implementation detail. The current behaviour — reject with
CONCURRENTLY option is currently unsupported for distributed tables — is fail-closed and
correct, and is much better than a version that succeeds on some shards.

Worth noting for whoever picks this up: repack.c:3234 has
INJECTION_POINT("repack-concurrently-before-lock", NULL), which would make the
lock-ordering parts testable deterministically.

Separately, while tracing this I confirmed #8756: PreprocessClusterStmt took
AccessExclusiveLock unconditionally, before the CONCURRENTLY check, which both made the
rejection wait on an exclusive lock and silently escalated the lock on non-distributed
tables inside a Citus-enabled database. That is a distinct, small fix and is tracked there.

---

Edited to fix two errors in the original version of this comment: (1) PGC_POSTMASTER and
the default of 5 were attributed to src/include/replication/slot.h, which only carries the
extern declaration — the context lives in guc_parameters.dat and the default in
slot.c; (2) I claimed REPACK competes with the rebalancer for the same slot budget. It does
not — slot.c:440-441 gives REPACK a dedicated slice. Subsequently also softened a claim
that the dedicated slice leaves operators without guidance: core's own errhint
(slot.c:461-462) names the correct GUC. The capacity argument now rests on the hard
per-node ceiling of 5 rather than on contention, which is a stronger footing. The verdict is
unchanged throughout.

[ihalatci]

Correction to this issue body (author self-correction)

Two of the premises I wrote here do not survive checking against the PG19 beta2 source. Recording rather than editing, so the error stays visible.

1. The wal_level = logical premise is DISPROVEN. I stated REPACK CONCURRENTLY requires wal_level = logical. It does not - see repack.c:907+ and :914. This removes what I presented as the main deployment obstacle.

2. Slot contention is DISPROVEN as I framed it. I implied concurrent repacks would collide over replication slots. They do not collide in the way I described (slot.c:440-441, :461-462).

3. What actually survives, and it is still blocking. The real constraint is a fixed, dedicated, per-node ceiling of 5 slots, changeable only by restart (guc_parameters.dat:2108, PGC_POSTMASTER). Citus defaults citus.max_adaptive_executor_pool_size to 16. A distributed REPACK CONCURRENTLY fanning out to more nodes than the ceiling allows will fail partway, so admission control is mandatory, not optional. That is a stronger and more specific argument than the one I originally made.

4. Separate blocker, unchanged: PreventInTransactionBlock at repack.c:307.

Net effect: the recommendation to defer stands, but for one concrete reason (fixed per-node slot ceiling plus mandatory admission control) rather than the three I originally listed, two of which were wrong.

[ihalatci]

Status: no local implementation/PR; recommendation is defer. Needs: architecture for nontransactional placement execution, rebalancer exclusion, fixed per-node REPACK slot/background-worker admission control, partial-failure/retry semantics, then design approval before code. The smaller rejection-timing bug is tracked separately in #8756.