Refuse a peer's retired table generation, and keep newer or node-local tables when a peer drops one

[kriszyp]

⊙ Problem

harper#2962 gave each table generation a createdTime and left a durable drop marker, but core's create and drop never consult them. Every caller had to check "a generation created before a drop of its name is dead" itself, and harper-pro#956 did so at each replication ingress. Thirteen review rounds kept finding paths where that check and the catalog mutation it guarded came apart:

• A delayed forwarded drop_table dropped whatever generation held the name, including one created after the drop.
• A marker recorded by another worker after the ingress check did not stop the create.
• A joining node stamped a pre-stamp peer's unstamped definition with "now", so the stale copy spread as the newest generation.
• A peer's drop removed a populated replicate: false table, breaking harper-pro#943's node-local contract.

❓ Your call: is the requirement warranted at this size? Yes, per the owner's ruling on harper-pro#956: the rule moves to the layer that owns the catalog, so every producer inherits it, instead of a fourteenth round of per-ingress patches. Reverting this PR restores harper#2962's unconditional behavior.

💡 Solution

The rule is enforced where a generation becomes or stops being live, under the lock that publishes its primary catalog row or writes its drop tombstone:

• Conditional create. A create carrying a peer's generation (origin: 'cluster', or a supplied createdTime) throws TableGenerationDroppedError (409, TABLE_GENERATION_DROPPED) when a drop marker here retires it. So does a stamped peer definition of a name that is live here, before its attributes merge into the newer generation. A peer that kept no stamp is stored at 0: any later drop retires it, nothing backfills it, and it is never confused with undefined (a table created on a pre-stamp build). Local creates are unchanged; they are stamped above the newest marker.
• Conditional peer drop. dropTable({ peer: true, droppedTime }) never retires a replicate: false table, and with a time it retires only a generation created before it.
  • A cheap read skips the teardown; writeTombstone re-checks the row it locked, beside its table-id and storage-generation identity check.
  • A kept generation has its derived indexes and maintenance restored, and the call resolves false. The peer's time is recorded as a marker either way.
  • A peer drop that finds its class stale re-runs on the current class.
  • dropTable now resolves true when it dropped (including a tombstone left for a pending full-text retirement) and false otherwise.
• Peer provenance outside the body. server.operation(body, { user, replicatedFrom }) sets the REPLICATED_FROM symbol, which a JSON request cannot carry. The bridge treats a drop as a peer's only through it. schema.dropTable discards a client's droppedTime and rejects a malformed peer time.
• Node-local drops stay local. Dropping a replicate: false table writes no droppedTime and is not forwarded. A marker from an earlier, replicated generation of the name is kept and still sent. replicate is read from the catalog row first; the class flag alone covers runtime exclusions, and non-replicating system tables are always node-local.
• Generation-safe dropTableMeta. Under the catalog lock, it removes a name's rows only when no primary row exists, live or still dropping. It still announces dropTable if its lock times out, and schema.dropTable no longer fails a completed drop over it.
• Facts the replication layer reads:
  • isDroppedPeerGeneration: one keyed read of the marker and any live tombstone, with the recorded rolling-upgrade exception.
  • createdBefore: an upper bound written on each unstamped table at the first load on a stamping build. A table created during a rollback gets its own bound at the next load.
  • Every recorded marker is announced to all threads and advances that thread's tableDropEpoch.
• LMDB drops are marked in progress, as RocksDB drops already were. A schema reload on the dropping thread no longer completes the tombstone mid-drop, which used to make the drop throw "a replacement table became current". That race also made core's own LMDB lifecycle suites flaky on main. Every exit releases the mark, a failed table cleanup included, so the next load still completes a failed drop's tombstone.
• A test-only switch, HARPER_TEST_OMIT_TABLE_LIFECYCLE, makes a node store no stamps, markers or bounds; harper-pro#956's cluster test uses it to stand in for a pre-stamp build.

How this composes with #3118

Rebased onto Preserve 5.2 rollback compatibility for first-time table creates (#3118), which touches the same create, drop-marker and retirement paths. Each path of this PR traced against it; none conflicts in behavior. Two terms differ: #3118's "unstamped" create is one whose stores have no name suffix (T/@<generation>), while this PR's "unstamped" generation has no createdTime. They are separate fields.

This PR's path	What #3118 changed there	Result
A peer's create is refused at or before a drop marker	Every RocksDB create writes its own create journal, and bare or suffixed store names follow name history	The refusal (under the lock) runs before the journal and every store open, so a refused create leaves neither (test).
An unstamped peer definition is never stamped "now"	A peer's generation attribute is replaced by this node's naming	Core still stores the peer's missing createdTime as 0. The create journal carries no createdTime, its recovery writes none, and the createdBefore bound skips a finite 0.
A peer's drop retires only an older generation, atomically	Tombstone identity is checked before the retirement journal is written. A dropper whose tombstone another thread completed retires only its own stores that the live catalog does not own	The rule runs in writeTombstone, before any tombstone exists. A kept generation writes no tombstone and no retirement journal, so neither path runs for it. Reclamation's live-store guard (liveStoreNamesFor) also covers it, since its row is live. Test: it journals nothing and survives a reload, which runs reclamation.
A node-local table is never retired by a peer, and its own drop sends no marker	On RocksDB, a drop with no droppedTime leaves an untimed /dropped/<table> row as name history	A node-local drop has no time, so it now leaves that row. Every drop-time reader skips it: getTableDrops, pendingOrRecordedDropTime, isDroppedPeerGeneration, both create checks, and harper-pro#956's tableLifecycle.ts. Nothing reaches peers, and a peer's timed drop overwrites it (test).
The per-table createdBefore bound	Nothing in the load path	Written at load after reclamation, on primary rows only. A table created during a 5.2 rollback has no createdTime and gets its own bound at the next load.
The LMDB drop mark is released when cleanup() throws	Nothing: #3118 changed only RocksDB retirement and journals	Independent.
dropTable resolves true when it dropped	retireRocksStores now also resolves false when another thread completed the tombstone	This PR's two branches after it both resolved true. That is still right, because this drop wrote the tombstone, so they are now one call with that comment.
Recovery	Interrupted-drop and journal recovery read journal phases under the catalog lock	Recovery completes only tombstones. A peer's drop writes one only after the rule passed under the lock, so recovery has nothing to judge.

The two lifecycle descriptions are now one note: untimed name history beside the timed markers, and store names and 5.2 rollback beside the stamps.

⚖️ Alternatives

❓ Your call: the planning review returned Framing-Verdict: chosen-approach-sound (acd3af81cd7a) for moving the rule into core's catalog sections. Two amendments came from it and are adopted: node-local drops are suppressed at the source, and client lifecycle fields are scrubbed. The legacy per-table layout (v3 schema/ directory, where databasePath is not the database name) is not covered: it has no database catalog to hold a marker, so its drops still leave none. Giving it one means opening a second root for a database the loader assembles from per-table files, which is a migration.

• Checks at each replication ingress (harper-pro#956 as it was): rejected. The forwarded RPC never passes those checks, and a check on a replication worker cannot be atomic with a catalog mutation another thread makes.
• Cluster-wide incarnation IDs: rejected earlier (harper-pro#956 round 1). Component tables are created independently on every node, so one live table carries several IDs.
• Do less (drop harper-pro's dropTableMeta call and add a tableReplicates guard): this leaves the delayed-RPC drop, the create race, the joining-node "now" stamp and the RPC drop of a node-local table.
• A single database-level "first stamped load" epoch in place of the per-table createdBefore: rejected. After a rollback to a pre-stamp build, it would retire tables recreated during the rollback.

❓ Your call: where the pre-stamp heuristic lives. Core treats a generation with no stamp as created at 0, so a peer's timed drop retires it. The row-time judgment for tables a pre-stamp build created stays in harper-pro#956, which applies it to a schema-exchange marker before calling dropTable; core supplies only the createdBefore bound. A forwarded drop_table skips that judgment. It arrives while both nodes are connected, so it names the generation the origin dropped. Moving the judgment under the catalog lock later is additive.

❓ Your call: a kept, replaced or node-local peer drop answers with one 200 "was not dropped" message rather than a machine-readable outcome. harper-pro collects forwarded replies and never retries on them, so nothing depends on telling the three apart. Making the outcome machine-readable later changes the wire contract.

❓ Your call: the LMDB drop mark protects only reloads on the dropping thread. Another worker's reload could already complete an active LMDB drop before this PR. Announcing the drop generation before the stores drop, as the RocksDB drop does, would close that, and adding it later is additive.

🔧 Changes

Product and architecture tour

The rule runs where a generation goes live

Where is "a generation created before a drop of its name is dead" enforced?

A table generation becomes live or stops being live only under the catalog lock that writes its primary row or its drop tombstone. That is where a peer's create is refused and a peer's drop is judged, so no caller can check the rule and then act on a catalog another thread has changed.

Creating a peer's generation, under the catalog lock

1. Newest drop — read the name's drop marker, after any interrupted drop of it completed.
2. Peer generation — origin: 'cluster' or a supplied createdTime; a definition with no stamp is taken as 0, never "now".
3. Refuse — a generation the marker retires throws TableGenerationDroppedError (409, TABLE_GENERATION_DROPPED).
4. Stamp — a peer's generation keeps its stamp; a local create is stamped above the newest drop of its name.

• A stamped peer definition of a name that is live here is checked the same way, before its attributes merge into the live generation.
• 0 and undefined mean different things. 0 is a peer generation whose node kept no stamp: any drop retires it and nothing backfills it. undefined is a table created on a build that stored no stamp.

A peer's drop retires only an older, replicating generation

What does a peer's drop do to a generation created after it, or to a replicate: false table?

A peer's drop

1. Pre-check — a read of the primary row: a node-local table, or one created at or after the peer's drop time, is kept with no teardown.
2. Under the lock — writeTombstone re-reads the row, confirms it is the same generation, and applies the same rule to the locked row.
3. Kept — derived indexes and maintenance are restored, the peer's time is recorded as a marker, and the call resolves false.
4. Replaced — a drop that reached a class another thread already replaced re-runs on the replacement.
5. Dropped — the stores are retired, the tombstone becomes the marker, and the call resolves true.

• Kept, still recorded. The peer's drop time becomes a marker here whether or not this generation goes, so the fact reaches nodes that still hold the older copy.
• Result. dropTable resolves true when it dropped and false when it kept the table or found another generation. schema.dropTable answers false with a 200 "was not dropped" message and does not forward it.

A peer's drop is named outside the body

How does core tell a peer's forwarded drop_table from a client's?

Before	After
Core read replicatedFrom from the operation body. A body carrying it with a finite droppedTime retired whatever generation held the name, a newer one included, and any client could write both fields.	server.operation(body, { user, replicatedFrom }) sets the REPLICATED_FROM symbol, which a JSON body cannot carry. Only a drop with it is a peer's, judged by the rule above. A client's droppedTime is discarded, and a peer's malformed one is rejected.

Node-local tables stay node-local

What happens to a replicate: false table when a peer drops a table of the same name?

Before	After
A peer's drop retired a populated replicate: false table. A node-local table's own drop left a marker and was forwarded to peers.	A peer's drop never retires a node-local table; the marker is still recorded for relay. A node-local table's own drop writes no droppedTime and is not forwarded; on RocksDB it leaves untimed name history, which no peer reads. replicate is read from the catalog row first, so another thread's redeclaration counts, and non-replicating system tables are always node-local.

A marker from an earlier, replicated generation of the name is kept and still sent after the name becomes node-local. Filtering markers by the name's current flag would leave an offline peer's replicated copy alive.

Facts the replication layer reads

What does core give the replication layer, which judges a peer's records without mutating the catalog?

Fact	Kept by core	Read in harper-pro#956
isDroppedPeerGeneration(db, table, createdTime)	one keyed read of the marker and of a live tombstone's droppedTime, with the rolling-upgrade exception	judging a definition or structure before its records are applied
createdBefore (catalogCreatedBefore)	written on each unstamped primary row at the first load on a stamping build; a rollback's tables get theirs at the next load	the upgrade heuristic, only for drops older than the bound
tableDropEpoch()	advanced on every thread when a marker is recorded: the recording thread at commit, the others by broadcast	the per-record recheck of an accepted generation

Drop bookkeeping a live generation survives

What can the metadata cleanup and a schema reload no longer do to a live table?

• dropTableMeta used to remove every catalog row of the name. Under the catalog lock it now removes them only when no primary row exists, live or still dropping, and it announces the drop even when it cannot take the lock. schema.dropTable logs a cleanup failure instead of failing a drop that completed.
• An LMDB drop is marked in progress from the moment its tombstone commits until it finishes, as a RocksDB drop already was. A schema reload on the dropping thread therefore leaves the tombstone to it instead of completing it underneath. Every exit releases the mark, including a failed table cleanup.

Every changed file:

• resources/databases.ts
  • Admission under the lock: a peer's create and a stamped definition of a live name.
  • Facts for replication: isDroppedPeerGeneration, isNodeLocalTable and replicateIsFalse, which reads the catalog row first.
  • withCatalogWrite: the RocksDB lock or the LMDB write transaction, one helper for both.
  • The createdBefore bound: written once per unstamped row, read, at each load.
  • Drop announcement: epoch and broadcast at commit, the other threads' handler, tableDropEpoch, and recordTableDrop's lock-free fast path.
  • dropTableMeta keeps live and dropping rows, and the test switch.
• resources/Table.ts
  • dropTable's options and boolean result, keptFromPeer, recordPeerDrop.
  • The recheck under the lock, the kept path, the stale-class re-dispatch.
  • The LMDB in-progress mark and its release when cleanup throws; one call for retireRocksStores, whose false no longer means only a pending retirement.
• dataLayer/schema.ts: provenance and the droppedTime scrub, a peer's drop of a table already gone, a node-local drop is not forwarded, the "was not dropped" reply, a cleanup failure is logged.
• dataLayer/harperBridge/ResourceBridge.ts: a drop is a peer's only through REPLICATED_FROM.
• server/serverHelpers/serverUtilities.ts: operation() takes replicatedFrom from its context.
• utility/hdbTerms.ts: the REPLICATED_FROM symbol.
• utility/errors/hdbError.ts: TableGenerationDroppedError, a 409 with code TABLE_GENERATION_DROPPED.
• resources/DESIGN.md: the lifecycle note gains where the rule is enforced, and is rewritten as one description with Preserve 5.2 rollback compatibility for first-time table creates #3118's: untimed name history and store names and 5.2 rollback; DESIGN.md: its index line.

❓ Your call: merge order. Core alone changes nothing for forwarded drops: harper-pro main never sent peer provenance (its operation connection has no node name), so forwarded drops already run as local-only drops there. The conditional peer drop takes effect when harper-pro#956 supplies the context. Merge this first, then #956 with its pointer bump.

❓ Your call: the forwarding decision for a node-local drop is read just before the drop, and the marker decision is made under the lock. A replicate flip by another thread during the drop's awaits makes them disagree. Either way the result is benign: the drop is forwarded as it was issued, or the marker reaches peers through the schema exchange. Carrying the locked decision out of dropTable would change its result type again, so it is left as is.

✅ Verification

On head 6ec39b226, rebased onto main at 87394e7c3 (which includes #3118):

• dropTableLifecycle.test.js with Preserve 5.2 rollback compatibility for first-time table creates #3118's dropTableGeneration.test.js: 56 passing on RocksDB, 39 on LMDB (HARPER_STORAGE_ENGINE=lmdb; the rest skip on that engine).
• Full test:unit:resources: 4087 passing on RocksDB and 3090 on LMDB, none failing. test:unit:main: 6837 passing, none failing. These full runs were on 19e28e894; 6ec39b226 changes only one test assertion and three comments, and the two lifecycle files above were re-run on it.
• Preserve 5.2 rollback compatibility for first-time table creates #3118's 5.2 round trip, integrationTests/upgrade/first-create-downgrade.test.ts against harper 5.2.15: 1 of 1 passing, not skipped. This and harper-pro#956's cluster suite are the kill-and-restart coverage. The lifecycle unit tests stand in for a crash by reloading in-process.
• check:design-docs, prettier and oxlint pass.
• New cases cover:
  • Create: refused for a stale stamped or unstamped peer generation; an unstamped peer create stored at 0 and never backfilled; a stale stamped definition of a live table refused before merging.
  • Peer drop: a newer generation is kept and its marker recorded; the lock-time recheck keeps a table stamped between the pre-check and the lock (a deterministic interleaving in closeMaintenance); a node-local table is never retired, and its own drop leaves no marker and is not forwarded.
  • Provenance and catalog: replicate read from the catalog first, with system-table exclusions kept; dropTableMeta keeps live and dropping rows; provenance comes from server.operation's context, a body's claim is a client's drop, and a client's droppedTime is ignored.
  • Lifecycle facts: isDroppedPeerGeneration for equal, 0 and undefined stamps and for a drop still completing; createdBefore written once and kept; a recorded marker advances the drop epoch.
  • Composition with Preserve 5.2 rollback compatibility for first-time table creates #3118: a node-local drop's untimed name history reads as no drop, and a stale peer create is refused before any create journal or store; a kept generation journals nothing and survives a reload; the announced marker carries the peer's exact time.
  • LMDB drop mark: a drop whose table cleanup throws leaves its tombstone for the next load to complete. Without the release it fails on LMDB, because the leaked mark makes every reload on that thread skip the tombstone.
• unitTests/dataLayer/schema.test.js: its bridge stub resolves true, matching dropTable's new result.
• Before the rebase, against core main with the sources swapped back and dist rebuilt, 11 of the first 24 lifecycle tests failed: every new behavior test, plus the tests updated for the new API. On that main, the LMDB adjacent-suite combination failed 6 of 6 runs from the drop race fixed here.
• End to end, harper-pro#956 with core at this head (6ec39b226), built in its own worktree and node_modules:
  • dropTableOfflinePeer.test.mjs: 8/8, scenarios 1, 1b, 2/2b, 2c, 3, 4, 5 and 6. Scenario 3's pre-stamp node runs with HARPER_TEST_OMIT_TABLE_LIFECYCLE. Before this work, scenarios 5 and 6 failed on harper-pro#956's previous head.
  • Adjacent cluster suites replicateFalseFullCopy, excludeTablesReplication, protocolCapabilityRegistry and selectiveTableSubscription: 7/7. The replication unit suite: 1214 passing.
• Pre-push review after the rebase: one full round and one delta (Codex graded, Gemini, Cursor Composer, Harper domain adjudication). Its two findings original to this round, a loosened assertion and three comments, are fixed. Three Gemini findings were refuted against the code. The open findings from earlier rounds are unchanged.

Refs #1212

Signed: Claude Opus 5.5 (rebase onto #3118 and its composition trace by Claude Opus 5.5)

_{🤖 Generated by Anthropic Claude (Opus 5.5) in Claude Code; posted via @kriszyp.}

🤖 Generated with Claude Code

https://claude.ai/code/session_01KMpTV1zehf49BFieVUy7d1

Related PRs: #3118 overlaps (merged and now this branch's base; the composition is traced above), #3124 overlaps (the v5.3 backport of #3118; a backport of this PR would carry the same composition), #2155 overlaps (the tombstone write in dropTable), #2901 overlaps (the same DESIGN sections; its note that an offline peer re-creates a dropped table goes stale once this and harper-pro#956 land), #2962 overlaps (merged; the stamps and markers this enforces), 8 others independent
Complexity: complicated

_{Review-Coverage: authored=claude; ran=cursor-composer,codex,gemini; adjudicated=domain; declined=cursor-grok,cursor-kimi,cursor-muse; rounds=11; full=2 @ 6ec39b2}

_{Review-Attention: deep ~50m (critical: ResourceBridge.ts, schema.ts +2; decisions: peer-provenance-symbol, unstamped-peer-create-at-zero, core-retires-unstamped-on-peer-drop, kept-drop-reply-shape, test-env-flag-in-production, do-less-alternative) @ 6ec39b2}

[gemini-code-assist]

Code Review

This pull request enhances the table drop lifecycle and replication mechanism by introducing lifecycle stamps and catalog-locked enforcement for table creation and deletion. It ensures that peer drops do not retire node-local tables, and introduces a REPLICATED_FROM symbol to safely track forwarded operations from peer nodes. Additionally, it implements boundPreStampCreations to bound pre-stamp table creations during catalog loading, and updates the thread communication to broadcast table drop markers. A review comment identifies a critical issue in boundPreStampCreations where snapshot.key is accessed on the primary attribute object, which does not inherently carry a key property, potentially causing attributesDbi.getSync(undefined) to fail.