Stamp table generations and keep a durable drop marker so a peer that missed a replicated drop_table cannot bring the table back

[kriszyp]

⊙ Problem

A replicated drop_table is a one-shot operation RPC, and the dropping tombstone is removed once the drop completes, so nothing durable records that a table was dropped. A peer that was offline for the drop keeps its copy and, on rejoin, brings the table back through harper-pro's schema handshake; its rows then leak by name into any same-name recreate. This is the cluster half of Dropped table can be resurrected after cluster restart — drop is not a durable, replayable txn-log event (#1212); the single-node half is Add a restart regression test: dropped tables stay dropped and same-name recreates start empty (#2901).

❓ Your call: Is the cluster half warranted at this size, and does it merge with its companion? isDeadGeneration has no caller in core, so this PR alone changes no rejoin outcome. The rule takes effect when Retire a table generation a peer dropped while this node was offline instead of resurrecting it on rejoin (harper-pro#956) calls it.

💡 Solution

Every table generation gets a createdTime, and every drop leaves a durable droppedTime that survives completion as a dropped row. A generation is dead when its createdTime is strictly below the newest drop time of its name, so every node applies the same rule to an offline peer's old copy. Core supplies the primitives (getTableDrops, recordTableDrop, onTableDropRecorded); the companion PR, Retire a table generation a peer dropped while this node was offline instead of resurrecting it on rejoin, exchanges them and enforces the rule.

User-visible change in core: a forwarded drop of a table this node already lacks answers success and records the origin's time, where it previously failed with a 404.

Refs #1212. Companion: Retire a table generation a peer dropped while this node was offline instead of resurrecting it on rejoin (harper-pro#956), which carries the closing reference.

⚖️ Alternatives

Planning review: Framing-Verdict: better-alternative-exists, resolved by overrule.

• Opaque incarnation ID per generation (framing round 1; overruled). Schema-defined tables are created by each node's own declareTable, and create_table is not a replicated RPC, so one live table commonly has a different ID on each node. Retiring one node's ID would still admit a copy another node minted independently. Time ordering retires every copy created before the drop, whoever minted it. Its clock-skew exposure is the one every replicated record already has under last-writer-wins versions, narrowed by the causal floor.

🔧 Changes

Product and architecture tour

One comparison decides whether a generation is live

A generation carries two stamps. createdTime is written at create and never changes. droppedTime is the drop's time: it sits on the tombstone while a drop runs and on a dropped row once the drop completes.

Lifecycle of one generation

A same-name recreate is stamped after the newest dropped row, so the drop it follows never retires it.

stateDiagram-v2
    [*] --> Live: declareTable stamps createdTime
    Live --> Tombstone: dropTable stamps droppedTime
    Tombstone --> DroppedRow: completion promotes the time before removing the tombstone
    DroppedRow --> Live: same-name create stamped after the dropped row
    DroppedRow --> DroppedRow: a newer drop overwrites the time

Loading

Both stamps come from one clock and order after what they follow

databases.ts:664

tableLifecycleTime(after)
	now = getNextMonotonicTime()
	if after is finite and now <= after → after + LIFECYCLE_STEP   ⚠ a stamp orders after what it follows
	→ now

isDeadGeneration(createdTime, droppedTime)
	→ (createdTime is finite ? createdTime : 0) < droppedTime     // strict: an equal stamp survives

• Floor. A stamp taken after a known fact is raised above it, so a peer whose clock runs ahead cannot make a live generation read as dead. The tests cover a recreate after a drop that is ahead of the local clock, and a drop of a generation stamped by a faster clock.
• Missing stamp reads as 0. A generation without createdTime, from a build before this change, is older than any drop of its name.

A drop writes its time first and promotes it before removing the tombstone

Every drop writes its tombstone with droppedTime before any destructive work. Each completion path promotes that time to a dropped row before it removes the tombstone, so a crash between the two writes cannot lose the time. The tombstone write goes through dropTable at Table.ts:2925.

How a dropped name reaches a peer that was offline

A forwarded drop carries the origin's time, so every peer records the same one. The rejoin step is harper-pro's schema handshake, which this PR does not implement.

sequenceDiagram
    participant Origin
    participant Peer
    participant Offline
    Origin->>Origin: tombstone with droppedTime
    Origin->>Peer: forwarded drop with origin droppedTime
    Peer->>Peer: tombstone, then dropped row
    Note over Offline: misses the drop, keeps its old copy
    Offline->>Peer: rejoin with schema handshake
    Peer-->>Offline: dropped row for the name
    Offline->>Offline: isDeadGeneration retires the old copy

Loading

Completion path	Where	Dropped row written before the tombstone is removed
First completion closure in dropTable	Table.ts:3028	Yes
Second completion closure in dropTable	Table.ts:3159	Yes
completeInterruptedDrop, at boot load or a same-name create	databases.ts:5868	Yes
dropTableMeta, for a tombstone still in place	databases.ts:5888	Yes, when dropping is set

A joining drop raises the tombstone's time and never lowers it

dropTable on a tombstone that already has dropping set

Table.ts:2911

dropTable(options) on a tombstone with dropping set
	if options.localOnly → joinedTime = none
	else if options.droppedTime is finite → joinedTime = options.droppedTime
	else if tombstone.droppedTime is unset → joinedTime = tableLifecycleTime(createdTime)
	else → joinedTime = none
	if joinedTime is set and above tombstone.droppedTime (or the tombstone has none) → write the tombstone with joinedTime   ⚠ a time only moves forward
	→ true                                                                                                                     // joins the drop in flight

⚠️ Look hardest: this is the only branch where a tombstone's time rises after its drop began. A replicated drop that joins a bare tombstone stamps it, so the completion promotes a time. A local-only drop that joins leaves the time as it was.

Drop requests take one of four paths

ResourceBridge.dropTable maps the operation's flags onto the table drop. replicated: false from a client is node-local. A forwarded drop is recognized by replicatedFrom together with a finite droppedTime, and the origin's time is used as it is.

Request	Flags	Time the drop carries	Dropped row
Client drop	none	This node's clock, floored above the generation's createdTime	Written on completion
Client drop, node-local	replicated: false	None; the tombstone stays bare	None
Forwarded drop	replicatedFrom and a finite droppedTime	The origin's droppedTime	Written on completion
Forwarded drop without a finite time	replicatedFrom only	This node's clock, as for a client drop	Written on completion

The last row has no test case; the test list covers the other three. It is stamped by this node's clock rather than refused.

Before	After
A forwarded drop of a table this node already lacks fails with a 404 and records nothing.	The drop records the origin's time and answers success, with a message that the table was already dropped.

The already-gone path is in dataLayer/schema.ts:231. It answers success even when recordTableDrop records nothing.

❓ Your call: recordTableDrop records nothing when the database is not loaded, and the path still answers success. Replication reaches it only after the schema load, so the case is believed unreachable. The alternative is to report a failure there. Changing it is one branch in dataLayer/schema.ts.

❓ Your call: replicatedFrom is the only proof that a drop came from a peer, and the operation schema allows unknown keys (validation/validationWrapper.ts:87). Any principal with the DROP_TABLE permission can therefore send replicatedFrom with a droppedTime of its choosing. That time retires every generation of the name stamped below it on this node, and on peers once harper-pro exchanges markers. Options: keep this trust, as the code does now, or accept a forwarded time only from the replication channel. This PR does not change it.

Marker writes are read-compare-write under the catalog lock

Every dropped-row write goes through one function. It compares the candidate time with the stored row, keeps the newer time, and re-reads the live tombstone inside the same critical section, so a copy read earlier cannot be promoted. A time only moves forward, and a row is overwritten only by a newer drop.

Read-compare-write of a dropped row

databases.ts:687

writeTableDropMarker(rootStore, dbis, table, droppedTime, tableId, exclusive, tombstoneKey)
	write():
		if tombstoneKey → live = dbis[tombstoneKey]   ⚠ re-read inside the section
			if live.dropping and live.droppedTime is above droppedTime → droppedTime = live.droppedTime
		if droppedTime is not finite → false          ⚠ no time, no dropped row
		if dbis['/dropped/' + table].droppedTime >= droppedTime → false
		dbis['/dropped/' + table] = { table, droppedTime, tableId? }
		→ true
	RocksDB: exclusive ? run write() under the catalog lock : run write() (the caller holds the lock)
	LMDB: transactionSync(write)
	if written → emit tableDropRecorded after commit   // listeners re-send schemas, which read this catalog

The dropped row a completed drop leaves behind

databases.ts:709

// key '/dropped/Orders' in the __dbis__ store; values illustrative
{ table: 'Orders', droppedTime: 1791300000.004, tableId: 7 }

The row outlives a same-name recreate and is never collected. Removing it would stop it retiring a stale copy that a peer delivers later.

❓ Your call: Dropped rows are kept forever, one per dropped name per database. Growth is bounded by distinct names, and the boot scan and getTableDrops walk every row. Is that acceptable at expected name counts?

Where a generation's createdTime comes from

Generation created by	createdTime	Source
This node's declareTable, no peer stamp	This node's clock, floored above the newest dropped row of the name	databases.ts:4309
A peer's propagated definition	The peer's value, kept as is	TableDefinition.createdTime, so every copy of a generation shares one stamp
A build that stored no stamp	Absent until stampTableCreatedTime writes it once from a peer's stamped definition	databases.ts:789

The backfill writes only an absent stamp, and the caller decides which peer definition supplies it. An unstamped generation reads as older than any drop of its name, so the next drop of that name retires it.

❓ Your call: Is retiring an unstamped generation on the next drop of its name the right default, or should such a generation survive until a stamped definition backfills it? The current behavior follows from the rule as written.

What this change leaves out

Case	Behavior	Why
Tombstone written before stamps existed (no droppedTime)	No dropped row, no invented time	A time made up at completion could postdate a peer's live recreate
Node-local drop (replicated: false)	Tombstone stays bare, no dropped row	Node-local tables keep the node-local contract
drop_database	Out of scope	It removes the root store and its catalog with it

A tombstone from before this change is left alone until a stamped drop of that name happens. The alternative is to synthesize a time at completion; this PR does not, because that time could retire a peer's live recreate.

❓ Your call: Pre-stamp tombstones keep no marker. Say so if you would rather synthesize a time there, and accept the risk above.

Changed files

• DESIGN.md — root index entry for the drop note, retitled to cover lifecycle stamps.
• dataLayer/harperBridge/ResourceBridge.ts — maps the drop request's flags onto dropTable; see the routing table above.
• dataLayer/schema.ts — a forwarded drop of an already-gone table records its time and succeeds.
• resources/DESIGN.md — the drop-tombstone note now covers lifecycle stamps. It was tightened to fit the file's line budget; Add a restart regression test: dropped tables stay dropped and same-name recreates start empty edits the same section, so whichever lands second rebases a small conflict.
• resources/Table.ts — createdTime on the table class (1673); dropTable options and the joining-drop branch; the tombstone's time (2925); promotion in both completion closures (3028, 3159).
• resources/databases.ts — the clock and comparison rule (664, 668); the marker write path (687); recording and reading (734, 744); the create-time stamp (4309); the stamp backfill (789); the pending drop time (812); promotion (822); the boot-scan skip of dropped rows (1480); interrupted-drop promotion (5868); dropTableMeta (5888).
• unitTests/resources/dropTableLifecycle.test.js — 17 cases; see Verification.

✅ Verification

• unitTests/resources/dropTableLifecycle.test.js (new, 17 cases) covers: create and supplied stamps survive a reload; a drop marker outlives a same-name recreate and a reload; a peer's drop time is applied, moves forward only, and is announced once readable; a recreate after a drop ahead of the local clock reads as newer; a generation stamped by a faster clock is retired; a pre-stamp interrupted drop completes without an invented time; a joining drop keeps the newer time; a node-local drop leaves no row; the operation flags map onto the drop; a replicating drop that joins a node-local one stamps the bare tombstone; a missing stamp is backfilled once; a forwarded drop of an already-gone table records its time; dropTableMeta promotes a tombstone it removes; the load completes an interrupted drop and promotes its tombstone. Passes on RocksDB and on HARPER_STORAGE_ENGINE=lmdb, alongside dropTableGhost.test.js and dropTableGeneration.test.js.
• npm run check:design-docs passes; prettier and oxlint are clean on the changed files.
• End to end: the companion's integrationTests/cluster/dropTableOfflinePeer.test.mjs runs against this core. Its six scenarios all pass: a connected drop with graceful and SIGKILL restarts; a peer offline for the drop, then both restarted; a drop and recreate while the peer is offline; a pre-stamp peer rejoining; a node-local drop.

Signed: Claude Sonnet 5

Related PRs: unchecked (no related-PR snapshot in the review receipt)
Complexity: complicated

🤖 Generated with Claude Code

https://claude.ai/code/session_01CSitdq3dnTG5EWQeCwVoSi

_{Review-Coverage: authored=claude; ran=cursor-composer,gemini; adjudicated=domain; blocked=codex(out-of-budget),claude(fallback)(out-of-budget); declined=cursor-grok,cursor-kimi,cursor-muse; rounds=5; full=1 @ 8f147ef}

_{Review-Attention: deep ~120m (raised: unmeasured diff) @ 8f147ef}

[gemini-code-assist]

Code Review

This pull request introduces table lifecycle stamps (createdTime and droppedTime) and durable drop markers to prevent offline peers from resurrecting dropped tables during schema handshakes (harper#1212). The changes span across ResourceBridge.ts, schema.ts, Table.ts, and databases.ts, and are accompanied by a comprehensive test suite in dropTableLifecycle.test.js. The review feedback recommends replacing implicit type coercion checks in range comparisons (where droppedTime might be undefined) with explicit checks to avoid potential bugs and improve code readability.