Skip to content

Live subscription can deliver a CREATE audit record as type: "delete" when the entry is already gone #2439

Open

Description

@kriszyp

Summary

The live subscription listener can deliver a record's CREATE audit record to subscribers as type: "delete", stamped with the create's own version. This is a wrong event rather than a missing one, and it reaches all three live surfaces (MQTT, SSE, WebSocket).

Mechanism (verified on origin/main @ fb762a365)

The per-key listener registered via addSubscription (from resources/Table.ts subscribe(); listener body at Table.ts:4230-4265) does not trust auditRecord.type. For any non-message/reload/end_txn audit record it recomputes the outgoing type from the current primary-store state:

const entry: Entry = primaryStore.getEntry(id);
if (entry) {
    if (entry.version !== (auditRecord.recordVersion ?? auditRecord.version)) return;
    value = entry.value;
    type = entry.metadataFlags & INVALIDATED ? 'invalidate' : value ? 'put' : 'delete';
} else {
    type = 'delete';               // Table.ts:4255
}
event.version = auditRecord.version;   // Table.ts:4262 — the ORIGINAL record's version

If entry is missing at delivery time — because the record was deleted or evicted after the CREATE committed but before that audit record's turn came up — a put (create) audit record is delivered as type: "delete" while carrying the create's version.

The window is real, not theoretical: dispatch comes from a batched, deferred audit-log tail. resources/transactionBroadcast.ts notifyFromTransactionData uses NOTIFY_BATCH_SIZE = 256 (:138), defers via setImmediate (:58), and calls subscription.listener(recordId, auditRecord, timestamp, beginTxn) at :222 — so a fast subsequent delete or a TTL eviction can already have removed the entry by the time the create's callback fires.

Consequence

A subscriber doing event-sourced replay never sees the create and instead sees a delete carrying the create's version — so ordering and identity are both wrong, not merely incomplete. Create-then-fast-delete and create-under-TTL are ordinary workloads, so the trigger is structural rather than exotic.

Confirming experiment (for whoever picks this up)

Instrument Table.ts:4248 (primaryStore.getEntry(id)) to log when it returns falsy inside this listener, alongside auditRecord.type. Then drive create-then-immediate-delete (or create-then-TTL-evict) on the same key under an open collection subscription and watch for auditRecord.type === 'put' reaching the else { type = 'delete' } branch. That log line, correlated with the delivered event's version equalling the create's version, distinguishes this misclassification from a coincidental legitimate delete.

Related

Sibling in the same file family: #2311 (subscription replay silently drops events committed after the audit cursor terminates) — that one is a silent drop in a different branch; this one is a wrong event.

Found by the qa-explorer QA loop (finding QA-675, flagged there at MEDIUM confidence and documented in the spec rather than asserted); mechanism since confirmed from source. Dup-searched in harper and harper-pro — nothing tracks it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Priority

    P1

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions