fix: multi-process-safe audit integrity chain via atomic per-org append

[rares04]

Problem

The tamper-evident integrity audit (createGovernance({ integrityAudit })) held the chain's sequence and previousHash as process-local state, resumed from the durable head only once at boot (loadChainHead, guarded by state.loaded), then advanced an in-process counter under an in-process lock.

Under any multi-process deployment — Kubernetes replicas, a pm2 cluster, or serverless instances all writing to one shared database — that assumption breaks:

• Collision → silent drop. Two processes derive the same sequence for the same org. One INSERT wins; the other loses to the UNIQUE (COALESCE(organization_id,''), integrity_sequence) index (Postgres 23505). With the default onFailure: 'allow', that governance decision's tamper-evident record is silently dropped.
• Fork (even without a collision). The per-process previousHash links to that process's last write, not the true durable head, so the hash chain forks across processes and isn't sound under multi-process regardless of collisions.

storage-postgres.ts createAuditEventWithIntegrity() was a single INSERT with the pre-computed sequence — no ON CONFLICT, no retry, no head re-read.

Fix

Sequence allocation and previous-hash derivation are now atomic against the database, per org, on every write — never from process-local state.

• New optional GovernanceStorage.appendToAuditChain(event, computeIntegrity) — the documented storage contract for atomic chain appends. The adapter, under a per-org lock spanning the whole operation, reads the org's durable head, calls computeIntegrity(head) (the HMAC signing key stays in the SDK core, never in storage), and persists event + integrity as one indivisible write.
• Postgres: BEGIN ISOLATION LEVEL READ COMMITTED → pg_advisory_xact_lock(<classid>, hashtext(org)) → head SELECT → INSERT → COMMIT. The advisory lock (released at COMMIT) serialises each org's chain against itself across all writers, including separate processes; unrelated orgs proceed in parallel. READ COMMITTED is pinned deliberately: it's what guarantees the post-lock head SELECT sees the prior writer's just-committed row (a fixed-snapshot isolation level would re-read stale and re-collide). The two-arg lock form namespaces the key under a fixed classid so other advisory-lock users of the same database can't contend. A pool exposing only query() (no transaction handle) falls back to a bounded read-head → insert → retry-on-23505 loop with jittered backoff, which is also multi-writer-safe.
• Memory adapter: a per-org async lock spanning head-read → compute → write. Correct within a single process; memory is not shared across processes, so it is single-process by design (documented).
• createGovernance() prefers appendToAuditChain when present; the previous createAuditEventWithIntegrity + process-local sequence path remains only as a single-writer fallback for third-party adapters that predate this method.

Why advisory lock over ON CONFLICT-retry

Retry alone doesn't fix the head-read race or the chain fork: each retry still needs to re-read the durable head and re-hash from it, so a naive retry that reuses process-local previousHash produces a forked chain. The advisory lock makes head-read → hash → insert indivisible in one pass. The retry-on-23505 path is kept only as the fallback for pools without connect() — there it does re-read the head each attempt, so it's correct; the bound is deterministic, not probabilistic (every 23505 means a competitor committed the derived sequence, so a writer loses at most once per concurrent same-org contender).

Verification order = sequence, not wall clock

createdAt is stamped before the append lock allocates the sequence, so under concurrent writers a lower sequence can carry a later timestamp (lock-wait inversion, cross-process clock skew). verifyAuditIntegrity() and integrityChain.export() now order by the HMAC-covered sequence (createdAt only tiebreaks legacy forked rows) — the old createdAt-first sort could report a valid multi-writer chain as tampered. Tamper-safe: the sequence is inside the signed hash, so forging it still fails the hash / previous-hash checks.

API surface unchanged

Existing consumers need no code change beyond upgrading — createGovernance({ integrityAudit }) picks up the safe path automatically once its storage adapter (a real pg.Pool) exposes connect(). Canonical hash form, per-org scoping, verify(), export(), and stats() behaviour are unchanged.

One behavioural note for reviewers: the transactional path now holds a pooled connection across BEGIN → lock-wait → SELECT → hash → INSERT → COMMIT (the old path was grab/insert/release per statement). A same-instant burst across many orgs would queue on the pool and degrade to connectionTimeout / statement_timeout → the same onAuditError path (not a silent drop).

Multi-process guidance + docs (this PR)

The fix is generalised for the SDK's whole audience, not only the deployment that surfaced it:

• README gains a "Multi-process deployments" section: what appendToAuditChain guarantees, a per-adapter support table (Postgres / memory / third-party fallback), and how a custom storage adapter should implement atomic append (row lock, advisory lock, serializable txn, or CAS-retry on its uniqueness constraint).
• The standalone createIntegrityAudit() wrapper (audit-integrity.ts) is the other entry point that keeps its chain in process memory — the same unsoundness class on a separate surface. It holds no storage handle to append against, so it cannot reuse the contract; it is now documented as single-process only (prominent JSDoc + README + a regression test asserting a second wrapper starts its own sequence at 1 rather than resuming the durable head), with consumers pointed at createGovernance({ integrityAudit }) for durable multi-process audit.
• integrityChain.stats() still reports this process's last-written sequence/hash (a process-local cache); export() + verifyAuditIntegrity() are the authoritative durable readers. A DB-backed stats() is deferred — it would turn a sync method async (a breaking signature change) — and noted in the CHANGELOG.

Tests

npm test — 1448 pass, tsc --noEmit clean.

• audit-chain-multiwriter.test.ts — the load-bearing regression test: two createGovernance instances sharing one storage (two-writer model) fire N concurrent interleaved writes → exactly N rows, contiguous 1..N sequences (no duplicates/gaps), and the interleaved chain verifies standalone. Also covers two writers × two orgs, and a timestamp-inverted-vs-sequence chain (fails on the old createdAt sort).
• audit-chain-append-postgres.test.ts — asserts the Postgres SQL shape (BEGIN ISOLATION → advisory-lock → head SELECT → INSERT → COMMIT), DB-head-derived sequencing across sequential appends, an orchestration run of two instances over one pool, retry-fallback recovery from a 23505, and dead-connection release on ROLLBACK failure.
• audit-integrity.test.ts — new: documents the createIntegrityAudit() single-process boundary.

Verified against a real Postgres (not just the mock)

The committed pg test models pg_advisory_xact_lock as an in-process mutex (the repo harness has no Postgres), so it validates orchestration + SQL shape but can't prove real cross-connection locking. The two-writer scenario was additionally run out-of-band against an ephemeral real Postgres 14 with two independent pg.Pools (real connect(), real advisory lock, real unique index), 60 concurrent writes to one org:

Path	Attempted	Persisted	Dropped (23505)	Chain
Before this PR (legacy path)	60	30	30	forked
After this PR	60	60	0	contiguous 1..60, valid

The "before" row reproduces the failure mode; the "after" row demonstrates the fix on real cross-connection locks. (Throwaway harness — not committed.)

Rollout note

The advisory lock only protects writers that take it. During a mixed-version rolling deploy, pre-0.18.2 processes still allocate from their process-local counters and can collide with or fork past locked writers. Replace all writers together; expect residual 23505 warnings until the last old process drains.

Compatibility

Backward compatible. appendToAuditChain is an optional interface method — adapters that omit it fall back to the existing single-writer path. Canonical hash form and per-org scoping are unchanged, so existing chains keep verifying. Versioned as a patch (0.18.2) per the repo's convention (fix + optional interface addition).

🤖 Generated with Claude Code

---

Note

High Risk
Changes audit integrity under concurrent production writers (silent event loss / chain fork before); rolling deploys need all writers on 0.18.2+ or collisions can persist during mixed versions.

Overview
Release 0.18.2 fixes tamper-evident audit (integrityAudit) when several processes share one store: process-local sequence/previousHash used to collide on the per-org unique index (silent drops) and fork the HMAC chain.

New storage contract appendToAuditChain — adapters read the durable head under a per-org lock, call back into the SDK for HMAC metadata, and insert atomically. createGovernance() uses it when present; legacy createAuditEventWithIntegrity + in-process counters remain for single-writer adapters, with a one-time onAuditError warning if durable integrity exists without atomic append.

Postgres implements transactional append (READ COMMITTED, pg_advisory_xact_lock, head SELECT scoped with COALESCE(organization_id,'')) plus a query-only pool fallback (bounded 23505 retries). Memory gets per-org async locks. Export/verify now sort by integrity sequence (not createdAt first), read integrity whenever getAuditIntegrity exists, and document that createIntegrityAudit() stays in-process only.

README adds a Multi-process deployments section; extensive regression tests cover multi-writer chains, partial adapters, and Postgres orchestration.

^{Reviewed by Cursor Bugbot for commit fadddbd. Bugbot is set up for automated code reviews on this repo. Configure here.}

[rares04]

bugbot run

[rares04]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 42c4a20. Configure here.}

[rares04]

bugbot run

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit fadddbd. Configure here.}

[scotty595]

Approving — this is a correct, well-tested fix for a real soundness bug in the 0.18.0 per-org chains under multi-process writers (silent 23505 drops + chain fork). Reviewed the load-bearing logic, not just the description:

• appendToAuditChain contract keeps the HMAC key in SDK core via the computeIntegrity callback — clean separation.
• Postgres advisory-xact-lock + READ COMMITTED reasoning is correct (a fixed-snapshot level would re-read a stale head and re-collide); poisoned-connection release(rollbackErr) on ROLLBACK failure is proper hygiene.
• Sorting export/verify by HMAC-covered sequence instead of createdAt is a genuine fix — the old createdAt-first sort could flag a valid multi-writer chain as tampered.
• Real-Postgres 60-writer before/after (30-dropped → 0-dropped) is the right evidence.

Action items on the consumer side (lua-core, multi-pod — the exact vulnerable case):

1. Bump lua-core to 0.18.2 once published; our 0.18.0 chains in staging are currently unsound under concurrency.
2. Our pnpm patch re-applies cleanly — this diff doesn't touch createAgent or mastra-processor.js (the only things we patch); just re-point the patch key to 0.18.2 and git apply --check.
3. Rolling-deploy: all pods must reach 0.18.2 together; expect transient 23505 warnings until the last 0.18.0/0.18.1 pod drains.

Watch (non-blocking): per-org audit writes now serialize through the advisory lock and hold a pooled connection per write; since enforce awaits the chain write with integrityAudit on, worth a load check on the governance pool under high per-org volume.