fix: multi-process-safe audit integrity chain via atomic per-org append

README.md

@@ -79,9 +79,12 @@ is exactly what it does and does not do:
   `integrityAudit: { signingKey }` for tamper-evident audit. Since 0.12
   the chain is persisted durably (survives process restart) when the
   storage adapter supports `createAuditEventWithIntegrity` (memory and
-  Postgres adapters both do). HMAC chains are still only tamper-evident
-  to holders of the signing secret — rotate and pair with an external
-  anchor if you need adversary-grade non-repudiation.
+  Postgres adapters both do). Since 0.18.2 that chain is also
+  **multi-process-safe** when the adapter implements `appendToAuditChain`
+  (Postgres) — see [Multi-process deployments](#multi-process-deployments).
+  HMAC chains are still only tamper-evident to holders of the signing
+  secret — rotate and pair with an external anchor if you need
+  adversary-grade non-repudiation.
 - **Cloud `register()` is a synthetic confirmation** — the API auto-registers
   on first `enforce()`.
 - **No built-in red team / jailbreak harness.** Use inspect-ai, PyRIT, or
@@ -453,6 +456,62 @@ didn't invoke `enforce()` or `recordOutcome()` itself.
   `'block'` to reject the enforce() call instead when you can't tolerate
   gaps.
 
+#### Multi-process deployments
+
+If more than one process writes to the **same audit store** — Kubernetes
+replicas, a `pm2` cluster, or serverless instances all pointed at one
+Postgres database — the chain must allocate each event's `sequence` and
+`previousHash` from the **current durable head**, not from process-local
+state. Otherwise two processes derive the same sequence for the same org
+(one `INSERT` wins, the other is dropped by the unique index) and their
+per-process `previousHash` forks the chain.
+
+This is the job of the optional storage-contract method
+`appendToAuditChain(event, computeIntegrity)`. The adapter, under a per-org
+lock that spans the whole operation, reads the org's durable head, calls back
+into the SDK to compute the HMAC (the signing key never leaves the SDK core),
+and persists the event + integrity as one indivisible write:
+
+```typescript
+import { createGovernance } from 'governance-sdk';
+import { createPostgresStorage } from 'governance-sdk/storage-postgres';
+
+const gov = createGovernance({
+  storage: await createPostgresStorage({ pool }), // pg.Pool — real transactions
+  integrityAudit: { signingKey: process.env.AUDIT_SECRET! },
+});
+// Every process using this config appends atomically against the shared DB —
+// no duplicate-sequence drops, no per-process chain fork.
+```
+
+`createGovernance()` uses `appendToAuditChain` automatically whenever the
+storage adapter provides it. **Support by shipped adapter:**
+
+| Adapter | `appendToAuditChain` | Multi-process safe? |
+|---|---|---|
+| **Postgres** (`createPostgresStorage`) | ✅ per-org `pg_advisory_xact_lock` transaction (falls back to a bounded `23505`-retry loop for query-only pools) | ✅ across processes sharing the database |
+| **Memory** (`createMemoryStorage`) | ✅ per-org in-process async lock | Single process **by design** — memory is not shared across processes |
+| Third-party adapter without the method | — | Falls back to the legacy process-local-sequence path (correct under a **single writer** only) |
+
+**Custom storage adapters:** to be multi-process-safe, implement
+`appendToAuditChain` so the head-read → compute → insert sequence is atomic
+against concurrent writers (a row lock, an advisory lock, a serializable
+transaction, or a compare-and-set retry on your uniqueness constraint). If you
+can't, leave it unimplemented and run a single writer — the SDK falls back
+safely and warns.
+
+**The standalone `createIntegrityAudit()` wrapper is single-process only.** It
+keeps its chain in process memory and never persists integrity metadata, so it
+forks across processes and loses verifiability across restarts. Use it for
+prototyping and tests; use `createGovernance({ integrityAudit })` (above) for
+durable, multi-process audit.
+
+**Rolling deploys:** the lock only protects writers that take it. During a
+mixed-version window (some processes pre-0.18.2), the old processes still
+allocate from process-local counters and can collide with or fork past the
+locked writers. Replace all writers together and expect residual
+unique-violation warnings until the last old process drains.
+
 ### Kill Switch
 
 Emergency halt for any agent, enforced via a reserved-priority policy rule

packages/governance/CHANGELOG.md

@@ -1,5 +1,107 @@
 # Changelog
 
+## [0.18.2] - 2026-07-15 — Multi-process-safe audit integrity chain
+
+Fixes silent audit-event loss and hash-chain forking when the integrity audit
+(`integrityAudit`) runs behind more than one process (e.g. multiple replicas, a
+`pm2` cluster, or serverless instances sharing one Postgres database).
+
+Previously the chain's `sequence` and `previousHash` were held as
+process-local state, resumed from the durable head only once at boot. Every
+process then advanced its own counter independently, so two processes derived
+the same sequence for the same org — one `INSERT` won and the other lost to
+the `UNIQUE (COALESCE(organization_id,''), integrity_sequence)` index
+(Postgres `23505`), dropping that governance decision's tamper-evident record.
+Even without a collision the per-process `previousHash` forked the chain.
+
+Sequence allocation and previous-hash derivation are now atomic **against the
+database, per org**: the storage adapter reads the current head, derives the
+HMAC, and inserts — all under a per-org lock — on every write. No consumer
+code change is required beyond upgrading; `createGovernance({ integrityAudit })`
+picks up the safe path automatically.
+
+Backward compatible: the canonical hash form, per-org scoping, `verify()`,
+`export()`, and `stats()` are unchanged.
+
+### Added
+
+- `GovernanceStorage.appendToAuditChain(event, computeIntegrity)` — atomically
+  reads the org's durable head, invokes `computeIntegrity(head)` (the signing
+  key stays in the SDK core), and persists event + integrity as one operation.
+  Implemented by the memory adapter (per-org async lock) and the Postgres
+  adapter.
+- Postgres path uses a transaction with
+  `pg_advisory_xact_lock(<classid>, hashtext(org))` → head `SELECT` → `INSERT`
+  → `COMMIT`, serialising each org's chain against itself across all writers
+  (unrelated orgs proceed in parallel). The two-arg lock form namespaces the
+  key under a fixed classid so other advisory-lock users of the same database
+  can't contend with the audit chain.
+- `PgClientLike` type and an optional `connect()` on `PgPoolLike` for the
+  transactional path. Pools exposing only `query()` fall back to a bounded
+  read-head → insert → retry-on-`23505` loop with jittered backoff, which is
+  also multi-writer-safe: every unique-violation means a competitor committed
+  the derived sequence, so a writer loses at most once per concurrent same-org
+  contender (the retry cap absorbs 12-way same-instant contention).
+
+### Changed
+
+- `createGovernance()` prefers `appendToAuditChain` when the storage adapter
+  provides it; the previous `createAuditEventWithIntegrity` + process-local
+  sequence path remains only as a fallback for third-party adapters that
+  predate this method (correct under a single writer).
+- The Postgres chain-head `SELECT` now scopes by `COALESCE(organization_id,'')`
+  — the exact partition of the unique index and the advisory-lock key — so the
+  head read can never disagree with the uniqueness/lock scope (a literal-`''`
+  org and the org-less chain were previously read as different heads while
+  colliding on the same index partition). Matching the index expression also
+  lets the head read walk the unique index directly.
+
+### Fixed
+
+- `verifyAuditIntegrity()` and `integrityChain.export()` now order entries by
+  the HMAC-covered `sequence` (wall-clock `createdAt` only tiebreaks) instead
+  of `createdAt`-first. `createdAt` is stamped before the append lock, so
+  under concurrent writers a lower sequence can carry a later timestamp
+  (lock-wait inversion, cross-process clock skew) — the old ordering could report
+  a valid multi-writer chain as tampered. Sequence ordering is tamper-safe:
+  the sequence is inside the signed hash, so forging it still breaks the
+  hash/previous-hash checks.
+- On the transactional append path, a failed `ROLLBACK` now destroys the
+  pooled connection (`client.release(err)`) instead of returning a dead or
+  aborted-transaction client to the pool.
+- `integrityChain.export()` / `verifyAuditIntegrity()` now read durable
+  integrity whenever the adapter implements `getAuditIntegrity`, no longer
+  gating that read on the legacy `createAuditEventWithIntegrity`. An adapter
+  implementing the new `appendToAuditChain` write contract plus
+  `getAuditIntegrity` (but not the legacy write method) previously exported an
+  empty chain even though its writes persisted integrity durably.
+- A storage adapter with durable integrity but no `appendToAuditChain` now
+  emits a one-time `onAuditError` advisory: it uses process-local sequence
+  allocation, which is multi-process-safe only under a single writer. The
+  README's "falls back safely and warns" guidance previously held only for the
+  older session-local (no `createAuditEventWithIntegrity`) fallback.
+
+### Notes
+
+- `integrityChain.stats()` still reports this process's last-written sequence
+  and hash (a process-local cache), so it can lag another process's writes. The
+  durable chain is authoritative; `export()` + `verifyAuditIntegrity()` read
+  from storage. A DB-backed `stats()` is deferred (it would change the method
+  from sync to async — a breaking signature change); tracked in #39.
+- The standalone `createIntegrityAudit()` wrapper in `audit-integrity.ts`
+  keeps its chain in process memory and is **single-process only** — it is a
+  separate, in-memory construct from the durable
+  `createGovernance({ integrityAudit })` path this fix hardens (the wrapper
+  holds no storage handle to append against). It now carries a prominent
+  single-process warning in its JSDoc and the README "Multi-process
+  deployments" note; use `createGovernance({ integrityAudit })` for durable,
+  multi-process audit.
+- Rolling deploys: the advisory lock only protects writers that take it.
+  During a mixed-version window, pre-0.18.2 processes still allocate from
+  their process-local counters and can collide with or fork past locked
+  writers. Replace all writers together; expect residual unique-violation
+  warnings until the last old process drains.
+
 ## [0.18.1] - 2026-06-26 — Evasion-resistant injection normalization
 
 Hardens `detectInjection()` against obfuscated prompt-injection attacks that

packages/governance/README.md

@@ -79,9 +79,12 @@ is exactly what it does and does not do:
   `integrityAudit: { signingKey }` for tamper-evident audit. Since 0.12
   the chain is persisted durably (survives process restart) when the
   storage adapter supports `createAuditEventWithIntegrity` (memory and
-  Postgres adapters both do). HMAC chains are still only tamper-evident
-  to holders of the signing secret — rotate and pair with an external
-  anchor if you need adversary-grade non-repudiation.
+  Postgres adapters both do). Since 0.18.2 that chain is also
+  **multi-process-safe** when the adapter implements `appendToAuditChain`
+  (Postgres) — see [Multi-process deployments](#multi-process-deployments).
+  HMAC chains are still only tamper-evident to holders of the signing
+  secret — rotate and pair with an external anchor if you need
+  adversary-grade non-repudiation.
 - **Cloud `register()` is a synthetic confirmation** — the API auto-registers
   on first `enforce()`.
 - **No built-in red team / jailbreak harness.** Use inspect-ai, PyRIT, or
@@ -453,6 +456,62 @@ didn't invoke `enforce()` or `recordOutcome()` itself.
   `'block'` to reject the enforce() call instead when you can't tolerate
   gaps.
 
+#### Multi-process deployments
+
+If more than one process writes to the **same audit store** — Kubernetes
+replicas, a `pm2` cluster, or serverless instances all pointed at one
+Postgres database — the chain must allocate each event's `sequence` and
+`previousHash` from the **current durable head**, not from process-local
+state. Otherwise two processes derive the same sequence for the same org
+(one `INSERT` wins, the other is dropped by the unique index) and their
+per-process `previousHash` forks the chain.
+
+This is the job of the optional storage-contract method
+`appendToAuditChain(event, computeIntegrity)`. The adapter, under a per-org
+lock that spans the whole operation, reads the org's durable head, calls back
+into the SDK to compute the HMAC (the signing key never leaves the SDK core),
+and persists the event + integrity as one indivisible write:
+
+```typescript
+import { createGovernance } from 'governance-sdk';
+import { createPostgresStorage } from 'governance-sdk/storage-postgres';
+
+const gov = createGovernance({
+  storage: await createPostgresStorage({ pool }), // pg.Pool — real transactions
+  integrityAudit: { signingKey: process.env.AUDIT_SECRET! },
+});
+// Every process using this config appends atomically against the shared DB —
+// no duplicate-sequence drops, no per-process chain fork.
+```
+
+`createGovernance()` uses `appendToAuditChain` automatically whenever the
+storage adapter provides it. **Support by shipped adapter:**
+
+| Adapter | `appendToAuditChain` | Multi-process safe? |
+|---|---|---|
+| **Postgres** (`createPostgresStorage`) | ✅ per-org `pg_advisory_xact_lock` transaction (falls back to a bounded `23505`-retry loop for query-only pools) | ✅ across processes sharing the database |
+| **Memory** (`createMemoryStorage`) | ✅ per-org in-process async lock | Single process **by design** — memory is not shared across processes |
+| Third-party adapter without the method | — | Falls back to the legacy process-local-sequence path (correct under a **single writer** only) |
+
+**Custom storage adapters:** to be multi-process-safe, implement
+`appendToAuditChain` so the head-read → compute → insert sequence is atomic
+against concurrent writers (a row lock, an advisory lock, a serializable
+transaction, or a compare-and-set retry on your uniqueness constraint). If you
+can't, leave it unimplemented and run a single writer — the SDK falls back
+safely and warns.
+
+**The standalone `createIntegrityAudit()` wrapper is single-process only.** It
+keeps its chain in process memory and never persists integrity metadata, so it
+forks across processes and loses verifiability across restarts. Use it for
+prototyping and tests; use `createGovernance({ integrityAudit })` (above) for
+durable, multi-process audit.
+
+**Rolling deploys:** the lock only protects writers that take it. During a
+mixed-version window (some processes pre-0.18.2), the old processes still
+allocate from process-local counters and can collide with or fork past the
+locked writers. Replace all writers together and expect residual
+unique-violation warnings until the last old process drains.
+
 ### Kill Switch
 
 Emergency halt for any agent, enforced via a reserved-priority policy rule

packages/governance/package.json

@@ -1,6 +1,6 @@
 {
   "name": "governance-sdk",
-  "version": "0.18.1",
+  "version": "0.18.2",
   "description": "AI Agent Governance for TypeScript — policy enforcement, scoring, compliance, and audit for AI agents",
   "type": "module",
   "main": "./dist/index.js",