feat(verify): the handle observes the writes the engine receives, a hook's refused write included (#22781)

Part of #22301
Clause-②: yes (widening)

Item 6 of the card: there was no observation of what a hook handed the
engine. A refused write left no row, an `async: true` hook's completion
was invisible, and a refusal could not be staged without a spy on the
engine. Item 7 and item 1's remaining divergence are not in this PR, and
the card stays open for them.

## What changes

- **`stack.observeWrites(fn)`** (`packages/verify/src/handle.ts`) runs
`fn` and resolves with every insert, update and delete the booted engine
received meanwhile, in the order each reached it. Each entry
(`ObservedWrite`) carries `object`, `operation`, `data` (copied on
arrival), `where`, `multi`, the execution `context` the write carried,
and `outcome`. The outcome starts `pending`, then becomes `resolved`
with the engine's result, or `refused` with the error the engine threw
(the same object) and that error's own `code`.
- `fn` is handed the live observation (`WriteObservation`).
`observation.settled(match)` resolves with the first write `match`
accepts once the engine has answered it. That is how a test waits for an
`async: true` hook's write. It rejects after `timeoutMs` (default 2000),
naming every write observed, or when the observation closes first.
- **The mechanism:** one global middleware on the engine's own chain
(`registerMiddleware`, a member of the `IObjectQLEngine` contract),
registered on the first call and kept. With no observation open it is
`return next()`. It changes no payload, no answer and no error, and
decides nothing. No engine file and no spec file is touched.
- A `fn` or `match` that is not a function, and a `timeoutMs` that is
not a positive number, are refused with `INVALID_REQUEST` / `400`, the
handle's existing call-shape refusal. No error code is added.
- New exported types: `ObservedWrite`, `ObservedWriteOutcome`,
`WriteObservation`.
- Door rosters: `handle.ts`'s header and `harness.ts`'s `VerifyStack`
docblock name the door. `packages/verify/README.md` adds an example and
a bullet.
- `packages/qa/dogfood/test/rls-runner.test.ts`: its fake `VerifyStack`
lists every handle member as `never`, and gains `observeWrites` (the
item 8 lesson).
- `.changeset/22301-verify-observe-writes-door.md`:
`@objectstack/verify` `minor`, `Clause-②: yes (widening)`.

## What was measured before the door was built

These were measured with a scratch probe (since deleted) on a booted
stack at `490cb6d9fa`. The probe registered a global middleware on the
booted `objectql` after boot, which is exactly what the door does.

1. **Where a hook's writes travel.** A hook's `ctx.api` is a
`ScopedContext`. Its `object(name).insert(...)` calls
`engine.insert(name, row, { context })` (`ObjectRepository`,
`packages/objectql/src/engine.ts`), so every write a hook makes passes
the engine's middleware chain like any other. The candidates:
   - **The middleware chain:** the seam the door sits on.
- **`WriteObservabilityOptions`:** per-call options chosen by whoever
makes the write, and a hook's `ctx.api.insert` passes only `{ context
}`, so a test cannot set them on a hook's write.
- **The hook metrics recorder:** it sees a hook's outcome, not its
writes, and it is read when hooks are bound, so it cannot be swapped
after boot.
- **The kernel's `objectql` slot:** that is how the door reaches the
engine.

The probe saw a refused hook write that leaves no row: an `afterInsert`
hook's `ctx.api` insert, refused by a declared validation rule, was
recorded `refused` `VALIDATION_FAILED`, and no row was found.
2. **The boundary the position fixes.** A middleware registered after
boot sits after the write gates the boot registered. A hook's write that
the permission check refused was not recorded at all, and the hook
caught `PERMISSION_DENIED`. That gate throws before calling `next()`
(`plugin-security`'s write middleware). Two refusals happen before the
chain on all three verbs, so no middleware sees them: an external
datasource (`assertWriteAllowed`) and a cross-driver write in a
transaction (`enforceTransactionOrigin`).
3. **How `async: true` hooks are scheduled.** `wrapDeclarativeHook`
(`packages/objectql/src/hook-wrappers.ts`) starts the handler as `void
runWithErrorPolicy(detached)`. No promise, queue or drain is kept
anywhere a test could await. The probe's fire-and-forget write was
absent when `hooks.run` returned and present 300 ms later. So the honest
await is the hook's write reaching the engine (`settled`), and awaiting
the hook's completion itself needs a new seam (carried need C2 below).
4. **Staging a refusal through existing doors.**
- **Seeded state:** a seeded holder of a `unique` value made the hook's
insert refused `DUPLICATE_RECORD`, and the refusal was recorded.
- **A declared validation rule:** refused `VALIDATION_FAILED`, and
recorded.
- **A permission:** the write is refused, but past the observer's reach
(point 2).
- **`strictReadonlyWrites`:** a per-call option the writer chooses,
which a test cannot set on a hook's write.

   So need 3 is documentation plus pins, not new code.
5. **The observer is global.** After an observation,
`hasObjectMiddleware` still answers `false` for a fixture object, so the
analytics native-SQL strategy's per-object check is unaffected.

## Per need: door, or decision

- **Need 1, a refused write leaves no row:** a door. `observeWrites`
records the refusal with the engine's own error, for every refusal past
the write gates.
- **Need 2, an async hook's completion:** a door for the hook's write
(`settled`). A hook that writes nothing stays invisible; that is carried
need C2.
- **Need 3, staging a refusal:** documentation plus pins. Arrange a
refusal the engine makes itself: seed the row that already holds a
unique value, or set up the state a declared validation rule refuses.

### Out of scope, by decision

- **A switch that refuses a write in the engine's place** (the spy's
`mockRejectedValue`) is out of scope.
- Business need: the real need is a hook's error path, and both measured
real refusals reach it.
- Long-term: #15951 B′ rules out a door that answers in place of the
engine.
- AI-error resistance: a fabricated refusal can be one the engine never
makes (a code, a shape, a phase), and a suite would stay green over it.
  - Startup focus: no new surface is needed for it.
- **Seeing a gate's refusal by moving the observer ahead of the gates at
boot** is out of scope.
- The kernel has no boot hook between `init` and `start`, so it would
take a plugin added to `bootStack`'s composition. Ruling `6070767186`
(A) has `bootStack` compose what `serve` composes, and the handle is not
a second boot path.
- The engine has no "register first" option, and pushing into its
private middleware list would be a spy by another name.
  - The real home is carried need C1.

## Carried needs (new engine seams, named and not built)

- **C1, observing a write before the write gates.** A hook's write
refused by a gate (the permission check, record-level security, the
tenant wall) cannot be observed without a seam ahead of the chain. One
example is a write-observer member at the engine's operation door, which
would also see the two pre-chain refusals. Another is a position option
on `registerMiddleware`. Its home is the engine contract
(`IObjectQLEngine` in `packages/spec/src/contracts/objectql-engine.ts`)
and `packages/objectql/src/engine.ts`. Until then, a gate's refusal
reaches the hook that made the write, and a test can read it there.
- **C2, an engine-owned record of fire-and-forget hook runs.** For
example, `wrapDeclarativeHook` could register each detached run with the
engine, which would expose an awaitable drain. This makes an `async:
true` hook's completion observable even when it writes nothing, and its
own failure (today only an `error` log line) assertable. Its home is
`packages/objectql/src/hook-wrappers.ts` plus an engine member.

## Pins: `packages/verify/src/handle.observe-writes.test.ts` (11 tests)

The fixture is neutral. Inserting an `ow_order` fires `afterInsert`
hooks that write an `ow_task` (unique `code`, validation rule
`title_not_blocked`), an `ow_vault` (granted to nobody but the admin),
or, fire-and-forget behind a latch the test releases, an `ow_task`
later. Each hook swallows what the engine answered into a map, so the
triggering write succeeds either way.

- **T1, need 1:** a hook's write the validation rule refuses is recorded
`refused` `VALIDATION_FAILED`, with its payload, the triggering user's
context and the engine's `ValidationError`. No row. Control: an accepted
title is recorded `resolved`, and its row is there.
- **T2, the observer changes nothing:** the caller's error is the very
object recorded (`toBe`), and the same call unobserved answers the same
`code` / `status`.
- **T3, need 2:** the fire-and-forget write is absent when `hooks.run`
returns. `settled` resolves with it `resolved`, and the row is there.
- **T4:** `settled` also answers a fire-and-forget write the engine
refuses, which leaves no row.
- **T5, need 3:** a seeded holder makes the hook's write refused
`DUPLICATE_RECORD`, and only the holder's row exists. Control: a fresh
code is written.
- **T6, the boundary:** a hook's write the permission check refuses
leaves no entry, and the hook caught `PERMISSION_DENIED`. Control: the
admin's same hook write is recorded `resolved`.
- **T7:** a seed, a predicate update and a system delete are each
recorded with their own `where`, `multi`, caller and result (the
update's affected-row count, 1).
- **T8:** the window closes with `fn`: a later write is not added, and
`settled` after the close rejects.
- **T9:** `settled` rejects on its timeout, and its message names the
writes it saw.
- **T10:** the observer is global, and `hasObjectMiddleware` still
answers `false`.
- **T11:** the three malformed calls are refused `INVALID_REQUEST` /
`400`.

## Ablations

All four ran on the committed head `60d6583edc` through `node
scripts/ablation-replace.mjs` in wrap mode. In each, the anchor went 1 →
0 and the blob changed. Each was restored to the `HEAD` blob
`62a14c3b71ef`, with `git diff HEAD` empty. The predictions were written
down before the first run. The subject is reached by a relative `src`
import, so no `dist/` is involved.

- **A1, a refusal is not recorded (it stays `pending`).** Predicted red:
T1, T2, T4, T5. Observed: 4 failed and 7 passed, exactly those four.
- **A2, `settled` accepts a write the engine has not answered.**
Predicted red: T3, T4. Observed: 2 failed and 9 passed, exactly those
two.
- **A3, the observation is never closed.** Predicted red: T8. Observed:
1 failed and 10 passed.
- **A4, the observer rethrows a copy of the engine's error.** Predicted
red: T2. Observed: 1 failed and 10 passed.

## Verification

Each result below names the commit it ran at. The final head is
`6f6793c636`, a merge of `origin/main` `e84aeb36ce`, which touches no
path of this PR.

- **Builds:**
- `turbo run build --filter='@objectstack/verify^...'` passed 47/47 at
`490cb6d9fa`.
- `--filter='@objectstack/dogfood^...' --filter=@objectstack/verify`
passed 63/63 at `6f6793c636`.
- **`@objectstack/verify` full suite:** `vitest run --maxWorkers=2`
passed 29 files and 234 tests, at `60d6583edc` and again at
`6f6793c636`. The new file passed 11/11 on its own.
- **`pnpm --filter @objectstack/verify typecheck`** (`tsc --noEmit` and
`check:test-typecheck`) exited 0 at `6f6793c636`. `tsc -p
tsconfig.test.json --listFiles` reaches 29/29 of the package's test
files, including the new one.
- **Consumer typecheck:** `pnpm --filter @objectstack/dogfood typecheck`
exited 0 at `6f6793c636`. Verify was rebuilt as a cache miss first, and
its `dist/index.d.ts` names `observeWrites` 3 times.
- **Reverse check, from dogfood:** deleting the fake stack's
`observeWrites` line turned `tsc` red with TS2741 ("Property
'observeWrites' is missing … required in type 'VerifyStack'"). It was
restored through the tool.
- **Gates:** `node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` with no paths derived 68 commands at
`6f6793c636`, and all 68 ran there.
  - 67 exited 0.
- `check:dual-build-cjs-loads` exited 3 (PREREQUISITE NOT MET: it reads
a whole-workspace build, which this dispatch does not run). It is NOT
MEASURED and left to CI.
- `--ran` reconciliation: 68 derived, 67 run, 1 NOT-MEASURED (derived
from the recorded exit 3), 0 UNRUN.
- **Lint:** `pnpm lint` (the repo-wide scan) is CI's run and is not
claimed here. A targeted `eslint --no-inline-config --format json` over
the 5 touched `.ts` files linted 5 files, with 0 errors and 0 warnings.
`eslint.config.mjs` never enables type-aware linting, so this diff
cannot move the verdict on an untouched file.

## Acceptance notes

- **An outcome is the engine's answer, not row state.** The probe
measured this: an `afterInsert` hook that throws (`onError: 'abort'`,
the default) rejected the triggering insert, and both that row and an
earlier hook's row stayed stored. Neither `hooks.run` nor the REST data
ingress (`createData` → `engine.insert`) opens a transaction. The door's
docs say so ("read rows back with `rows`"). The spec half is reported to
the seat on the card as a finding, not filed from here.
- **The recorded `result` is the executor's answer** before the outer
gates' post-phase (field-level masking). It is the system-side view,
stated in the type's docblock.
- **The middleware stays registered once installed.** The engine has no
way to take one back, and with no observation open it is a pass-through.
- **The window is time, not cause.** A sign-in's key rotation
(`sys_jwks`) was observed in the probe's window, so the pins filter by
object, and the docs say to.
- **For downstream apps (the `repo:hotcrm` seat):** `recordEngineWrites`
maps onto `observeWrites`, refused writes and fire-and-forget writes
included. A staged refusal moves to a real one (seeded state, or the
app's validation rule). A refusal by a permission gate and a hook that
writes nothing are the two cases still out of reach (C1, C2).

---
_Generated by [Claude
Code](https://claude.ai/code/session_016njDy8ozy9B9Ns5Y8kAWEK)_

Co-authored-by: Claude <noreply@anthropic.com>

Author: objectstack-fleet[bot]

.changeset/22301-verify-observe-writes-door.md

@@ -0,0 +1,18 @@
+---
+'@objectstack/verify': minor
+---
+
+feat(verify): the handle gains a door that observes the writes the engine receives, a hook's refused write included
+
+Clause-②: yes (widening)
+
+An app's tests can see what a hook handed the engine through the in-process handle, so a suite no longer wraps the booted engine's write methods in a spy to find a refused write, to wait for a fire-and-forget hook, or to stage a refusal.
+
+- **`stack.observeWrites(fn)`** runs `fn` and resolves with every insert, update and delete the booted engine received meanwhile, in the order each reached it. Each entry (`ObservedWrite`) carries `object`, `operation`, the payload (`data`), the update or delete selection (`where`), `multi`, the execution context the write carried (`context`), and the engine's answer (`outcome`): `pending`, then `resolved` with its result, or `refused` with the error the engine threw, unchanged, and that error's own `code`.
+- **What it is.** One global middleware on the engine's own chain (`registerMiddleware`), added on the first call. It passes every operation on unchanged and decides nothing. So it sits after the write gates the boot registered. Every write those gates let through is recorded, from any door (`hooks.run`, `seed`, a flow, a route, a hook's `ctx.api`), and so is every refusal the engine raises past them: a declared validation rule (`VALIDATION_FAILED`), a hook's throw, a unique conflict (`DUPLICATE_RECORD`). A write a gate refuses, the permission check among them, never reaches it and leaves no entry. Its refusal still reaches whoever made the write.
+- **An outcome is the engine's answer to that call, not a statement about stored rows.** A write an `after*` hook refuses was refused after the driver wrote, and its row can stay. Read rows back with `stack.rows`.
+- **The window is time, not cause.** A write the stack makes for its own reasons while `fn` runs is recorded too, so filter by `object`.
+- **An `async: true` hook.** Its write reaches the engine after the write that fired it has returned. `fn` is handed the live observation (`WriteObservation`), and `observation.settled(match)` resolves with the first write `match` accepts once the engine has answered it. It rejects after `timeoutMs` (default 2000), naming every write observed, or when the observation closes first. A hook that writes nothing leaves nothing to observe.
+- **Staging a refusal.** Arrange one the engine makes itself: seed the row that already holds a unique value the hook will write, or set up the state a declared validation rule refuses. The handle has no switch that refuses a write in the engine's place.
+- A `fn` that is not a function, a `match` that is not a function, and a `timeoutMs` that is not a positive number are refused with `code: 'INVALID_REQUEST'`, `status: 400`. A rejection of `fn` is rethrown unchanged once the observation has closed. No error code is added.
+- New exported types: `ObservedWrite`, `ObservedWriteOutcome`, `WriteObservation`.

packages/qa/dogfood/test/rls-runner.test.ts

@@ -136,6 +136,7 @@ function fakeStack(opts: FakeOpts): VerifyStack {
     automation: undefined as never,
     seed: undefined as never,
     rows: undefined as never,
+    observeWrites: undefined as never,
     metadata: undefined as never,
     tenancy: undefined as never,
   };

packages/verify/README.md

@@ -76,9 +76,9 @@ await stack.stop();
 
 Every `VerifyStack` carries it; nothing extra to boot. Each method is a thin
 facade over a door the kernel wired at boot — the ObjectQL engine's own write,
-dry-run and read calls, the runtime's `/automation` and `/actions` routes
-driven in-process, the automation engine's condition evaluator, the
-`SchemaRegistry`, the `tenancy` service — with **zero
+dry-run and read calls and its middleware chain, the runtime's `/automation`
+and `/actions` routes driven in-process, the automation engine's condition
+evaluator, the `SchemaRegistry`, the `tenancy` service — with **zero
 re-implemented semantics**: the handle assembles no execution context, orders
 no hooks, evaluates no permission. What the engine does is what you assert on.
 
@@ -137,6 +137,16 @@ expect(await stack.automation.evaluateCondition(
 const [acc] = await stack.seed('crm_account', [{ name: 'Globex' }]);
 const mine = await stack.rows('crm_opportunity', { crm_account: acc.id }, { as: rep });
 
+// What a hook handed the engine: every write the engine receives while the
+// function runs, and what it answered. A hook's write the engine refused
+// leaves no row; here it is, refused, with the engine's own error.
+const writes = await stack.observeWrites(async (observation) => {
+  await stack.hooks.run('crm_lead', 'insert', { last_name: 'Ada', company: 'Globex' }, { as: rep });
+  // An `async: true` hook writes after the triggering write has returned.
+  await observation.settled((w) => w.object === 'crm_task');
+});
+const refused = writes.filter((w) => w.outcome.status === 'refused');
+
 // What the boot actually holds.
 stack.metadata.object('crm_opportunity')?.fields;      // system columns injected
 stack.metadata.items('permission');                    // the registry's singular names
@@ -164,6 +174,22 @@ await stack.stop();
   touched, with `INVALID_REQUEST` / `400` (a caller named twice, or a
   `hooks.updateWhere` the engine would write by id). Assert on `code` (and
   `status` / `statusCode`), never on a message alone.
+- `observeWrites(fn)` records each insert, update and delete the engine
+  receives while `fn` runs (`object`, `operation`, `data`, `where`, `multi`,
+  `context`) and what the engine answered (`pending`, then `resolved` with its
+  result or `refused` with its error and that error's own `code`). It is one
+  middleware on the engine's own chain that passes everything on unchanged,
+  so it sits after the write gates the boot registered: a write a gate refuses
+  (the permission check among them) never reaches it and leaves no entry,
+  while every refusal past the gates (a validation rule, a hook's throw, a
+  unique conflict) is recorded. An outcome is the engine's answer to that call,
+  not a statement about stored rows: read those back with `rows`. The window is
+  time, not cause, so filter by `object`. `observation.settled(match)` waits
+  for a write that lands later, such as an `async: true` hook's (default
+  timeout 2000 ms). To stage a refusal of a hook's write, arrange one the
+  engine makes itself: seed the row that already holds a unique value, or set
+  up the state a declared validation rule refuses. The handle has no switch
+  that refuses a write in the engine's place.
 - Many files, one boot: `bootStackOnce(config, opts?)` memoises `bootStack` per
   `(config, opts)` object identity for the life of the process. Share it from
   one module, under vitest `isolate: false`, and never `stop()` a stack other
@@ -221,7 +247,7 @@ run" must never read like "nothing to find".
 
 - `bootStack(config, opts?)` → `VerifyStack` (`api` / `raw` / `signIn` / `signUp` / `apiAs` / `stop`, plus the handle:
   `hooks.run` / `hooks.updateWhere` / `validate` / `flows.run` / `flows.resume` / `actions.run` /
-  `automation.evaluateCondition` / `seed` / `rows` / `metadata` / `tenancy` / `contextFor`).
+  `automation.evaluateCondition` / `seed` / `rows` / `observeWrites` / `metadata` / `tenancy` / `contextFor`).
 - `bootStackOnce(config, opts?)` → the same, memoised per `(config, opts)` identity for the process.
 - `deriveCrudCases(config)` → the auto-derived round-trip cases (write one, read one, assert) for every object.
 - `runCrudVerification(stack, token, config)` → `VerifyReport`; `formatReport(report)` for a log summary.