Skip to content

fix(objectql): refuse undeclared insert fields at the schema, and keep bound values out of the write-path logs (#8682) - #8737

Merged
hotlong merged 2 commits into
mainfrom
claude/issue-8682-undeclared-field-preflight
Aug 14, 2026
Merged

hotlong merged 2 commits into
mainfrom
claude/issue-8682-undeclared-field-preflight

Conversation

@hotlong

@hotlong hotlong commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Fixes #8682

Both halves of the card land. Verified at d99df41.

Half A — validation ordering

An undeclared write key is now refused by the object's field map as the first act inside insert()'s middleware body — after middleware (which may legitimately rewrite data), before applyFieldDefaults, and therefore before the defaults, the summary seeding, the beforeInsert hooks, the secret writes, validation, the autonumber and the statement.

Measured on origin/main @ 3508678 with a real engine and a recording driver, one mistyped key on a single insert:

observable before after
autonumber around the refused request 0001 → refused → 0003 0001 → refused → 0002
beforeInsert hook runs ok-1, bad, ok-2 ok-1, ok-2
driver create() calls 3 2
refusal envelope SQLITE_ERROR, no status INVALID_FIELD + 400

The hook did not merely run — its derived value reached the statement, which is the card's period_label = 'Q3 2026' shape: a hook is not a pure function, so "it ran and we threw the result away" is a side effect of a request the server had already decided to refuse.

The wire answer is deliberately unchanged. The refusal carries code: 'INVALID_FIELD', status: 400, field and object, and its message is byte-identical to the one mapDataError's driver-string branch produced. @objectstack/rest re-emits that verbatim, so the caller sees exactly what it saw before — the refusal moved, the answer did not.

Where the door deliberately has no opinion (each measured, not assumed — all three surfaced as test failures in the first full run and are now pinned as controls):

  • schema.fields absent or empty — an empty map is indistinguishable from an unpopulated one, so refusing everything on it would be a verdict made from an absence.
  • id / created_at / updated_at — tolerated even when a declaration omits them, mirroring the three names find() / findOne() already add to their known set, so a key accepted by a read is not refused by a write.
  • Schema drift — a DECLARED field whose physical column is missing is invisible to this door by construction and stays the driver's to refuse.

In all three the driver remains the backstop it has always been; nothing is widened.

insertMany culls such a row per row instead of failing the batch around it — before this, an undeclared key defeated partial-success mode completely, because the row travelled to bulkCreate with the rest and took the whole batch down with it.

Half B — value logging

Logger.error(msg, error, meta) serializes exactly error.message and error.stack, and a knex-shaped driver error opens both with the fully bound statement. Confirmed with the card's canaries on origin/main: message and stack each carried SENSITIVE-CANARY-9f3a2b and the row's other values — so redacting one and not the other would have moved the leak rather than closed it. Both are rebuilt.

What is kept, deliberately: the level (error), the message (Insert operation failed), the object meta, the database's own diagnostic — which names the failing column — and every stack frame. The card's triage is explicit that the failing column and object must still be logged, and this narrows only WHAT is written.

The cut is structural rather than lexical: @objectstack/types' looksLikeInternalErrorLeak already owns "is this a driver dump?" for both HTTP boundaries, so this asks it the verdict and adds only the one thing it does not answer — where the statement ends. Every inlined value sits in the statement and the statement always comes first, so the last - separates the part that may carry values from the part that may not. Last rather than first because a bound value may itself contain -; cutting early would leave a fragment of that value standing in what we then log as "the diagnostic". A dump with no separator carries no statement and is returned untouched, same reference.

The rethrown error is untouched. mapDataError reads the driver's raw message to answer 400 INVALID_FIELD with the failing field, so the redaction applies to the log slot only — one argument at one call site.

One scope call worth a reviewer's eye

The card measured the INSERT logger. The Update operation failed and Delete operation failed loggers three screens away are the same one-argument call, the same knex-shaped error and the same security consequence — an UPDATE inlines the caller's values in its set clause exactly as an INSERT does in its values list. I applied the same redaction to all three and pinned the update and delete phrasings, rather than leave two thirds of a security-labelled hole open for a later round. Happy to trim it to insert-only if you would rather keep the diff on the card's exact letter.

The first sub-claim was already fixed, and I confirmed it

The card reports the false [REST] Unhandled error label as FIXED upstream. Confirmed on current main: packages/rest/src/rest-expected-error-logging.test.ts pins exactly that ("[#4886] Expected 4xx must not be logged as [REST] Unhandled error"). Nothing was needed for it, so this PR implements less than the card describes for that one row.

Fixture triage

The first full run of the objectql suite went from 3591 green to 49 failures, every one an accept-set narrowing. Triaged individually rather than batch-re-spelled:

  • 15 × title on task, 1 × customer on sys_license — fixtures whose registry stub is fields: {}. Fixed in the DOOR, not the fixtures: an empty field map now yields no verdict, which is the correct rule and not a test accommodation.
  • 1 × id on bf_leave_request — the fixture declares neither id nor the audit columns. Fixed in the door too, mirroring the read path's existing tolerance.
  • 24 × package_id on sys_metadata — four minimal stubs omitted a column the REAL object declares (metadata-core/src/objects/sys-metadata.object.ts:65, part of the row's uniqueness key) and that SysMetadataRepository writes. The production writer is correct; the stubs were never spec-valid, and nothing noticed while an undeclared key simply travelled on to the driver. Declarations added.
  • 1 × engine-insert-many — the bad row was spelled { slug: 'no-name' } to mean "a row missing the required name", but slug is itself undeclared, so the door refused it before the hooks and the assertion would have measured the wrong refusal. Re-spelled to {}, which says the one thing intended.

No production code outside engine.ts needed a change, and no undeclared-write finding survived triage as a real defect.

Verification — at d99df41

  • pnpm --filter @objectstack/objectql test — 205 files, 3617 tests, 0 failures
  • pnpm --filter @objectstack/objectql typecheck — clean
  • Downstream consumer sweep, pnpm --workspace-concurrency=2 --no-bail --filter '...@objectstack/objectql' test (prefix form = the 43 packages that DEPEND on objectql), after a full pnpm build: 43/43 green, zero Unknown field refusals anywhere — packages/qa/dogfood included, which boots real example apps end to end. That sweep is the evidence that the narrowed accept set costs no live caller anything.
  • Reverse verification — engine.ts reverted to origin/main, pins re-run, then restored and proved byte-identical (git hash-object == the committed blob). The direction is deliberately not uniform, and that is the point: 8 of 10 half-A cases go red (expected '0003' to be '0002'; hooks ['ok-1','bad','ok-2']; 1 driver create), while the two half-A controls stay green — they assert the door has NO opinion, so they must pass on origin/main too. On half B exactly one engine-level case flips (the canary case); "the entry survives" and "the failing column is still named" stay green, because origin/main also logged at ERROR with the object and the column — it just also logged the values.
  • Gates re-derived from the actual diff with scripts/pm/dispatch-gates.mjs, all green: check:durability-log-level, check:stack-collection-maps, check-engine-split-ratio, plus seven the dispatch list did not name — check:nul-bytes, check:changeset-gate-self-tests, check:objectui-changeset, check:query-options-erasure, check:type-check-coverage, check:type-check-debt (the --re-measure ratchet, 33 entries, none above ceiling), and the three changeset scripts.

The regression pin is the log-independent observable the triage asked for — the autonumber gap across the rejected request. No test in this PR greps log contents for half A, precisely because half B rewrites the logger in the same commit.


Generated by Claude Code

claude added 2 commits August 14, 2026 16:09
…p bound values out of the write-path logs (#8682)

Half A — an undeclared write key is refused by the object's field map before
`applyFieldDefaults`, so no default, no summary seeding, no beforeInsert hook,
no secret write, no validation and no AUTONUMBER runs for a request that was
already going to be refused. Measured on origin/main, one mistyped key made a
valid create numbered 0001 be followed by 0003; it is now followed by 0002.
The wire answer is unchanged: 400 INVALID_FIELD with the same message.

Half B — the insert/update/delete loggers no longer write the driver's bound
statement. `Logger` serializes exactly `message` and `stack`, and the statement
opened both, so a single mistyped field name wrote an entire row's values to
disk at ERROR. The level, the message, the object and the database's own
diagnostic (which names the failing column) are all kept; the rethrown error is
untouched so the caller's 400 does not move.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XeQRiAa7vYRVX5Fog7Zby8
@vercel

vercel Bot commented Aug 14, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated (UTC)
objectstack Ignored Ignored Aug 14, 2026 5:01pm

Request Review

@github-actions

Copy link
Copy Markdown
Contributor

📓 Docs Drift Check

This PR changes 1 package(s): @objectstack/objectql.

15 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:

  • content/docs/concepts/metadata-lifecycle.mdx (via @objectstack/objectql)
  • content/docs/data-modeling/formulas.mdx (via packages/objectql)
  • content/docs/deployment/migration-from-objectql.mdx (via @objectstack/objectql)
  • content/docs/deployment/vercel.mdx (via @objectstack/objectql)
  • content/docs/kernel/contracts/data-engine.mdx (via @objectstack/objectql)
  • content/docs/kernel/runtime-services/examples.mdx (via packages/objectql)
  • content/docs/kernel/services-checklist.mdx (via @objectstack/objectql)
  • content/docs/kernel/services.mdx (via @objectstack/objectql)
  • content/docs/permissions/authentication.mdx (via @objectstack/objectql)
  • content/docs/permissions/system-context.mdx (via packages/objectql)
  • content/docs/plugins/index.mdx (via @objectstack/objectql)
  • content/docs/plugins/packages.mdx (via @objectstack/objectql)
  • content/docs/protocol/kernel/index.mdx (via @objectstack/objectql)
  • content/docs/protocol/objectql/query-syntax.mdx (via packages/objectql)
  • content/docs/protocol/objectql/state-machine.mdx (via @objectstack/objectql)

⛔ 1 release-owned page(s) also reference the affected code. These are read-only:

  • content/docs/releases/implementation-status.mdx (via @objectstack/objectql)

content/docs/releases/ is RELEASE-OWNED (AGENTS.md "Documentation Guardrails"): release
notes are written centrally at release time, and a code PR that edits them is the exact PR
that guardrail exists to stop. They are still audited — read-only. If one of them is actually
wrong, file an issue or open a dedicated docs-only PR; do not edit it here.

Advisory only. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs origin/main → pass the list as args.docs.

@github-actions github-actions Bot added documentation Improvements or additions to documentation tests tooling labels Aug 14, 2026
@hotlong
hotlong marked this pull request as ready for review August 14, 2026 17:25
@hotlong
hotlong added this pull request to the merge queue Aug 14, 2026
Merged via the queue into main with commit 8a9e7f4 Aug 14, 2026
27 checks passed
@hotlong
hotlong deleted the claude/issue-8682-undeclared-field-preflight branch August 14, 2026 17:41
akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Aug 17, 2026
…the beforeUpdate hooks (objectstack-ai#8738) (objectstack-ai#8775)

* fix(objectql): refuse undeclared update fields at the schema, before beforeUpdate (objectstack-ai#8738)

The insert path's declared-field door (objectstack-ai#8682) applied to the second write verb —
the same function, given its second caller and a verb-neutral name, rather than a
second predicate to drift against.

The card filed this half as INFERRED and asked for a reproduction first. Both
claims reproduce on origin/main @ e5eeb49: an undeclared key reached
driver.update on the by-id branch and driver.updateMany on the predicate branch
and was refused THERE, after beforeUpdate had run and stamped a derived value
onto the payload the driver then rejected. Unlike insert there is no autonumber,
so nothing durable is consumed — the hook side effect and declared = enforced
(PD objectstack-ai#10) are the case, and the hook run is what the suite pins.

The wire answer is unchanged: the same 400 INVALID_FIELD with the same message.
objectstack-ai#8737's three no-opinion cases (absent map, empty map, id/created_at/updated_at)
are reused unchanged and pinned as controls.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XeQRiAa7vYRVX5Fog7Zby8

* test(runtime): exercise the objectstack-ai#4271 driver split through bodies on both verbs (objectstack-ai#8738)

The one red in the downstream sweep that was NOT a fixture field-map defect.
`undeclared-field-write-driver-split` pinned the schemaless family PERSISTING an
undeclared key on update by calling engine.update() with the typo in the
CALLER's payload — on the file's stated reasoning that a beforeUpdate body
"would only add the flat-input envelope to the thing under test".

The declared-field door falsifies that equivalence, and the same thing is
already true of insert on main: measured here, a caller-supplied undeclared key
on the MEMORY driver is refused INVALID_FIELD/400 with nothing persisted, by
objectstack-ai#8737's insert door. The file survived that only because its insert arm injects
the key through a hook body, which runs after the door.

So both update cases now carry a real beforeUpdate body, matching the insert
arm: the key is added below the engine's validation, still reaches the driver,
and the split the file exists to pin is measured on both verbs (SQL: `no such
column: stagee`, whole write lost; memory: persisted alongside the declared
key). The caller-payload half is pinned separately as what it now is — a schema
refusal on BOTH families, with no split to observe.

The three prose surfaces the file guards (the two lint messages and
content/docs/automation/hook-bodies.mdx "What still happens at runtime")
describe BODY writes, which the door does not touch, so they stay accurate as
written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XeQRiAa7vYRVX5Fog7Zby8

* test(objectql,runtime): type the new options bags and drop an unused parameter (objectstack-ai#8738)

Two ratchets the new test code moved, both fixed at the author's end rather than
by raising a ledger:

- check:query-options-erasure — the test surface grew 240 → 242. The two new
  `as any` options bags are not deliberately off-contract input, so they are
  TYPED (EngineUpdateOptions for the predicate branch, EngineQueryOptions for
  the read-back) instead of erased. Back at the 240 ceiling.
- check:type-check-debt — objectql's TEST_DEBT re-measured 355 → 356 on one
  TS6133 (an unused `object` parameter in the recording driver's `create`).
  Renamed to `_object`; re-measured at 355, matching the ledger, with zero
  errors attributable to this card's files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XeQRiAa7vYRVX5Fog7Zby8

---------

Co-authored-by: Claude <noreply@anthropic.com>
akarma-synetal pushed a commit to akarma-synetal/framework that referenced this pull request Aug 17, 2026
…in the uncovered dialect (objectstack-ai#8824)

`DIALECT_LEAK_PHRASINGS` said MySQL/MSSQL/Oracle "nobody here runs them".
Measurably false for MySQL on the same tree: driver-sql branches on
mysql/mysql2, CI stands up a live mysql:8.0 for the required Temporal
Conformance check, and live MySQL 8.0.46 measurements landed driver fixes
(objectstack-ai#8621, objectstack-ai#8622).

The claim is a security-reasoning input and was consumed as one on PR objectstack-ai#8737,
so the correction replaces the reachability claim with a COVERAGE statement
and adds the pins that keep it honest. Behaviour is unchanged: no keyword is
added, and whether MySQL is a supported deployment target or a tested dialect
stays an open product question on objectstack-ai#8739.

Part of objectstack-ai#8739


Claude-Session: https://claude.ai/code/session_01NaS1PAHJcPfAA2acnV53Tn

Co-authored-by: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation size/l tests tooling

Projects

None yet

2 participants