fix(objectql,metadata-protocol): the data door honours a field's internal flag in its filter, sort, group-by, aggregate, $search and $expand positions (#22646)

[objectstack-fleet]

Fixes #22646

Clause-②: no (narrowing)

Summary

A field declared internal: true is withheld from every generic exit (#21197): the engine omits it from every row. The ROW position honoured that; the generic data door's EVALUATE positions did not, so a value the platform withholds from rows could still be learned through a position that answers by its stored value. Class ① product defect, security — class, position and function level only.

Every position now judges the one flag reader (collectInternalReadFields in @objectstack/objectql, or its byte-identical @objectstack/core twin collectInternalWriteResponseFields on the door, which sits below objectql), and a position that reveals a stored value is refused with a declared ADR-0112 envelope — INVALID_FIELD / 400. No new error code; Clause-② stays no.

Measurement (real stack, before)

Measured on origin/main as a seeded administrator and an ordinary member, on the list and query data routes, over a platform object's internal: true credential-digest column and a non-hidden internal text column (the dev's report on #22646 holds the run; ⛔ no request shapes here):

• filter (implicit, filter=, query where, nested $or, cross-field comparand): served — the predicate reached the driver, a confirmation oracle.
• sort ($orderby): served — the order leaked the comparative stored value.
• group-by / aggregate operand: refused, but as an undeclared 500 INTERNAL_ERROR (ObjectQL.rejectCredentialAggregation threw a bare Error).
• per-aggregation filter (aggregations[].filter) and filter beside an aggregation: served — the same oracle at a second filter position.
• $search field list: a non-hidden internal text column entered the auto-default scan set and an explicit $searchFields naming one was accepted.
• $expand target field: a one-level expand's own where / orderBy on the target object's internal field reached the expansion sub-read — the expanded record's presence is an oracle on the related row.
• nested-relation condition (a condition on a lookup's related object, at the top level, under $or, and in the count total), measured in patch round 1 on fixture objects: served — the parent row came back only when the related object's internal value matched. The same condition inside aggregations[].filter was already refused (INVALID_FILTER), and the dotted array spelling already refused (INVALID_FIELD).
• dotted sort / group-by path through a lookup: not reachable — the ingress sort gate refuses every spelling (INVALID_SORT) and the group-by gate refuses it as an unknown field (INVALID_FIELD); a dotted sort inside an $expand entry is admitted but changes no answer. No code for this position.
• $expand deeper than one level: served — a depth-2 entry's own where, and a depth-1 entry whose where is a relation condition onto the grand target, made the expanded record's presence depend on the stored value.
• row position (control): already withheld (omit), unchanged.

Fix, by location

• packages/objectql/src/engine.ts
  • rejectCredentialAggregation: the refusal now carries the ADR-0112 envelope (INVALID_FIELD / 400, located at object + field) instead of the bare Error. Covers the group-by and aggregate-operand positions for every caller; the secret / password case that shared the bare Error is upgraded with it.
  • expandSearchOnAst: internal fields are dropped from the set the search expansion resolves over (a withhold, like the auto-default's hidden / credential-type exclusions), so $search never scans one — for find / findOne / aggregate, and for searchAll through them.
• packages/metadata-protocol/src/protocol.ts (the generic data door, findData)
  • assertNoInternalFieldEvaluated: refuses an internal field in the filter, sort, group-by, aggregate-operand and per-aggregation-filter positions, after the field-existence gates and before the engine — the shape, envelope and caller-independence of the stored-metadata body/hash family one axis over.
    • Every filter position runs through refuseInternalInFilter, which judges the filter's own reads and then each nested-relation condition (found by relationConditionSites) recursively against the RELATED object's own flag reader; the refusal names the object that declares the field.
    • refuseInternalInExpand walks the whole $expand tree: every level's where and orderBy against that level's target object.
  • assertSearchFieldsAreSearchable: an explicit $searchFields naming an internal field is refused INVALID_FIELD / 400 rather than silently widened back to the default set.

The engine's privileged consumers (the credential verifier's lookup by its stored digest under a system context; the share-link / SCIM / approval-token lookups) call the engine directly and never pass the generic data door, so authentication is untouched — exactly as #7823 kept the engine's write results whole while the generic-data-path ingress stripped them. This is pinned by the NEGATIVE case in the dogfood file and by internal-fields.test.ts's "keeps the field usable as a WHERE filter".

Compile-surface checklist

No filter COMPILATION semantics change; the fix is a refusal at the data-door ingress (and the engine aggregate-refusal envelope + the search field-set withholding), before any filter compiler runs. Each named surface is out of scope:

1. driver-sql applyFilterCondition (and driver-sqlite-wasm / local driver-turso by inheritance) — out of scope: untouched; refusal precedes the driver.
2. driver-turso RemoteTransport buildWhereSQL — out of scope: untouched.
3. service-analytics compileScopedFilterToSql — out of scope: untouched; the analytics door's own internal-field refusal landed in PR fix(service-analytics)!: the analytics door judges the generic-exit declarations — an object's enable block and a field's internal flag (#22634) #22645 (security(analytics): the ad-hoc analytics query serves objects that declare apiEnabled: false and columns declared internal: true, which every other generic exit refuses or withholds #22634).
4. service-analytics lowerAnalyticsWhere — out of scope: untouched.
5. formula matchesFilterCondition — out of scope: untouched.

• half-surface: objectql applyHaving / matchesHaving — out of scope: untouched; having names the aggregated row's columns, where an internal field cannot be named.
• driver-mongodb translateFieldOperators — out of scope: untouched.

Tests

• New pins: packages/objectql/src/internal-field-positions.test.ts (aggregate envelope + $search withhold, with controls), packages/metadata-protocol/src/protocol.data-door-internal-field-positions.test.ts (every evaluate position refused, controls), packages/qa/dogfood/test/internal-field-data-door-positions.dogfood.test.ts (member + admin on a real stack, plus the authentication-still-works negative).
• Each negative pin is ablation-verified: the door refusal, the aggregate envelope, the $search withhold and the explicit-$searchFields arm were each mutated on disk (scripts/ablation-replace.mjs), the pins observed to fail, and the file restored byte-for-byte (git diff HEAD empty).
• @objectstack/objectql and @objectstack/metadata-protocol build and typecheck clean. Affected-surface regression: objectql aggregate/search/judge files (145 tests) and metadata-protocol data-door/search files (159 tests) pass. Implicated gates run green locally (changeset family, error-status-conformance, dispatcher-error-vocabulary, engine-double-contract, test-source-alias, query-options-erasure, where-matcher, filter-alias-parity, issue-citations, nul-bytes, and others); the full farm is CI's.

Docs

The docs-drift list's hand-written pages were read, the data-door pages in full. One correction: content/docs/api/error-catalog.mdx's INVALID_FIELD cause said only that a named field does not exist; it now adds that the same code refuses an existing internal: true field in an evaluate position, naming the positions. No other page claims an internal field is usable in filter, sort, $search or $expand. content/docs/releases/ is untouched.

Patch round 1 (seat review 6100268814)

New pins: 14 member + administrator refusal cases (relation condition in four spellings, $expand depth-2 filter and sort, depth-1 relation where onto the grand target) and 6 controls; protocol.data-door-internal-field-positions.test.ts 41/41. Ablations from a committed head, restored byte for byte: the relation-site recursion emptied (10 of 41 red) and the expand-tree recursion removed (4 of 41 red). The engine is unchanged this round.

Scope

Target-OBJECT exposure through $expand is a separate axis (#22661). The cross-object search sweep (searchAll) is #22640's region; it inherits the engine's search-set withholding through engine.find with no change to its own code.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JfJfBUC3cQ6hhgm9MQK76T

[github-actions]

📓 Docs Drift Check

This PR changes 2 package(s): @objectstack/metadata-protocol, @objectstack/objectql, touching 22 documentable anchor(s).

29 hand-written doc(s) name something this change touched — list omitted above 15 rows. Re-derive on the tree named below: node scripts/docs-audit/affected-docs.mjs --json 61bea24207c5d366b5b59568890b2567caf89d29.

⛔ 12 release-owned page(s) also affected — read-only, see AGENTS.md Documentation Guardrails.

What this run could not see

• 1 anchor(s) matched too much of the corpus to be a work list: ObjectQL (symbol, 72 pages)
• the SDK route bridge reached 54 of 206 client-bound route-ledger rows — the other 152 have no registrar path: tail to select them, so pages documenting THEIR client methods cannot appear above, on this or any run. Of those 152: 0 are remediable by widening that discovery convention (an in-repo file declares the path; the convention did not scan it); 55 are structural — on a ledger where NOT ONE row is declared in-repo, so no discovery change reaches them at any price; 97 are undecided (no in-repo declaration, on a ledger that has other in-repo registrars — absence and an unreadable spelling are not distinguishable here). The rows themselves: node scripts/docs-audit/affected-docs.mjs --bridge-coverage
• a page that states a rule by its inputs shares no identifier with the emitter that implements the rule, so an emitter-only diff cannot list it — not on this run and not on any run. Measured on fix(driver-sql): emit varchar(maxLength) for a text field a declared index keys on #11430: content/docs/protocol/objectql/types.mdx documents the text-family column mapping by the ObjectQL type names it maps FROM (text / textarea / html) while the diff changed createColumn; it went unlisted, and it was the page that diff falsified, in four places. No shared token exists to detect this on, so a rule your change carries has to be re-read by hand in the pages that restate it.
• a key NAME is not a key, so the hand re-read the line above prescribes can land on the wrong schema. The same spelling is authorable on one governed type and a [REMOVED] tombstone on another for each of active, aria, joins, objects, template, tools and version (censused on [finding] tools is a key on BOTH AgentSchema (tombstoned, dead) and SkillSchema (live, cloud-attested), so a name-based search attributes skill examples to the agent key — it produced a false stop-the-line alarm on PR #19059 #19093 over the liveness ledger's governed types, top-level keys); nothing in a search result distinguishes the two, so a grep hit on a LIVE example reads as evidence about the DEAD key. Measured on fix(spec): the agent.tools liveness row says dead — it claimed live on a key the schema tombstoned #19059: content/docs/ai/agents.mdx was reported as contradicting the agent.tools tombstone over its tools: example at :161, which is inside the defineSkill({ block opened at :155 — the page was already correct. Settle ownership by PARSING the value against both schemas, never by the name: that literal PASSES SkillSchema, and as an AgentSchema it FAILS at tools with the tombstone prescription. ⛔ These names are not the whole class — a key retired through a .strict() guidance map leaves no tombstone in the walked shape and none of them here (tool.category, live as AIToolDefinition.category).

Coarse fallback — 24 page(s) merely mention a changed package (the pre-#9192 predicate, kept for the deliberately-wide backstop): node scripts/docs-audit/affected-docs.mjs --json 61bea24207c5d366b5b59568890b2567caf89d29 → packageMentionDocs.

Which tree this was computed on

This run read content/docs from 6b3ba9d5f1e10e4ae2422f9c2d7df8af300cb3c2 — the merge of head 4191804adbfba888048b4960c4c72eab41f76a45 into base 61bea24207c5d366b5b59568890b2567caf89d29, which is what actions/checkout gives a pull_request run. Not the PR head.

A worktree cut from an older main holds a different content/docs, so re-deriving there can legitimately return a different list — that is a different tree, not a wrong row. To answer on the same tree:

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 6b3ba9d5f1e10e4ae2422f9c2d7df8af300cb3c2 && git checkout 6b3ba9d5f1e10e4ae2422f9c2d7df8af300cb3c2
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 61bea24207c5d366b5b59568890b2567caf89d29 4191804adbfba888048b4960c4c72eab41f76a45 && git checkout -B drift-repro 61bea24207c5d366b5b59568890b2567caf89d29 && git merge --no-ff 4191804adbfba888048b4960c4c72eab41f76a45

node scripts/docs-audit/affected-docs.mjs --json 61bea24207c5d366b5b59568890b2567caf89d29

⚠️ That checkout carried uncommitted changes, so the commit above does not fully identify what was read.

Advisory only, and a precision-first one (#9192): a page is listed because it names a
symbol, wire route or SDK method this diff touched — not because it mentions a changed
package. Each row says which anchor put it there, so a wrong row is reportable rather than
merely annoying. To re-verify, run the docs-accuracy-audit workflow scoped to these files:
node scripts/docs-audit/affected-docs.mjs 61bea24207c5d366b5b59568890b2567caf89d29 → pass the list as
args.docs, on the commit named under Which tree this was computed on.

[objectstack-fleet]

Contract review

Served-tier: CONTRACT_REVIEW_TIER
Head-sha: 4191804adbfba888048b4960c4c72eab41f76a45
Local-runs: none

Card #22646 (security, p1) · PR #22702 · the net diff against the merge-base with main (8 files, +883/−4) · read at 2026-10-10T18:49Z. Inputs: the card body and its five comments (triage 6095889166, claim 6099170359, os-dev-reports 6100232363 and 6100842864, seat rework 6100268814), the PR body and file list, the diff, and the head's check-runs — nothing else. ⛔ Classes, positions and functions only.

Check-runs on the head: 42 runs, 38 success, 4 skipped (Check PR Size and Auto Label first attempts, Console Pin Gate, Packed-tarball smoke — each path-conditional or opt-in), none failed. Test Core (3/6) was still in_progress on the first read and concluded success before this record was written. Governed Surface Queue Guard success: the file list touches no governed surface; the head repo is the base repo.

① Derived judgments

Every accept-set and public-surface change the diff implies, each judged against the head's own code:

1. @objectstack/metadata-protocol findData — a new refusal, INVALID_FIELD / 400, for a field declared internal: true in the filter (own reads, cross-field comparand included), sort, group-by, aggregate-operand and per-aggregation-filter positions, and in every expand level's own filter and sort against that level's target (assertNoInternalFieldEvaluated, refuseInternalInFilter, refuseInternalInExpand, refuseInternalNames). RIGHT. A narrowing of the generic door's accept set that binds every caller (no context carve-out). It sits after the field-existence gates and the stored-metadata body/hash refusals and before the aggregate routing, so an unknown field keeps its existing answer, a FilterAST array is lowered first (parseFilterAST), and the count leg, which reads the same where, is covered by the same judgement. The envelope (code, status, httpStatus, field, fields, object, param) is the family's, and error-catalog.mdx now states the behaviour.
2. Nested-relation conditions judged on the RELATED object (relationConditionSites, module-private, recursive through refuseInternalInFilter). RIGHT, and a superset of what the engine evaluates: the engine's admitRelationCondition (relation-filter-lowering.ts) refuses a dotted-key and a second-level relation condition and its walker skips dotted keys, so the engine lowers exactly one hop of a plain-record condition through lowerRelationConditions; the door finds every reference-field key whose value carries a non-$ key, under $or / $and / $not too, and judges it against the related object's own flag reader. A value holding only $ keys is a relation site for neither side (the engine refuses it as undeclared-key), so no shape the engine evaluates on the related object escapes the door. The existence gate ahead of it (collectFilterFieldKeys) does not descend into relation values, so the refusal names the declaring object, as the pins assert.
3. INTERNAL_FIELD_WALK_DEPTH = 8, a fail-open backstop past it. RIGHT at this head: the engine admits one relation hop (item 2) and ObjectQL.MAX_EXPAND_DEPTH = 3, so nothing the engine executes reaches depth 8. Recorded that a fail-closed backstop would be the stricter posture should either engine cap ever rise.
4. Every expand level walked, the target resolved through the source field's referenceTargetOf over REFERENCE_VALUE_TYPES. RIGHT — the same resolution expandRelatedRecords uses, so the door's tree is the engine's tree; a nested entry's where is the one key the sub-read forwards as a filter (wire aliases are refused in a nested entry), and the door judges it with its relation sites. The door also refuses an expand entry's sort on the target's internal field although the expansion's order is unobservable (the sub-read's rows are injected keyed by foreign-key id; no limit is forwarded): a conservative over-refusal, declared in the changeset, not a defect.
5. assertSearchFieldsAreSearchable — an explicit $searchFields naming an internal field is refused INVALID_FIELD / 400 before the searchable-set resolution. RIGHT: the REST 读路径：searchFields / groupBy / aggregations 指向不存在的字段时被静默降级（#4226 收口后剩下的三条轴） #4254 posture, an unhonoured narrowing is a 400 and never a silent widening. Both spellings (the standalone parameter and the object-form search.fields) pass through the one function.
6. @objectstack/objectql expandSearchOnAst — internal fields dropped from the scan set the search expansion resolves over (find / findOne / aggregate, and searchAll through engine.find). RIGHT: a withhold keyed on collectInternalReadFields. searchAll forwards no caller filter or sort (only the term, the narrowed searchable set and a fixed updated_at order), so it inherits the withhold with no code of its own. A direct engine caller naming an internal field in searchFields has it intersected away — the engine's existing tolerant posture for a requested search field it will not scan; the loud refusal is the door's.
7. rejectCredentialAggregation — the refusal carries INVALID_FIELD / 400 (status and httpStatus, field, fields, object) instead of a bare Error. RIGHT: additive to the thrown error (message unchanged), the code the door and the analytics door use for the same flag, taken from StandardErrorCode; the secret / password arm that shared the bare Error is upgraded with it; the sole call site is aggregate, which the door routes group-by and aggregations to.
8. One flag reader. The door judges collectInternalWriteResponseFields (@objectstack/core), the engine collectInternalReadFields (@objectstack/objectql). RIGHT: both read def.internal === true off the same declaration and return the same names — two readers of one flag, not a second list. @objectstack/metadata-protocol depends on core, not on objectql, so the objectql reader cannot be imported at the door, and the write-response strip and the analytics door already use the core twin there.
9. Dotted spellings left to the existing gates (the door's orderByFieldNames / groupByFieldNames judge the head only). RIGHT: assertSortFieldsExist refuses every dotted sort (INVALID_SORT); assertGroupByFieldsExist judges against the known set, so a dotted group key is unknown (INVALID_FIELD); assertFilterFieldsExist's dotted branch refuses a dotted filter key whose head is a relation, virtual or scalar field (classifyDottedFilterHead); and the engine's walker skips dotted keys. No dotted spelling reaches an evaluation of the flagged column.
10. Positions unchanged, and correctly so: the row position (the engine strip, select-named included); getData by id, whose expand is a name list with no filter or sort; the engine's direct privileged consumers (the Check whether sys_session.token — a live session credential — serializes over the data API (ADR-0100 channel 3 has no read protection) #7823 split), pinned by the dogfood NEGATIVE (the credential still authenticates) and by internal-fields.test.ts.
11. Pins: protocol.data-door-internal-field-positions.test.ts (41: every position, member and administrator, engine never asked; controls reach the engine), internal-field-positions.test.ts (6: envelope, search withhold, controls), the dogfood file (15: both personas on a real stack, controls, NEGATIVE). The position enumeration is the load-bearing shape the card asked for. engine-double-contract.pinned.json gains the one row the new fake findOne pin owes. Ablations A–F are the dev's report, not re-run here (read-only); each pin they name is in the diff and green on the head.
12. No new export, error code, spec key or stored shape. relationConditionSites and INTERNAL_FIELD_WALK_DEPTH are module-private, the class methods private, INVALID_FIELD an existing catalog member. RIGHT.

Nothing judged wrong.

② Semver level

Clause-②: no (narrowing) on the PR body and in the changeset — RIGHT: no new key on a published payload; the accept set narrows (positions that evaluated a withheld value are now refused). (narrowing) is BREAKING, and the changeset is written as one: **BREAKING**, FROM → TO per position, bindings, scope, and exactly one ADR-0087 marker, not-required (no-migration-prescription), a category the gate declares — no metadata moves, no authorable spelling, export or stored shape is removed or renamed, so there is nothing for a migration to rewrite. Level minor on @objectstack/objectql and @objectstack/metadata-protocol: RIGHT — the two published packages the diff moves, at the ceiling the launch-window guard (check-changeset-no-major) allows a breaking changeset. @objectstack/dogfood is private: true; content/docs and scripts/ publish nothing. Check Changeset success on the head.

③ Boundary flags

Dev open_questions: [] in both reports — none to answer. Every dev flag, deviation and out-of-scope note, answered:

• R1 deviation — refusals at the metadata-protocol door rather than the engine (the Check whether sys_session.token — a live session credential — serializes over the data API (ADR-0100 channel 3 has no read protection) #7823 split): ANSWERED, right; the privileged engine callers never pass the door, pinned by the NEGATIVE.
• R1 out-of-scope finding — expand deeper than one level: ANSWERED, closed by round 1 (every level walked, ① item 4).
• R2 (a) — position 2 (a dotted sort or group-by path through a lookup) needs no code: ANSWERED, verified at the head (① item 9); an expand entry's dotted sort changes no answer (① item 4).
• R2 (b) — the nested-relation condition judged at the door, not in the engine's walker: ANSWERED, right (① item 2).
• R2 (c) — error-catalog.mdx cause line corrected: ANSWERED, the page now states the refusal.
• R2 (d) — changeset title / FROM / TO renamed for the two positions: ANSWERED (②).
• R2 (e) — PR body not patched by the dev: the body on this head carries the Patch round 1 section and the measurement lines; no flag open.
• R2 NOT-MEASURED gates (check:dual-build-cjs-loads, spec check:skill-examples): CI-owned; Build Core and Lint & Repo Gates success on the head.
• Seat rework 6100268814, three positions: (1) relation filter — fixed and pinned; (2) dotted sort / group path — measured unreachable, verified; (3) expand depth 2 and deeper — fixed and pinned. All closed.
• PR compile-surface checklist (seven surfaces out of scope): RIGHT — the refusal precedes every filter compiler; the analytics door's own refusal is PR fix(service-analytics)!: the analytics door judges the generic-exit declarations — an object's enable block and a field's internal flag (#22634) #22645.
• PR scope lines — target-object exposure through expand is security(data, analytics): a lookup target's exposure declaration is not judged when the data door's $expand, or the dataset door's dimension-label pass, reads it — detail withheld pending maintainer #22661, the cross-object search sweep is metadata-protocol: the cross-object search (searchAll) skips an object on searchable / apiEnabled only and never consults the apiMethods whitelist, so a whitelist that omits list (and so search) does not keep an object out of the sweep #22640: RIGHT, disjoint regions; searchAll inherits the engine withhold (① item 6).
• Card ask "⛔ no second list": ANSWERED (① item 8), no escalation.
• Depth-8 fail-open backstop (① item 3): unreachable at this head; recorded, not escalated.

Nothing escalated.

Implemented-by: claude/issue-22646-internal-field-positions
Reviewed-by: session_01JfJfBUC3cQ6hhgm9MQK76T

VERDICT: PASS

---

Generated by Claude Code