Skip to content

Commit 4cd104f

Browse files
os-zhuangclaude
andauthored
docs(qa): R4 coverage sweep — retire both stale waivers, author 8 items for the post-2026-08-08 window (#9309)
* docs(qa): R4 coverage sweep — retire both stale waivers, author 8 items for the post-2026-08-08 window (#9299) Re-audit of the platform-checklist ledger against the window since the 2026-08-08 sweep. 182 -> 190 items; coverage 28 mapped / 2 waived -> 30 mapped / 0 waived. Both surviving coverage waivers were factually stale. `book` and `doc` both claimed the kinds are "display-only, no independent runtime behavior to gate beyond serving". The REST read layer in fact runs a three-member audience vocabulary with an ADR-0049 fail-closed branch, a two-layer gate (book audience, then per-entry doc-audience union), a deliberate orphan-group non-leak rule, and locale resolution -- an access-control surface. Retired both; `platform-core.docs-audience-gate` now covers them. Running total: 6 of 6 waivers ever written turned out stale. New items, each grounded in a real file path before assertion: - platform-core.docs-audience-gate book/doc audience gating - access-security.no-active-org-session-semantics ADR-0123 (P0) - api-backend.filter-comparand-conformance pinned, 6 members - api-backend.date-range-preset-matrix pinned, 13 members - api-backend.retired-def-refusal retired DEFS, not keys - integration-system.datasource-credential-refusal-matrix (P0) - cli.migrate-meta-codemod the prescription's honesty - cli.scaffold-console-first-paint documented first-run seam The dominant find matches the ledger's structural blind spot: new behavior inside an already-mapped kind is invisible to both ratchets (no new kind, no new enum member), so none of the above could have been caught by `check:platform-checklist` however often it ran. SWEEP.md fact correction: the runbook claimed the validator runs in "CI, every PR". It does not -- by maintainer decision it is manual and periodic (README "Operating cadence"; the recorded note at .github/workflows/lint.yml:426, which has no `run:` step). Corrected the fact and spelled out the consequence a sweep depends on: the ratchets catch drift at the NEXT manual run, so a green ledger is not evidence that drift is gated. CI posture itself left untouched. FOLLOW-UPS.md gains a 2026-08-17 section: one docs-drift finding (the blank template's `dev` script omits the `--ui` flag that serves the console quick-start sends newcomers to), one stale liveness note, and the checked-and-clean readings so the next sweep need not re-derive them. Fixes #9299 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ai9gUdihXjg7FumzsBNSSs * docs(qa): retract the --ui divergence finding — source check refutes it (#9299) The R4 sweep flagged the blank template's bare `"dev": "objectstack dev"` against quick-start's `npx os dev --ui` and inferred that `npm run dev` would strand a newcomer with no console. It was reported as unbooted, and the hedge was right: tracing the chain disproves it. serve.ts:221 ui: Flags.boolean({ ..., default: true, allowNo: true }) dev.ts:370 ...(flags.ui ? ['--ui'] : []) adds only, never --no-ui dev.ts:69 ui declared with NO default cli.mdx:139 "--ui | Force Console UI on (already on by default in dev)" An unflagged `dev` spawns `serve` with no ui flag, so serve's own default-on applies. `--ui` on `dev` is a no-op forwarder. The bare script is correct and there is no divergence. An expected-divergence probe that encodes correct behavior as a divergence teaches the next run to grade a working platform as broken — the records-forms.action-location-matrix failure mode. So: - FOLLOW-UPS.md: the E1 drift row is gone; the reading is rewritten as a checked-and-clean entry in 5b with the full trace, because the wrong inference is an easy one to make twice and deletion would invite it. The stale liveness-note row renumbers E2 -> E1. - cli.scaffold-console-first-paint: clause 3 now asserts the two invocations AGREE, with a difference between them as the failure; the negatives cover the real regressions (a future `ui` default of false on dev, or --no-ui being forwarded); source cites the whole chain; the history entry records the refutation so it is not re-derived. The item itself stands — the seam it covers is real: no other item drives the console against a freshly scaffolded blank project. Validator green, unchanged at 190 items / 30 kinds mapped / 0 waived. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ai9gUdihXjg7FumzsBNSSs --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 68cd4e7 commit 4cd104f

8 files changed

Lines changed: 876 additions & 9 deletions

File tree

‎docs/qa/platform-checklist/FOLLOW-UPS.md‎

Lines changed: 70 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
# Follow-ups — open items from the capability-coverage sweep (2026-08-08)
1+
# Follow-ups — open items from the capability-coverage sweeps
2+
3+
Standing decision register, one section per sweep (append, never rewrite — a resolved
4+
row stays with its resolution so the next sweep can see what was already decided).
5+
Sections §1–§4 are the **2026-08-08** sweep; §5 is the **2026-08-17** re-audit (R4).
26

37
Decision register from the capability-coverage sweep. The gap items found by the
48
five-angle sweep have all been authored into `areas/*.json` (checklist grew 84 → 170
@@ -67,3 +71,68 @@ these to the showcase would make them runnable:
6771
- The checklist itself (`areas/*.json`, `coverage.json`, `README.md`, `RUNNER.md`,
6872
`scripts/check-platform-checklist.mjs`) ships in this branch; this file carries the
6973
decisions that remain with the maintainer.
74+
75+
## 5. Sweep 2026-08-17 (R4) — re-audit against the window since the 2026-08-08 ledger
76+
77+
Ledger 182 → 190 items; `coverage.json` 28 mapped / 2 waived → **30 mapped / 0 waived**.
78+
79+
### 5a. Docs drift (PD#10 class)
80+
81+
| # | drift | evidence | captured in |
82+
|---|---|---|---|
83+
| E1 | **Stale defect note in `packages/spec/liveness/doc.json`.** Its `_note` records "DocSchema declares no `tags`, yet the book-side `include: { tag }` rule and the REST corpus both expect one — the tag rule can currently never match". That defect is **fixed**: `DocSchema` now declares `tags` (`packages/spec/src/system/doc.zod.ts:126`, with the history spelled out in the surrounding comment). The ledger note now describes a bug that no longer exists, which is the same failure class this sweep is correcting in `SWEEP.md`. Outside this card's file surface (`docs/qa/platform-checklist/**`), so it is reported, not edited. | `packages/spec/liveness/doc.json` `_note` vs `packages/spec/src/system/doc.zod.ts:111-127` | — (liveness ledger prose, not a checklist item) |
84+
85+
### 5b. Checked and CLEAN (recorded so the next sweep does not re-derive it)
86+
87+
- **The blank template's `dev` script omits `--ui`, and that is CORRECT — the console is
88+
served anyway.** This sweep first read `"dev": "objectstack dev"` against the
89+
quick-start's `npx os dev --ui` and inferred that a newcomer running `npm run dev`
90+
would land on a server with no console. **Source-checking the chain refuted it**, and
91+
the refutation is recorded here because the inference is an easy one to make twice:
92+
- `packages/cli/src/commands/serve.ts:221` — `ui: Flags.boolean({ …, default: true,
93+
allowNo: true })`. The console is **default-ON** at `serve`; `--no-ui` is the off
94+
switch.
95+
- `packages/cli/src/commands/dev.ts:370` — `...(flags.ui ? ['--ui'] : [])`. `dev` only
96+
ever **adds** `--ui`; it never forwards `--no-ui`. With `dev.ts:69` declaring `ui`
97+
with no `default`, an unflagged `dev` spawns `serve` with no ui flag at all, so
98+
serve's own default takes over — on.
99+
- `content/docs/deployment/cli.mdx:139` says it outright: "`--ui` | Force Console UI on
100+
(**already on by default in dev**)", and `:196` documents `--ui / --no-ui … (default
101+
on)`.
102+
103+
So `--ui` on `dev` is a **no-op forwarder**, there is no divergence between the
104+
template script and quick-start, and the newcomer is not stranded. What made the wrong
105+
reading tempting is `dev.ts:69`'s own flag description ("Enable the bundled Console
106+
portal at /_console/"), which reads like the mechanism when it is only a forwarder —
107+
not worth a change on its own, but worth knowing before inferring from it.
108+
`cli.scaffold-console-first-paint` clause 3 now asserts the two invocations **agree**,
109+
with a difference between them as the failure.
110+
- **No published doc prescribes a retired scaffolder template.** Grepped all of
111+
`content/docs/` for `todo` / `compliance` / `content` / `contracts` / `procurement` as
112+
`-t` / `--template` operands: zero hits. Every documented invocation is a bare
113+
`npx create-objectstack my-app`, which resolves to `blank`, the only entry in
114+
`TEMPLATES`. The delisting is cleanly reflected on the docs side.
115+
- **`npm run validate` and `npx os validate` are the same binary** (the blank template's
116+
`validate` script is `objectstack validate`), so the spelling difference between
117+
`cli.scaffold-first-run` and the quick-start is not a divergence.
118+
- **The first-run *commands* are covered** by `cli.scaffold-first-run` (P1, CI-automated)
119+
and `studio-authoring.first-run-loop` (P0). Only the seam between them was open, and
120+
that is now `cli.scaffold-console-first-paint`. No duplicates were authored.
121+
122+
### 5c. For the maintainer
123+
124+
- **Both surviving coverage waivers were stale** (`book`, `doc`) and are retired in this
125+
PR — see `SWEEP.md` "Discipline". Running total: 6 of 6 waivers ever written turned out
126+
stale. Worth considering whether a waiver should carry a mandatory re-audit date.
127+
- **Cost figures in the corrected `SWEEP.md` sentence are this run's own measurements**,
128+
which came out higher than the ones supplied with the card (validator ~0.12–0.18 s vs
129+
~80 ms; `pnpm check:platform-checklist` ~2.9 s wall vs 0.165 s). The container was under
130+
load from a parallel cold monorepo install, and the `pnpm` wrapper's startup dominates
131+
the wall figure. The corrected text therefore records the *direct-node* cost (~0.25 s
132+
for self-test + validator) and explicitly warns that timing through `pnpm` measures the
133+
wrapper — a durable statement rather than a number that goes stale.
134+
- **No security-sensitive finding to withhold from this PR.** The two highest-severity
135+
items authored (`access-security.no-active-org-session-semantics`,
136+
`integration-system.datasource-credential-refusal-matrix`) assert guards that are
137+
already shipped and already public in their ADRs/issues; nothing here discloses an
138+
unfixed hole.

‎docs/qa/platform-checklist/SWEEP.md‎

Lines changed: 31 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,28 @@ executable by the AI session itself. Expected cadence: before each major release
66
after any large platform surface lands.
77

88
This is the AI-participation half of keeping the checklist current. The other half is
9-
automatic and needs no human at all: `scripts/check-platform-checklist.mjs` (CI, every
10-
PR) fails when a new metadata **kind** is unmapped (coverage ratchet) or a spec **enum**
11-
grows a value a matrix item was pinned against (`enumSource` freshness ratchet). Those
12-
catch drift on the PR that causes it. This sweep catches the harder class — a whole
13-
surface or behavior nobody wrote an item for — which no deterministic gate can find.
9+
**mechanical but still operator-triggered**: `scripts/check-platform-checklist.mjs` fails
10+
when a new metadata **kind** is unmapped (coverage ratchet) or a spec **enum** grows a
11+
value a matrix item was pinned against (`enumSource` freshness ratchet).
12+
13+
⚠️ Those ratchets are **NOT wired into per-PR CI** — by maintainer decision they run on a
14+
periodic **manual** cadence (see this dir's [README.md](./README.md) "Operating cadence",
15+
and the recorded note at `.github/workflows/lint.yml:426`). So they do **not** catch drift
16+
on the PR that causes it; they catch it at the **next manual run**. Do not read a green
17+
ledger as "drift is already gated" — between runs, an unmapped kind or a grown enum sits
18+
undetected on `main`, which is precisely why a sweep must re-derive from source rather
19+
than trust the last green check. Cost is not the reason it stays out of CI: measured
20+
2026-08-17 on a 190-item ledger, the validator alone is ~0.12–0.18 s and the full
21+
`pnpm check:platform-checklist` (selector self-test + validator) is ~0.25 s of actual
22+
work, zero-dependency and token-free — the `pnpm` wrapper's own startup dominates any
23+
figure you measure through it, so time the scripts directly if the number matters. The
24+
reason is PR independence: the checklist is a QA ledger, not a code gate, so an unrelated
25+
PR is never blocked by checklist drift.
26+
27+
This sweep catches the harder class — a whole surface or behavior nobody wrote an item
28+
for — which no deterministic gate can find. Note what that leaves uncovered even in
29+
principle: **new behavior inside an existing kind** is invisible to BOTH ratchets (no new
30+
kind, no new enum member), so it is found only here.
1431

1532
## What a sweep is
1633

@@ -58,6 +75,14 @@ single reader would not have covered.
5875
corrected them against source rather than parroting.
5976
- **Stale waivers are the highest-value find.** Four of six coverage waivers turned out
6077
false (api/datasource/mapping/hook all ship stock fixtures). Re-audit every waiver
61-
each sweep — a waiver is a claim that ages.
78+
each sweep — a waiver is a claim that ages. **Confirmed again 2026-08-17**: the two
79+
waivers that survived that sweep (`book`, `doc`) were *also* false. Both claimed the
80+
kinds are "display-only, no independent runtime behavior to gate beyond serving"; the
81+
REST read layer in fact runs a three-member audience vocabulary with an ADR-0049
82+
fail-closed branch, a two-layer gate (book audience, then per-entry doc-audience
83+
union), a deliberate orphan-group non-leak rule, and locale resolution — an
84+
access-control surface. `coverage.json` now carries **zero** waivers. Running total:
85+
**6 of 6** waivers ever written turned out stale, which is the strongest argument in
86+
this file for auditing them against the runtime rather than against their own prose.
6287
- **Defects found while grounding go to FOLLOW-UPS.md as expected-fail probes**, not
6388
silent passes; security-sensitive ones are not filed publicly without the maintainer.

‎docs/qa/platform-checklist/areas/access-security.json‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1910,6 +1910,115 @@
19101910
"ref": "claude/issue-7680-package-capability-doctrine"
19111911
}
19121912
]
1913+
},
1914+
{
1915+
"id": "access-security.no-active-org-session-semantics",
1916+
"title": "A session with no active organization is a LEGAL state with fail-closed semantics: tenant-scoped reads resolve to nothing, tenant-scoped writes are refused loudly naming the missing org, and nothing is ever NULL-stamped",
1917+
"since": "v17",
1918+
"status": "active",
1919+
"revision": 1,
1920+
"priority": "P0",
1921+
"surface": "api",
1922+
"personas": [
1923+
"a freshly signed-up user on their FIRST session (structurally predates their sys_member row)",
1924+
"a user removed from their organization",
1925+
"seeded admin (control — has an active organization)"
1926+
],
1927+
"fixtures": {
1928+
"app": "showcase",
1929+
"requires": [
1930+
"a tenancy-walled deployment (ADR-0095 Layer 0 active on at least one tenant-scoped object) — confirm the posture before scoring, since on an unwalled deployment every clause here is vacuous",
1931+
"the ability to complete a signup and use the resulting first session before the membership reconciler settles, OR to remove a member and reuse their live session"
1932+
],
1933+
"knownGaps": [
1934+
"The state is timing-derived on the signup path (better-auth defers the membership write past the signup transaction, ADR-0093), so reproducing it by racing signup is flaky. The DURABLE reproduction is the removed-member path, or a multi-organization deployment whose reconciler binds nobody (ADR-0093 D1 `no-target-org`). Record WHICH producer the run used — a clause proven only on a racy producer is weaker evidence and must say so.",
1935+
"No stock showcase fixture is known to configure `invite-only` or SSO-JIT-pending; those two producers are blocked(fixture) unless the run configures them."
1936+
]
1937+
},
1938+
"steps": [
1939+
"establish the state and PROVE it: obtain a session whose `activeOrganizationId` is null, and cite the evidence (session row / session endpoint payload) — do not infer the state from the symptom you are about to test",
1940+
"read a tenant-scoped object over the data API with that session",
1941+
"write (create) a row on a tenant-scoped object with that session",
1942+
"immediately read back, as the SAME caller, anything the write may have produced",
1943+
"query the underlying table as admin for rows whose tenant column is NULL, before and after the write attempt",
1944+
"repeat the read and the write against a NON-tenant-scoped object to bound the blast radius",
1945+
"sign in and out with the state in force, then inspect the audit ledger rows the session produced (plugin-audit auth-session-audit login/logout)",
1946+
"run the same read+write as the control admin persona to show the refusal is state-derived, not a blanket denial"
1947+
],
1948+
"acceptance": [
1949+
{
1950+
"clause": "tenant-scoped READS resolve to nothing — an empty result set, NOT an error: Layer 0's deny sentinel (RLS_DENY_FILTER) yields zero rows and a 200",
1951+
"oracle": "api",
1952+
"verify": "GET returns 200 with an empty collection; a 4xx/5xx here is a FAIL (the read side is deliberately not an error path)",
1953+
"evidence": "status + body"
1954+
},
1955+
{
1956+
"clause": "tenant-scoped WRITES are refused LOUDLY with a 4xx whose message NAMES the missing active organization — not a generic denial, not a 500",
1957+
"oracle": "api",
1958+
"verify": "status is 4xx; the error `code` is drawn from the CLOSED ADR-0112 vocabulary (an invented code is a FAIL of D3) and the message text names the missing active organization",
1959+
"evidence": "status + full error envelope, quoted"
1960+
},
1961+
{
1962+
"clause": "⛔ NO silent NULL stamping, anywhere: the refused write leaves NO row behind, and no subsystem writes a row whose tenant column is NULL because the caller had no organization to supply",
1963+
"oracle": "api",
1964+
"verify": "the admin-side NULL-tenant row count is IDENTICAL before and after the attempt; a row that appears and is invisible to its own creator is the #8208 defect and is a P0 FAIL",
1965+
"evidence": "both row counts + the diff"
1966+
},
1967+
{
1968+
"clause": "the refusal does not fall OPEN in the other direction: an org-scoped permission holder with no active organization reads and writes NOTHING across tenants (the #8158 class — `scope: 'org'` grants nothing when there is no org)",
1969+
"oracle": "api",
1970+
"verify": "as the stateless caller holding an org-scoped permission (e.g. manage_sharing), attempt a cross-tenant read and write; both must be empty/refused, never another tenant's rows",
1971+
"evidence": "responses + a cross-check that another tenant's rows exist and were not returned"
1972+
},
1973+
{
1974+
"clause": "the audit carve-out is paid for by ORDERING, not by an exemption: audit rows from a first session carry a real tenant — no audit row is NULL-tenanted and therefore permanently invisible to RLS readers (#8245)",
1975+
"oracle": "api",
1976+
"verify": "read the login/logout audit rows the session produced as a tenant-scoped reader; they must be visible. A row present in the raw table but invisible to every RLS reader is a FAIL even though nothing errored",
1977+
"evidence": "raw rows + the RLS-scoped read of the same rows"
1978+
},
1979+
{
1980+
"clause": "the state is LEGAL, not a crash: the session authenticates, non-tenant-scoped surfaces still work, and nothing 500s",
1981+
"oracle": "api",
1982+
"verify": "the session's whoami/session endpoint answers 200; a read on a non-tenant-scoped object succeeds; no 5xx anywhere in the run",
1983+
"evidence": "the response set"
1984+
},
1985+
{
1986+
"clause": "three subsystems agree — sharing, the Layer 0 wall + write path, and the audit ledger give the SAME answer for the SAME state (this item exists because they once gave three contradictory ones on the same working day)",
1987+
"oracle": "api",
1988+
"verify": "the run record states each subsystem's observed behavior side by side; any disagreement is the finding, whichever way it leans",
1989+
"evidence": "the three-way comparison table"
1990+
}
1991+
],
1992+
"negative": [
1993+
"fell OPEN: an org-scoped holder with no active organization reading or writing another tenant's rows (#8158) — P0",
1994+
"fell CLOSED SILENTLY: an HTTP-created record stamped organization_id NULL and immediately invisible to its own creator (#8208) — P0, and the reason the NULL-count clause is measured rather than inferred",
1995+
"wrote INVISIBLE rows: audit rows carrying a NULL tenant, permanently unreadable through RLS (#8245)",
1996+
"a 500 or an unhandled rejection on any of the three paths — the state is declared legal, so crashing is a defect",
1997+
"a refusal whose message does NOT name the missing active organization: it is indistinguishable from an ordinary permission denial and sends the debugger to the wrong layer",
1998+
"an error code invented for this refusal instead of drawn from the ADR-0112 closed vocabulary"
1999+
],
2000+
"traps": ["auth-state-leak", "eventual-consistency"],
2001+
"source": [
2002+
"ADR-0123 (an authenticated session with no active organization — reads resolve to nothing, tenant-scoped writes refused loudly), accepted 2026-08-13",
2003+
"ADR-0095 D1/W1/W2 (Layer 0 independent always-first AND-composed tenant filter)",
2004+
"ADR-0105 D1/D2 (posture spectrum, group union wall, empty/absent scope → deny)",
2005+
"ADR-0093 D1/D2 (membership reconciler; session.create.before resolving activeOrganizationId from sys_member)",
2006+
"ADR-0112 D3 (closed error-code vocabulary this refusal draws from)",
2007+
"ADR-0049 (a declared wall the write path does not meet is the defect)",
2008+
"#8247 (the ruling), from #8158 (fell open) · #8208 (fell closed silently) · #8245 (wrote invisible rows)",
2009+
"PR #8471 (ADR-0123 + enforcement)",
2010+
"@objectstack/plugin-security security-plugin.ts step 3.7 + tenant-layer.ts (computeTenantLayer0Filter, RLS_DENY_FILTER)",
2011+
"@objectstack/plugin-auth auth-manager.ts composeDatabaseHooks + reconcile-membership.ts",
2012+
"@objectstack/plugin-audit auth-session-audit.ts (login/logout rows)"
2013+
],
2014+
"history": [
2015+
{
2016+
"revision": 1,
2017+
"date": "2026-08-17",
2018+
"change": "new — the ADR-0123 tenant-semantics wall had ZERO checklist coverage. Neither ratchet could see it: no new metadata kind (it is new behavior across the already-mapped permission/object kinds) and no new enum member. Authored P0 because the ADR records three independently measured production-shaped defects in three subsystems for one state, two of which are silent (a row invisible to its own creator; audit rows invisible to every RLS reader) and therefore invisible to any test that only asserts 'no error'",
2019+
"ref": "#9299"
2020+
}
2021+
]
19132022
}
19142023
]
19152024
}

0 commit comments

Comments
 (0)