Repository navigation
Commit db9cf80
feat(cli): os migrate security-catalog-overlays — list, and with --apply delete, the environment rows a v18 cold boot refuses (#22523)
Fixes #22371
Clause-②: yes (widening)
Item 1 of ruling A′ (record 6073500921), as amended by ruling letter B
(record 6074838935): the offline step that lists, and with `--apply`
deletes, the environment-wide rows a v18 cold boot refuses. Item 2
(retiring the three in-kernel remedies in `plugin-security`) is a
separate `domain:services` card and is not in this PR.
## What it is
`os migrate security-catalog-overlays`, a sibling in the `os migrate`
family (the `meta --stored` conventions: preview by default, `--apply`,
`--yes`, `--force`, `--json`, `--database-url` / `$OS_DATABASE_URL`).
**Why a sibling, not a mode of `os migrate meta --stored`.** The ruling
named `meta --stored` as the nearest landing, and the step departs from
it on three counts:
- **The operations differ.** `--stored --apply` rewrites every stored
row through `saveMetaItem`, while the step deletes one refused
population. Under one `--apply`, the flag would carry a second,
destructive meaning.
- **The boots differ.** `--stored` boots without the host config and
hydrates `sys_metadata`, which is exactly the boot the cold-boot refusal
stops over a compiled artifact. The step needs `serve`'s composition,
with the auth-gated security plugin and hydration off.
- **The exit contracts differ.** `--stored` exits 1 when rows remain
uncanonical; the step exits 1 when the next boot would be refused.
As a sibling, the step keeps the family's conventions (preview by
default, `--apply` / `--yes` / `--force` / `--json` / `--database-url`,
the occupancy gate), and the refusal message names it directly.
- **It lists** every active, environment-wide `sys_metadata` row
(`organization_id IS NULL`, `state = 'active'`) of type `permission` or
`position`, the legacy plurals `permissions` / `positions` included,
whose name a configured package holds: the population the cold-boot
check (ADR-0048 N.3) refuses. A draft row and an organization-scoped row
are never listed.
- **It composes what `os serve` composes, for the first phase only, and
hydrates nothing.** The host config's plugins and the application or
compiled artifact (the existing `composeHostStack` declaration boot),
plus the security plugin behind `serve`'s auth gate. The boot runs with
`sys_metadata` hydration off, so the cold-boot check meets an empty
environment half and the boot comes up with no server.
- **It reads the holders off the registry** through one new
`@objectstack/objectql` export,
`findPackageHeldSecurityCatalogNames(registry)`. The cold-boot check now
computes its conflicts from the same private reading (that list met with
the bare slot), so the step and the boot share one reading of "who holds
a name".
- **`--apply` deletes through the engine's audited write path.** A row
stored under the canonical type goes through the protocol's
`deleteMetaItem` (the `DELETE /api/v1/meta/TYPE/NAME` door): a
`sys_metadata_history` tombstone and a `sys_metadata_audit` row, actor
`os migrate security-catalog-overlays`. A legacy-plural row is out of
that door's reach (it folds the type before reading, and the audit trail
refuses a non-canonical type), so it goes through the
`SysMetadataRepository` delete beneath it, addressed by its stored type,
name, `package_id` and checksum, which writes the same history
tombstone. One audit line per row names the door it took. Nothing is
adopted; `managed_by` and `package_id` are never rewritten.
- **Exit status.** Preview: 1 when it lists a row (the next boot would
be refused), 0 when it lists none. `--apply`: 0 when every listed row is
gone, 1 when one could not be deleted. A host config that exists and
cannot be loaded is refused before any row is read.
- **The refusal names it.** The cold-boot `NAMESPACE_CONFLICT` message
now points at the step for the environment's rows. The envelope is
unchanged.
## The first reading: does `materializeStackPlugin` cover the auth-gated
security plugin?
No. `materializeStackPlugin` (`packages/core/src/stack-plugins.ts`, from
`97610a533`) is the rule for one entry of a stack's own `plugins` array.
The security plugin is composed by `serve`'s "5d" auth step, gated
inline in `serve.ts` on four conditions: the stack mounts no
`AuthPlugin`, the `auth` tier is on, the composition is not a host
kernel, and an auth secret resolves (with the development fallback).
**Extraction done here:** `packages/core/src/stack-auth.ts`, exported
from `@objectstack/core`:
- `resolvePlatformAuthComposition({ plugins, tiers, secret })` returns
`{ composes: true, secret }` or `{ composes: false, reason }`. The
checks run in `serve`'s order.
- It comes with the tier rule `resolveStackTiers` and its
`STACK_TIER_PRESETS` / `CAPABILITY_TO_TIER`. `Serve.TIER_PRESETS` and
`Serve.CAPABILITY_TO_TIER` are now handles over these.
- Also exported: `resolveAuthSecret`, `stackSuppliesAuthPlugin` and
`isHostKernelComposition`.
`serve` asks this rule. Its gate line `if (!hasAuthPlugin &&
tierEnabled('auth'))` is kept, now reading the rule's own predicate and
tier set; the host-kernel and no-secret branches read the rule's answer.
What `serve` composes is unchanged.
**Still owed (not in this PR's surface):**
- **`--preset` and `--dev` (patch round 2, `7f961c71a9`).** The step
takes `serve`'s two flags with `serve`'s meaning, through the rules
`serve` itself now calls. `--preset`: both commands list
`Object.keys(STACK_TIER_PRESETS)` as options, and the value reaches
`resolveStackTiers`. `--dev`: `isDevelopmentBoot` (`@objectstack/core`,
now `serve`'s `isDev`) decides the secret fallback, and
`stackBootPlugins` (`cli/utils/stack-collections.ts`, now `serve`'s
`plugins` line) merges `devPlugins`. The flags move the composition
only. The step keeps the one-shot boot posture: `NODE_ENV` is untouched,
no `.env*` file loads, and the standalone stack gets no `dev` key. None
of these moves the gate or the held names.
- `AuthPlugin`, the organizations plugin and the audit plugin are still
constructed by `serve` alone. None of them holds a catalog name.
Measured: the only manifest registration of permission sets among
serve's platform plugins is `security-plugin.ts`'s.
- `@objectstack/verify`'s harness composes `AuthPlugin` and the security
plugin unconditionally. That is #22301's territory and is untouched
here.
## Premises re-measured (zone 2), on `origin/main` `e148ca9842`
1. `packages/runtime/src/standalone-stack.ts:910` hard-coded
`hydrateMetadataFromDb: true`. Every `os migrate` database command
booted through `schema-migrate.ts:492` (`new Runtime`) and `:522`
(`runtime.start()`). **Held.** `createStandaloneStack` now accepts
`hydrateMetadataFromDb: false`, and `bootSchemaStack` accepts
`hydrateMetadata: false`. Both default to on.
2. `findEnvironmentHeldSecurityCatalogNames`,
`declaredSecurityCatalogNames`, `BUILT_IN_SECURITY_CATALOG_NAMES` and
`ENVIRONMENT_HELD_SECURITY_CATALOG_TYPES` had 0 exports from
`packages/objectql/src/index.ts` and `core.ts`. **Held.** None of the
four is exported now either; the one new export is the package half of
the reading.
3. `materializeStackPlugin` does not cover it (above).
4. The refused set depends on the boot env. **Reproduced in the pin
below.** On one database and one compiled artifact, the cold boot was
refused on 3 names without `OS_AUTH_SECRET` and 4 with it (the fourth is
`member_default`, `com.objectstack.plugin-security`, stored under the
legacy plural).
5. Do a non-hydrated kernel's deletions write the same history and audit
rows as a hydrated kernel's? **Yes, for `deleteMetaItem`.** Measured in
a scratch run (not committed), with one database and two kernels of the
step's own composition: one booted without hydration over stored rows,
and one booted with hydration before the rows were written into it. Each
called `deleteMetaItem` on one permission set and one position. The
`sys_metadata_history` rows and the `sys_metadata_audit` rows were
column-for-column identical apart from ids and timestamps
(`operation_type: delete`, `source: protocol.deleteMetaItem`,
`recorded_by` / `actor` = the step, `outcome: allowed`, `code: ok`,
`lock_state: none`), and so were the receipts. What hydration does not
decide, and the step's composition does: the security plugin's mutation
projector registers in its `start()`, which a declaration boot
suppresses, so no `sys_permission_set` re-projection runs in the step.
The next boot's reconciler re-projects, as after Discard Overlay.
## The ADR-0087 marker on #22307's changeset
`.changeset/22307-cold-boot-catalog-refusal.md` (unreleased; `pre.json`
lists 0 consumed changesets):
- **The marker.** It flips from `not-required
(no-migration-prescription)` to `registered
security-catalog-environment-overlay-refused`. That is a new D3 semantic
entry
(`packages/spec/src/migrations/entries/semantic/18.security-catalog-environment-overlay-refused.ts`,
plus its generated region in `registry.ts`). Its replacement prescribes
the step before the first v18 boot.
- **Why a registration.** The gate's closed vocabulary has no other arm
for "a migration prescription naming a step". Measured with the gate's
own `findMigrationPrescription` on the base and on the rewritten body:
`null` for both. The old marker would therefore still have passed
mechanically; the flip carries out the ruling, not a gate demand.
`spec-changes.json` and the upgrade guide do not move: major-18 entries
do not project before protocol 18.
- **The upgrade shape** now says to run the step before the first v18
boot.
- **The after-upgrade bullets** point at the step, and the SQL stays as
the statement of what the step deletes.
- **"No `os` command deletes a `sys_metadata` row offline"** is
rewritten. The pre-upgrade remedies are unchanged.
## Tests
Local runs. Builds and tests went through
`scripts/pm/os-verify-lock.sh`; each step's exit code was captured
before any pipe. The branch merged `origin/main` `446c8b2a6`, which
moved `core`, `runtime` and `plugin-security`. After the merge, at
`ed5af305c8`, these all exited 0: the whole-workspace build (72/72);
typecheck of core, objectql, runtime and cli; core tests (88 / 2288);
runtime tests (345 / 4878 passed, 19 skipped); the 7 touched cli unit
files (186); and the 4 touched cli integration files (103). The numbers
below are the pre-merge runs at `5a5cb1057`.
- **Build** of `@objectstack/{objectql,spec,core,runtime,cli}` and their
closure: 59/59 tasks, exit 0.
- **Typecheck:**
- core 0, objectql 0, runtime 0;
- cli 0, including `check:test-typecheck`.
- **Package tests:**
- core 87 files / 2276 tests;
- objectql 391 / 7718;
- runtime 345 / 4878 passed, 19 skipped;
- spec `src/migrations/*` (3 files) 200 tests.
- **CLI `unit` tier:** 274 files / 4055 tests, 1 red. The red was
`test/normalized-call-sites.test.ts`: it flagged the new
`stack.requires` read in `schema-migrate.ts`. That `stack` is
`createStandaloneStack`'s result, whose `requires` the runtime already
resolves over package bodies. The read is now classified as a
`top-level` row with that reason. Re-run: 11/11.
- **CLI `integration`, for the files this diff touches:**
- the new `security-catalog-overlays.integration.test.ts`;
- `schema-migrate.one-shot-family.integration.test.ts`, where the step
is now a declared caller with a preview mode and an `--apply` mode;
- `schema-migrate.host-composition.integration.test.ts`;
- `schema-migrate.requires-providers.integration.test.ts`.
Total: 4 files / 103 tests, exit 0.
- **The step's pin** (`security-catalog-overlays.integration.test.ts`)
runs the command in-process through oclif over one SQLite database and
one compiled artifact.
- **The preview, auth off and auth on.** It lists the `permission` row,
the package-bound `position` row and the legacy-plural `positions` row.
With auth on it also lists the legacy-plural
`permissions/member_default` row, held by
`com.objectstack.plugin-security`. The controls are never listed: an
environment-wide name no package holds, an organization-scoped row and a
draft row.
- **Consistency.** In both postures, the list equals the cold-boot
refusal's `conflicts[]` over the same database and configuration. The
refusal comes from the same composition booted with hydration on.
- **The preview writes nothing and exits 1.**
- **`--apply`** deletes exactly the 4 listed rows and every control
stays. It writes 4 history tombstones and 2 ADR-0010 audit rows: one per
canonical row, actor = the step. Then the cold boot comes up, and a
second preview lists 0 and exits 0. With auth off, `member_default` is
left alone and the boot comes up.
- **Ablation**, run once and not committed: `composeAuthGatedSecurity:
true` set to `false` in the command through
`scripts/ablation-replace.mjs` (anchor 1 to 0, blob `a484cc0a4` to
`92908a4cb`).
- The pin went red, 3 of 4. The auth-on apply case listed 3 rows where
the cold boot refuses 4 (`member_default` missing), and both preview
cases lost their `securityPlugin` answer.
- Restore proven: the blob is back to `a484cc0a4` and `git diff HEAD` is
0 bytes.
- **The public door**, measured by hand with the built CLI
(`bin/run.js`, `NODE_ENV=production`) over one fixture database:
- `os serve` without `OS_AUTH_SECRET` exits 1 and refuses 3 names. With
it, `os serve` exits 1 and refuses 4: the same 3, plus `member_default`
held by `com.objectstack.plugin-security`.
- `os migrate security-catalog-overlays --json` lists exactly those 3
and those 4 rows, with `securityPlugin` `no-secret` and `composed`
respectively.
- After `--apply` with auth on (4 deleted: 2 via
`protocol.deleteMetaItem`, 2 via `sys-metadata-repository`), `os serve`
with auth on on the same database printed `Server is ready`. It was
stopped by its own `timeout`.
- **Lint**, measured over a stated narrowing:
- `pnpm exec eslint --no-inline-config --format json` over the 18
changed `.ts` files: 18 file results, 0 errors, 0 warnings, 0 ignored.
- The population is read from `eslint.config.mjs`: none of the 18 is
ignored.
- Invariance: `eslint.config.mjs` never enables type-aware linting (no
`parserOptions.project`, no typed rules, stated at its lines 326 to
328), so this diff cannot move a verdict on an untouched file.
## Patch round (head `ae87d67c8d`)
- **The merge.** `origin/main` `da989bbb24` was merged through
`scripts/pm/os-regen-merge.sh` (`16e58dc320`).
- **The protocol-18 regeneration** is its own commit (`7dc9788339`):
`spec-changes.json` +14 and `docs/protocol-upgrade-guide.md` +3,
additions only. `check:spec-changes`, `check:upgrade-guide`,
`check:migration-registry` and `check:generated` all exit 0, the last
against a freshly built spec dist.
- **Builds, typecheck and tests at `ae87d67c8d`:**
- the whole-workspace build passed (72/72), and typecheck exits 0 for
core, objectql, runtime and cli;
- core 88 / 2288, objectql 392 / 7739, runtime 345 / 4878 (+19 skipped);
- spec migrations 3 / 200, cli unit 275 / 4059, and the 4 touched cli
integration files 4 / 103.
- **Gates at `ae87d67c8d`:** 125 derived families. 124 exit 0, and
`check-empty-changeset` is red by design (the declared 22307
correction).
- `--ran`: 125 derived, 125 run, 0 NOT-MEASURED, 0 UNRUN.
- The 48 artifact-roster families all exit 0,
`check:error-code-provenance` included. The 3 PR-context ones were run
with `PR_NUMBER=22523`.
- **Changed lines after the merge:** +1801 / −65 = 1,866 across 23
files, under the 3,000 threshold.
## Patch round 2 (head `7f961c71a9`)
The contract review FAIL `6087652785` and seat order `6087686331`.
- **The spec changeset line.**
`.changeset/22371-security-catalog-overlays-step.md` names
`"@objectstack/spec": minor`. A bullet under 'New exports it is built
on' names the D3 entry `security-catalog-environment-overlay-refused`
and its upgrade-guide projection.
- **`--preset` / `--dev`, read the way `serve` reads them.**
- Three `serve` lines now call the shared rules: `isDev =
isDevelopmentBoot(flags.dev)`, `plugins = stackBootPlugins(config,
flags.dev)`, and the `--preset` options
`Object.keys(STACK_TIER_PRESETS)`. The truth table, the array identity
and the option order are unchanged, so `serve` composes what it
composed.
- The step passes `serveFlags` through `bootSchemaStack` to
`buildSchemaMigrationPlugins`. There `--dev` composes the host config's
`devPlugins` for their declarations, and the gate reads them; `--preset`
reaches `resolveStackTiers`.
- The JSON gains `serveFlags`, and the text report prints `Composed as:
os serve --preset NAME [--dev]`.
- **The pin.** With `--preset minimal` and a secret: `securityPlugin` is
`auth-tier-off`, `member_default` is not listed, and the list equals the
cold-boot `conflicts[]` of the same flags. With `--dev` and no secret:
composed, and 4 rows listed, equal to that boot's conflicts.
- Ablation: dropping the preset from `serveFlags` turned the minimal
case red (`{ composed: true }`). The restore was proven by blob.
- **Wording.** The cli.mdx 'Data migrations' row says to run with the
flags and environment the deployment boots with. So, beyond the order,
do the D3 entry's replacement and acceptance text (`registry.ts`,
`spec-changes.json` and the upgrade guide regenerated, `check:generated`
/ `check:spec-changes` / `check:upgrade-guide` /
`check:migration-registry` green) and one word-group of the 22307 note's
upgrade-shape sentence. All four surfaces give the operator one
instruction.
- **Verification at `7f961c71a9`:**
- build 72/72;
- typecheck: core and cli exit 0;
- core `stack-auth` 11/11; cli unit 9 files / 177; the step's
integration file 6/6; host-composition plus one-shot-family 97/97; spec
migrations 203/203;
- eslint over the 21 changed `.ts` files: 0 / 0.
- Full core, objectql and runtime suites are declared to CI.
- **Gates at `7f961c71a9`:**
- 125 derived: 124 exit 0, and `check-empty-changeset` is red by design.
`--ran`: 125/125, 0 NOT-MEASURED.
- The 48 artifact-roster families exit 0, the 3 PR-context ones with PR
context.
- `check-changeset-no-major` with the PR event reads `Clause-②: yes
(widening)` and passes.
- **Changed lines:** +2025 / −76 = 2,101 across 26 files, under 3,000.
## Landing step A (head `f59b85e44e`)
Seat order `6088904280`. `origin/main` `faf6348508` was merged through
`os-regen-merge.sh` as `4cbd442f11`, and the regeneration is
`f59b85e44e`.
- `main`'s `storage-scope-public-retired` entry and its
`flow-value-slot-template-dialect-refused` edit join this PR's entry in
`spec-changes.json` and the upgrade guide. `registry.ts` regenerates to
its merged bytes unchanged.
- Both sides survive, by quoted exact-name `git grep` on `origin/main`
and the head. Against `origin/main`, the PR's migration surfaces differ
by additions only, and all of them are this PR's entry.
- Green at the head: `check:migration-registry`, `check:spec-changes`,
`check:upgrade-guide`, `check:generated`, `check:authorable-surface`,
spec `src/migrations` (203), and the cli `migrate-meta-engine-guidance`
integration file. The re-derived 125 families give 124 exit 0, with
`check-empty-changeset` red by design, and the 48 roster families all
pass.
- The PR's own diff is unchanged: +2025 / −76 over the same 26 files.
- The hop is not a pure regeneration (`serve.ts`,
`normalized-call-sites.test.ts` and `registry.ts` moved on both sides),
so a fresh contract review is owed on this head.
## Gates
At `ed5af305c8`:
- **The 100 derived families** (`dispatch-gates.mjs --commands`): 99
exit 0, and 1 is red by design. `--ran` reconciliation: 100 derived, 100
run, 0 NOT-MEASURED, 0 UNRUN.
- **`check-empty-changeset.mjs --base origin/main` is red by design.**
This PR deliberately corrects a pending release note:
`.changeset/22307-cold-boot-catalog-refusal.md`, written by PR #22365
and not yet released. What changed under it: this PR adds the offline
step its upgrade shape now prescribes. The corrections are the ADR-0087
marker, the upgrade shape, the after-upgrade bullets, and the sentence
"No `os` command deletes a `sys_metadata` row offline". The gate's own
text says this class takes a confirmation on the PR, not a restore from
base. Requested here.
- **The 49 artifact-roster families:** 46 exit 0,
`check:error-code-provenance` among them. The step stamps no registered
error code, and no owner-key row is owed. The other 3 need PR context:
`check-partof-closing-keyword.mjs` with this body exits 0, and the two
that need a PR number were run after this PR opened (see the report on
the card).
## Acceptance notes
- `content/docs/deployment/cli.mdx` gains the step's rows in this PR
(`ae87d67c8d`): one in the "Data migrations" table and one in the "If
the database is in use" table, where `--apply` **refuses** a database a
live process holds. The carrier noted in round 1 is discharged here.
- A canonical row whose package item carries `_lock: 'full'` or
`'no-delete'` is refused by `deleteMetaItem`'s lock gate
(`ITEM_LOCKED`). The step reports that row `failed` with the reason and
exits 1; the SQL in the changeset remains the statement of what to
delete. Not measured on a real package: no shipped package declares a
lock on a permission set or position.
- `serve.ts` keeps a second spelling of `isDev` for port auto-shift:
`portAutoShiftAllowed = flags.dev || process.env.NODE_ENV ===
'development'`, earlier in `run()` than `isDev`. It decides the port
policy, not the composition, so it was left as is: outside this card's
surface, no carrier.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01KNKBCRDJCu5tGy3TEbvtrF)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent b3a3634 commit db9cf80
26 files changed
Lines changed: 2025 additions & 76 deletions
File tree
- .changeset
- content/docs/deployment
- docs
- packages
- cli
- src
- commands
- migrate
- utils
- test
- core/src
- objectql/src
- runtime/src
- spec
- src/migrations
- entries/semantic
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
9 | | - | |
| 9 | + | |
10 | 10 | | |
11 | 11 | | |
12 | 12 | | |
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
20 | 20 | | |
21 | | - | |
| 21 | + | |
22 | 22 | | |
23 | 23 | | |
24 | | - | |
25 | | - | |
| 24 | + | |
| 25 | + | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
30 | 30 | | |
31 | 31 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
937 | 937 | | |
938 | 938 | | |
939 | 939 | | |
| 940 | + | |
940 | 941 | | |
941 | 942 | | |
942 | 943 | | |
| |||
1114 | 1115 | | |
1115 | 1116 | | |
1116 | 1117 | | |
| 1118 | + | |
1117 | 1119 | | |
1118 | 1120 | | |
1119 | 1121 | | |
| |||
0 commit comments