Merge origin/main into claude/issue-21647-echo-never-in-memory-bucket

Claude-Session: https://claude.ai/code/session_01DiCSbmJrkzNhuEAier4VoJ
Co-authored-by: Claude <noreply@anthropic.com>

Author: claude

.changeset/21573-migrate-unmapped-columns.md

@@ -0,0 +1,16 @@
+---
+'@objectstack/cli': minor
+---
+
+feat(cli): `os migrate unmapped-columns --object NAME` reads the values of a retired field's columns, keyed by record id, for a conversion before `os migrate apply --allow-destructive` drops them (#21573)
+
+Clause-②: yes (widening)
+
+- **What it reads.** The columns `os migrate plan` reports as `unmapped_column` for one object's table: a column that is still in the table and that no metadata declares, typically one a retired field left behind. The column set is the plan's own findings, from the same differ on the same read-only boot, so the command never reads a column the plan does not report. Each record is emitted as `{ id, values }`. `--json` prints one document, `{ database, object, table, columns, count, records, duration }`. The text face lists the columns and each record's values.
+- **Why it exists.** A read or a write through the engine now serves an object's declared fields only, and naming an undeclared column is refused. An app that moves a retired field's values into the field that replaced it reads them once with this command, writes them with its own script, and then drops the columns with `os migrate apply --allow-destructive`. That is the route the read and write narrowing in `@objectstack/objectql` names for this case.
+- **Operator-only and read-only.** It runs under the database credentials you pass (`--database-url`, else `OS_DATABASE_URL`, else the project database), and it reads every organization's rows. No REST route, API flag or per-request option serves these values, and the runtime doors are unchanged. It boots the way `os migrate plan` does: no schema DDL, no seed data, and no database file created.
+- **Values as stored.** An unmapped column has no declared type, so each value is emitted as the database client returns it, with no field-type decoding; a PostgreSQL `timestamp` arrives as a date and is emitted as its ISO 8601 text. A value JSON cannot carry as stored (binary bytes, a `bigint`, or a non-finite number) is refused in both faces with exit 1, naming the column and the record id, and no record is emitted: read that column with the database's own client. No column the platform creates for a field type answers with one of these, on SQLite or on PostgreSQL.
+- **Answers.** An object with no unmapped column, or with no table yet: empty work, exit 0. No SQL driver: `os migrate plan`'s own `no_sql_driver` answer, exit 0. An undeclared object name: `OBJECT_NOT_FOUND`, exit 1. An object the plan does not diff (federated, or bound to another datasource): refused, exit 1. A read that cannot be complete, such as one stopped by `--max-records`: refused, exit 1, and no partial set is emitted. A value JSON cannot carry as stored: refused, exit 1, as above.
+- `MigrateUnmappedColumnsCommand` is exported from `@objectstack/cli` beside the other `os migrate` commands.
+
+Nothing that ran before changes. This is a new command.

content/docs/data-modeling/queries.mdx

@@ -290,7 +290,11 @@ system columns (`id`, `created_at`, `updated_at`, and the tenant, owner and audi
 the registry adds). A column no metadata declares is never returned — for example one a
 retired field left in the table until `os migrate apply --allow-destructive` drops it — and
 naming it in `fields` on the data API is refused with `400 INVALID_FIELD`. To read such a column's values
-for a one-time conversion, run the conversion before the field is retired.
+for a one-time conversion, run the conversion before the field is retired, or, once it is
+retired, read them with the operator-only
+[`os migrate unmapped-columns`](/docs/deployment/cli#os-migrate-unmapped-columns) and convert
+them before `os migrate apply --allow-destructive` drops the column. No runtime door serves
+them.
 
 ### Nested / Related Fields
 

content/docs/deployment/cli.mdx

@@ -519,7 +519,7 @@ as the project's **default** datasource it is refused at boot the same way.
 **What it boots:**
 - Reads the artifact's `manifest`, `objects`, `views`, `flows`, …
 - Auto-registers the platform services declared in `requires: [...]` (e.g. `ai`, `automation`, `analytics`, `auth`, `ui`). Declaring a **service** capability (`automation`, `analytics`, `ai`, `audit`, …) is a *requirement*: if its provider package isn't installed, boot **fails fast** with a clear error instead of silently starting without a capability you asked for. (`auth` and `ui` are tier-gated with their own opt-in rules — `auth`'s secret-gated skip is described below.)
-- Auto-detects the driver from the database URL scheme (`libsql://`/`https://*.turso.*` → Turso — via the optional `@objectstack/driver-turso` package, and a loud failure with the install command when it is missing rather than a fallback to sqlite —, `postgres[ql]://`/`pg://` → pg, `mongodb[+srv]://` → MongoDB, otherwise sqlite; `:memory:` is SQLite's own in-memory database). `memory://` and `mingo://` are refused — see the callout below.
+- Auto-detects the driver from the database URL scheme (`libsql://`/`https://*.turso.*` → Turso — via the optional `@objectstack/driver-turso` package, and a loud failure with the install command when it is missing rather than a fallback to sqlite —, `postgres[ql]://`/`pg://` → pg, `mongodb[+srv]://` → MongoDB, otherwise sqlite; `:memory:` is SQLite's own in-memory database). `memory://` and `mingo://` are refused — see the callout above.
 - Runs standalone boot mode with one active environment.
 
 **Authentication:**
@@ -853,6 +853,7 @@ diverges from the live schema, and the physical column wins at write time.
 | `os migrate plan` | Dry-run: show how the database has drifted from metadata, categorised safe / needs-confirm / destructive (no changes applied) |
 | `os migrate apply` | Reconcile the database to metadata. Applies loosening changes; destructive ones require `--allow-destructive` |
 | `os migrate multi-value-columns` | Migrate a stale `varchar`/`text` column to `json` where the field declares `multiple: true` — one of three drift ops `apply` never reconciles for you. Dry run by default; `--apply` runs the statement the finding prints |
+| `os migrate unmapped-columns` | Read the values of the columns `plan` reports as `unmapped_column` for one object, keyed by record id — the conversion route for a retired field's values before `apply --allow-destructive` drops its columns. Read-only, and operator-only: no runtime door serves these values |
 
 ```bash
 os migrate plan                              # Preview drift (no changes)
@@ -861,6 +862,7 @@ os migrate apply --yes                       # Skip the prompt (CI / scripts)
 os migrate apply --allow-destructive --yes   # Also drop orphaned columns, tighten NOT NULL, narrow types
 os migrate apply --force                     # Migrate even though another process is using the database
 os migrate plan --json                       # Machine-readable output
+os migrate unmapped-columns --object contact --json   # A retired field's stored values, keyed by record id (read-only)
 ```
 
 #### Nothing is written before you confirm
@@ -1048,6 +1050,54 @@ hook or an integration already copied into some *other* single-value column is
 not something it looks for, and it is deliberately not something it will grow
 into: that repair is specific to what your automations did with the value.
 
+#### `os migrate unmapped-columns`
+
+Retiring a field leaves its column in the table: the additive sync never drops
+one, and `os migrate plan` reports it as `unmapped_column` until
+`os migrate apply --allow-destructive` does. In between, the values are still
+stored, and no runtime door serves them: a read or a write through the data API
+returns the object's declared fields only, and naming the column in `fields` is
+refused. When the values have to move into the field that replaced it, this is
+the read:
+
+```bash
+os migrate unmapped-columns --object contact                    # The columns, and every record's values
+os migrate unmapped-columns --object contact --json > out.json  # The same, for a conversion script
+os migrate unmapped-columns --object contact --max-records 1000000 --json
+os migrate unmapped-columns --object contact --database-url postgres://…
+```
+
+The conversion route is three steps: read the values with this command, write
+them into the declared fields with your own script, then run
+`os migrate apply --allow-destructive` to drop the columns.
+
+- **One column set.** It reads exactly the columns `os migrate plan` reports as
+  `unmapped_column` for that object's table: the same differ, the same boot.
+  Columns the differ never reports are never read either, such as the driver's
+  own `id`, `created_at` and `updated_at`.
+- **Operator-only and read-only.** It runs under the database credentials you
+  pass, and covers every organization's rows. There is no REST route or API
+  flag behind it. It boots the way `plan` does, so it writes nothing, and it
+  drops nothing.
+- **Values as stored.** An unmapped column has no declared type, so each value is
+  emitted as the database client returns it, with no field-type decoding: a
+  retired `json` field on SQLite reads as its stored text, a retired `boolean`
+  as `0` or `1`, and a retired `datetime` on PostgreSQL arrives as a date and is
+  emitted as its ISO 8601 text. A value JSON cannot carry as stored (binary
+  bytes, a `bigint`, or a non-finite number) is refused with exit 1, naming the
+  column and the record id, and no record is emitted; read that column with
+  the database's own client.
+- **Empty work, exit 0**, for an object with no unmapped column, or with no table
+  in this database yet. `--json` prints one document:
+  `{ object, table, columns, count, records: [{ id, values }] }`.
+- **Refused, exit 1**: an object name the deployment does not declare
+  (`OBJECT_NOT_FOUND`); an object `plan` does not diff (federated, or bound to
+  another datasource), whose empty answer would be unmeasured; a read that
+  cannot be complete, such as one stopped by `--max-records`; and a value JSON
+  cannot carry as stored, described above. A partial set is
+  never emitted, because a conversion over part of a table, followed by the
+  drop, loses the rest.
+
 #### Data migrations
 
 The commands above reconcile **schema**. A *data* migration rewrites rows, and

packages/cli/src/commands/migrate/data-commands.absent-database.integration.test.ts

@@ -58,6 +58,10 @@
  * (`--json` and human), a booted database holding one row of work for each
  * (the control: the door READS the table and reports the row), and the four
  * doors that already exited 0 on the absent database, which still do.
+ *
+ * [#21573] `os migrate unmapped-columns` joined the roster later, born with
+ * the same answer: it asks `tableAbsent` before the differ. Its control row is
+ * a retired field's column on `os21529_contact`, holding a value.
  */
 
 import { describe, it, expect, beforeAll, afterAll } from 'vitest';
@@ -418,6 +422,9 @@ await k('sys_secret').insert({
 await k('sys_file').insert({
   id: 'file_21552', key: 'attachments/os21552.txt', name: 'os21552.txt', size: 12, scope: 'attachments', status: 'committed',
 });
+// A retired field's column: in the table, in no metadata, holding a value.
+await k.raw('ALTER TABLE os21529_contact ADD COLUMN legacy_note text');
+await k('os21529_contact').insert({ id: 'con_21573', name: 'Ann', legacy_note: 'kept-21573' });
 await raw.disconnect();
 process.stderr.write('[fixture] seeded\\n');
 process.exit(0);
@@ -513,6 +520,19 @@ const DOORS: readonly Door[] = [
     tables: ['sys_file', 'sys_attachment'],
     work: (doc) => expect(doc).toMatchObject({ filesScanned: 1, stranded: 1 }),
   },
+  {
+    // Born after #21552 with the family's answer: it asks `tableAbsent` before
+    // the differ, so an absent table is empty work and is never read.
+    name: 'migrate unmapped-columns',
+    argv: ['migrate', 'unmapped-columns', '--object', 'os21529_contact'],
+    empty: (doc) => expect(doc).toMatchObject({ object: 'os21529_contact', columns: [], count: 0, records: [] }),
+    humanEmpty: /No unmapped column on os21529_contact/,
+    tables: ['os21529_contact'],
+    work: (doc) => {
+      expect(doc.columns).toEqual([{ column: 'legacy_note', actual: 'text' }]);
+      expect(doc.records).toContainEqual({ id: 'con_21573', values: { legacy_note: 'kept-21573' } });
+    },
+  },
 ];
 
 /** The four doors that already answered the absent database with exit 0: they must still. */