Skip to content

Commit 1f89ba0

Browse files
fix(driver-turso): remote syncSchemasBatch registers read coercion and runs the canonical backfill (#19863)
Fixes #19844 Clause-②: no ## What was wrong `ObjectQLPlugin.syncRegisteredSchemas` takes its batch branch when a driver declares `supports.batchSchemaSync` and implements `syncSchemasBatch`, and `TursoDriver` does both. On the remote transport, `syncSchemasBatch` returned straight after its DDL. The two sibling remote doors (`syncSchema`, `initObjects`) go on to call `registerRemoteFieldMetadata` and then the canonical temporal backfill. So the door a remote-Turso boot actually takes was the one door that skipped all three halves: read-coercion registration, the managed-object record, and the backfill. ## Boot measurement (taken before any edit) One boot of an `ObjectKernel` with `ObjectQLPlugin`, a remote `TursoDriver` registered as the `driver.turso` service over the `libsql` SQLite double (`libsql-sqlite-stub.testkit.ts`), and an app object `w` with `flag: boolean`, `meta: json`, `at: datetime`. It ran as a scratch test and is not committed (see Acceptance notes). Before = `origin/main` 8cbc3c0; after = this branch at d8940d7. | reading | before | after | |:--|:--|:--| | schema doors called during the boot | `syncSchemasBatch` twice (phase 1 and phase 3); no `syncSchema`, `initObjects`, `registerExternalObject` or `registerObjectMetadata` call | `syncSchemasBatch` twice, each followed by `registerExternalObject` for all six objects in the batch | | `driver.findOne` flag / meta | `1` (number) / the string `{"k":1}` | `true` (boolean) / `{"k":1}` (object) | | engine `findOne` flag / meta | `1` (number) / a string | `true` (boolean) / an object | | `paginationTieBreaker('w')` | `null` | `id` | | `booleanFields.w` / `jsonFields.w` / `datetimeFields.w` | empty / empty / empty | `flag` / `meta` / `created_at, updated_at, at` | | `canonicalDatetimeFields.w` | empty | `created_at, updated_at, at` | So the card stands at p1: nothing else on the boot path populated the registries. ## The fix The landing site is the one the card named, the `isRemote` arm of `TursoDriver.syncSchemasBatch` in `packages/drivers/driver-turso/src/turso-driver.ts`. The producer is this driver, so no consumer changes. - A new private helper, `completeRemoteSchemaSync(objects)`, registers each object and then runs `backfillRemoteCanonicalTemporalQuietly()` once for the call. The registration is `registerRemoteFieldMetadata`: the `remoteManagedObjects` record, plus the coercion, tenant and autonumber registries `registerExternalObject` fills. An empty list is a no-op. - All three remote doors call the helper after their DDL resolves. `syncSchema` passes one object, keyed by its `object` argument. `initObjects` passes its objects. `syncSchemasBatch` now passes each entry as `{ ...schema, name: object }`, the same strict keying `syncSchema` uses. - Order: the DDL is awaited first, so a DDL failure rejects before anything is registered. Registration comes before the backfill because the backfill reads it to learn which columns are temporal. - The backfill runs once per batch, not once per object. It probes every unmarked column in one round-trip, so a steady-state boot costs one probe per sync call. - One docblock in the deferred-DDL refusal section said the latter two doors "also run" the backfill. It now says the batch door runs it too. Not touched: `packages/objectql/src/plugin.ts`, which was read only. `detectManagedDrift` in remote mode belongs to #19845. It is not in this diff, and #19845 remains open. ## Tests The new file `packages/drivers/driver-turso/src/turso-remote-batch-door-registration.test.ts` has 12 cases: - A `describe.each` over the three remote doors, with `syncSchemasBatch` (the boot door) as the case under test and `syncSchema` and `initObjects` as controls. Each syncs the reproduction object `{ fields: { flag: boolean, meta: json } }`, creates a row and calls `findOne`. Expected: `flag === true`, `meta` deep-equals `{ k: 1 }`, and the raw row on disk is still `{ flag: 1, meta: '{"k":1}' }`, which proves the test is not vacuous. `paginationTieBreaker('w')` goes from `null` to `'id'`. - Write side, on each of the three doors (patch round): a `datetime` written as `2025-07-28T08:00:00+08:00` reaches disk as `2025-07-28T00:00:00.000Z`. - The batch door keys by `object`, never by a `schema.name` that differs from it. - The batch door over a pre-existing legacy row: `backfillRemoteCanonicalTemporal` is called exactly once for a two-object batch. The naive `2025-07-28 00:00:00` is rewritten on disk to `2025-07-28T00:00:00.000Z`, and both columns are marked canonical. - A failing DDL batch rejects with the injected error. No table is created, and there is no tie-breaker, no boolean registry and no backfill call. ### Ablation (run once, after the fix was committed) Mutation: delete only the batch door's `completeRemoteSchemaSync(...)` call, using `node scripts/ablation-replace.mjs`. The anchor went from 1 hit to 0, the blob changed from b33d398 to 5f1d1c909d7d, and an on-disk grep count read 0. Command for both runs: `pnpm --filter @objectstack/driver-turso exec vitest run --maxWorkers=2 src/turso-remote-batch-door-registration.test.ts`. - Mutant run (re-run on `de31187a96`): exit 1, `Tests 5 failed | 7 passed (12)`. The new write pin reds on the batch door only (`expected [ { at: '2025-07-28T08:00:00+08:00' } ] to deeply equal [ { at: '2025-07-28T00:00:00.000Z' } ]`). The other four failures are the batch-door cases from the first run: `expected 1 to be true`, `expected null to be 'id'`, `expected undefined to deeply equal [ 'flag' ]`, and `expected "backfillRemoteCanonicalTemporal" to be called 1 times, but got 0 times`. The `syncSchema` and `initObjects` controls (including their write pins) and the DDL-failure case stayed green. - Restore: the blob matches HEAD (b33d398) and `git diff HEAD` is empty; the restored file is green in the full-suite run below. - No `dist/` is involved: the suite imports `./turso-driver.js` from source. ## Verification (HEAD d8940d7) **Re-run on `de31187a96` after the patch round:** `pnpm --filter @objectstack/driver-turso test` exited 0 (57 files, 1311 tests), `typecheck` exited 0, `node scripts/check-issue-citations.mjs` exited 0, `node scripts/check-changeset-no-major.mjs --base origin/main` exited 0, and `pnpm check:driver-conformance` exited 0. The `--commands` derivation was byte-identical (61 commands), and `--ran` read 59 run and 2 NOT-MEASURED, as below. - `pnpm --filter @objectstack/driver-turso test`: exit 0, 57 files and 1308 tests passed. - `pnpm --filter @objectstack/driver-turso typecheck`: exit 0. The package's tsconfig includes `src/**/*`, so the tests are type-checked too. - `node scripts/pm/dispatch-gates.mjs --commands` (no paths) derived 61 commands. All 61 ran, each exit code captured before any pipe. The `--ran` verdict exited 0: `61 derived famil(ies) accounted for — 59 run, 2 NOT-MEASURED`. - NOT MEASURED: `pnpm check:dual-build-cjs-loads` and `pnpm check:type-check-debt` both exited 3 (PREREQUISITE NOT MET), because each needs the whole-workspace build. CI runs both over the full build. Declared narrowing for the first: `require` of the rebuilt `packages/drivers/driver-turso/dist/index.js` loads (14 exports, `TursoDriver` a function), exit 0. For the second: driver-turso has no DEBT or TEST_DEBT entry, and its own `tsc --noEmit` is clean. - Gates the dispatch named, all green in that run: `pnpm check:driver-conformance`, `pnpm check:object-def-param-keys`, `pnpm check:issue-citations` and `pnpm check:nul-bytes`, each exit 0. - `node scripts/check-issue-citations.mjs`, the live diff-scoped verdict: exit 0, with 3 citations judged and all 3 resolving. - Lint, narrowed: `eslint --no-inline-config --format json` over the two changed `.ts` files reported 2 files, 0 errors, 0 warnings. The changeset `.md` is outside eslint's configured population ("no matching configuration"). `eslint.config.mjs` enables no type-aware linting (no `parserOptions.project`), so this diff cannot change the verdict on any untouched file. The full `pnpm lint` is CI's. ## Changeset `.changeset/19844-turso-remote-boot-read-coercion.md`, `@objectstack/driver-turso: patch`. It was rewritten in the patch round after the first contract review (FAIL on prose, comment 5794937369). Every claim was re-measured on the SQLite double, including what the pre-fix door wrote and which of those cells the backfill does and does not converge. ## Acceptance notes - The boot measurement is not kept as a test. `@objectstack/objectql` is not a dependency of `@objectstack/driver-turso`, and adding one (with its lockfile change) for a single test is outside this card's file surface. The door-level cases make the same call the boot's batch branch makes. - The header table in `turso-remote-deferred-ddl.test.ts` records the `syncSchemasBatch` row of prediction (b) as "REFUTED, no row write on this door". It was measured before that refusal existed, so it stays historically true; the patch round adds a one-clause footnote saying the door now runs the backfill too. - A behaviour change for review: an ordinary remote boot now runs the canonical temporal backfill, which the batch door never ran before. On a deployment that holds legacy datetime or time text, the first boot after upgrading rewrites those cells into the canonical spelling of the same value, as `syncSchema` and `initObjects` already did. The changeset says so, and it names the two kinds of cells the pre-fix door wrote unconverted that no remote backfill converges (a `date` stored as a full timestamp; a scalar `json` stored unencoded). --- _Generated by [Claude Code](https://claude.ai/code/session_01TEhopqrWQYBycZzyJHpAZr)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent b940f32 commit 1f89ba0

4 files changed

Lines changed: 270 additions & 25 deletions

File tree

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
"@objectstack/driver-turso": patch
3+
---
4+
5+
A remote Turso deployment now reads, writes and filters the objects it synced at boot by their declared field types. "Remote" means a `libsql://`, `https://`, `http://`, `wss://` or `ws://` URL with no `syncUrl`, or an explicit `mode: 'remote'` (#19844).
6+
7+
The engine's boot schema sync (`ObjectQLPlugin`) reaches this driver through `syncSchemasBatch`, because the driver declares `supports.batchSchemaSync`. On the remote transport that method ran the DDL and stopped. It skipped the field-type registration that its sibling doors, `syncSchema` and `initObjects`, run afterwards. For every object a remote app synced at boot, that meant:
8+
9+
- **Reads came back as stored.** A declared `boolean` read back as `1`/`0`, and a `json` field as its stored text. A `datetime`, `time` or `date` field read back exactly as stored, for example as an offset-bearing string or epoch text rather than the canonical `…Z` form. The `created_at` / `updated_at` audit columns were a partial exception: a zone-naive cell shaped `YYYY-MM-DD HH:MM:SS` or `YYYY-MM-DDTHH:MM:SS`, optionally with a fractional second, was read as UTC and read back canonical (the column default writes the first shape); any other cell read back as stored, including one ending in `Z` or in an offset such as `+08:00` (so a canonical cell stays canonical), an epoch number or its text, and anything that does not parse as a date. This held for records read through the driver and through the engine's `find` / `findOne`, including the rows an `afterFind` hook receives. A CEL expression or an in-memory `$ne: true` filter evaluated over such a record, such as `field != true`, was therefore true even for a stored `true`. Write-side hook contexts did see `true`/`false`, because the engine converts declared booleans there: the `afterInsert` / `afterUpdate` results and the `previous` record on update and delete hooks.
10+
- **Writes were not converted either.** A `datetime` or `time` value in any spelling other than the canonical one (with an offset, a zone-naive wall clock, an epoch number) was stored as sent. A `Date` given to a `datetime` was the exception: it was stored in canonical form. A `date` given as a `Date` or a full timestamp was stored as a full timestamp. An object or array in a `json` field was stored as it would have been anyway. A scalar `json` value (a string, number or boolean) was stored without its JSON encoding.
11+
- **Filters compared text as spelled.** A filter on a `datetime` or `time` field compared the stored text with the comparand exactly as the caller wrote it, converting neither side. Rows whose cell or comparand used another spelling of the same value were missed or matched wrongly. For example, a bare-day upper bound `$lte: '2025-07-28'` left out that day's rows stored as ISO text.
12+
- **Paging was not deterministic.** A paged read with no `orderBy` got no `id` tie-breaker, so walking the pages could serve one row twice and skip another. The driver logged `Paged read of '…' is NOT deterministic`.
13+
14+
`syncSchemasBatch` now finishes the way the other two doors do. It registers each object's field types, keyed by the `object` name it was given, and then runs the canonical temporal backfill once for the whole batch. A DDL failure still rejects before anything is registered. Reads, writes and filters on those objects now convert exactly as they do through `syncSchema` and `initObjects`.
15+
16+
What happens on disk at the first boot after upgrading: the one write this change adds is that backfill, which the `syncSchema` and `initObjects` doors already ran. It rewrites `datetime` and `time` cells stored in a non-canonical spelling, including any this door wrote unconverted, into the canonical spelling of the same value. It leaves alone a cell it cannot safely read as a time. Nothing else on disk is touched, so two kinds of cells this door wrote unconverted stay as they are:
17+
18+
- A `date` stored as a full timestamp reads back as its calendar day, but an equality filter on that day does not match it.
19+
- A scalar `json` value stored without its encoding reads back as whatever its text parses to. A stored `true` reads back as `1`, and a numeric-looking string reads back as a number.
20+
21+
Local and embedded-replica deployments are unaffected.

‎packages/drivers/driver-turso/src/turso-driver.ts‎

Lines changed: 59 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -359,8 +359,9 @@ function refuseRemoteTransaction(door: string, detail: string): never {
359359
* only when the method is absent) never fired — while every remote schema door
360360
* (`syncSchemasBatch`, the engine's boot sync; `syncSchema` / `initObjects`)
361361
* routes through `RemoteTransport`, which performs the DDL immediately, and the
362-
* latter two also run the #5770 canonical temporal backfill, which rewrites
363-
* stored rows. Measured (`turso-remote-deferred-ddl.test.ts`): the deferral was
362+
* latter two also ran the #5770 canonical temporal backfill, which rewrites
363+
* stored rows (the batch door runs it too since #19844). Measured
364+
* (`turso-remote-deferred-ddl.test.ts`, before this refusal existed): the deferral was
364365
* accepted, CREATE/ALTER ran on every door, the backfill rewrote rows on two of
365366
* them, and preview and flush both answered `[]` — a dry run that changed the
366367
* database and then reported no pending work.
@@ -1698,9 +1699,10 @@ export class TursoDriver extends SqlDriver {
16981699
*
16991700
* It also records the object as one whose table this driver created, which is
17001701
* the whole input to {@link paginationTieBreaker} in remote mode. That goes
1701-
* FIRST and outside the `try`: both callers reach here only after the DDL has
1702-
* already succeeded, so the table exists with its `id` primary key whether or
1703-
* not the best-effort coercion registration below does.
1702+
* FIRST and outside the `try`: its only caller, {@link completeRemoteSchemaSync},
1703+
* runs only after the DDL has already succeeded, so the table exists with its
1704+
* `id` primary key whether or not the best-effort coercion registration below
1705+
* does.
17041706
*/
17051707
private registerRemoteFieldMetadata(obj: { name: string; fields?: Record<string, any>; tenancy?: any }): void {
17061708
this.remoteManagedObjects.add(obj.name);
@@ -1711,6 +1713,39 @@ export class TursoDriver extends SqlDriver {
17111713
}
17121714
}
17131715

1716+
/**
1717+
* The post-DDL half every REMOTE schema door owes, in its one order: register
1718+
* each synced object's field metadata, then run the canonical temporal
1719+
* backfill ONCE for the whole call.
1720+
*
1721+
* All three remote doors (`syncSchema`, `initObjects`, `syncSchemasBatch`)
1722+
* send their DDL through `RemoteTransport` and so never reach
1723+
* `SqlDriver.initObjects`, which is what fills the read-coercion registries
1724+
* and runs the Knex backfill on the local faces. Each door has to finish the
1725+
* job itself, and they drifted apart once: `syncSchemasBatch` — the door
1726+
* `ObjectQLPlugin`'s boot sync takes whenever `supports.batchSchemaSync`
1727+
* holds, so every remote-Turso boot — returned straight after its DDL. A
1728+
* booted remote app then read a boolean back as `1` and JSON as a string, got
1729+
* no `id` tie-breaker on a paged read, and never converged its temporal
1730+
* columns (#19844). One helper called by all three is what keeps them from
1731+
* drifting again.
1732+
*
1733+
* Callers reach here only after their DDL resolved, so a DDL failure throws
1734+
* before anything is registered and no object is recorded as a table this
1735+
* driver created unless it exists. Registration precedes the backfill because
1736+
* the backfill reads it to learn which columns are temporal. The backfill
1737+
* probes every column it finds in one round-trip, so calling it once per call
1738+
* rather than once per object is what keeps a boot's steady state at a single
1739+
* round-trip.
1740+
*/
1741+
private async completeRemoteSchemaSync(
1742+
objects: Array<{ name: string; fields?: Record<string, any>; tenancy?: any }>,
1743+
): Promise<void> {
1744+
if (objects.length === 0) return;
1745+
for (const obj of objects) this.registerRemoteFieldMetadata(obj);
1746+
await this.backfillRemoteCanonicalTemporalQuietly();
1747+
}
1748+
17141749
/**
17151750
* Converge this driver's REMOTE `Field.datetime` / `Field.time` columns on the
17161751
* canonical storage form and mark the ones that are PROVED converged, so their
@@ -2014,14 +2049,10 @@ export class TursoDriver extends SqlDriver {
20142049
this.assertRemoteTransactionUnsupported(options, 'syncSchema');
20152050
if (this.isRemote) {
20162051
await this.remoteTransport!.syncSchema(object, schema);
2017-
// See initObjects(): populate the read-coercion registries for remote mode.
2018-
// Key strictly by `object` (what find()/formatOutput look up) — never let a
2052+
// Registration + canonical backfill, see completeRemoteSchemaSync(). Key
2053+
// strictly by `object` (what find()/formatOutput look up) — never let a
20192054
// stray `schema.name` shadow it.
2020-
this.registerRemoteFieldMetadata({ ...(schema as Record<string, any>), name: object });
2021-
// #5770: the remote twin of the `backfillCanonicalDatetimes` call
2022-
// `SqlDriver.initObjects` makes at exactly this point. Must run AFTER the
2023-
// registration above — that is what tells it which columns are temporal.
2024-
await this.backfillRemoteCanonicalTemporalQuietly();
2055+
await this.completeRemoteSchemaSync([{ ...(schema as Record<string, any>), name: object }]);
20252056
return;
20262057
}
20272058
return super.syncSchema(object, schema, options);
@@ -2064,17 +2095,11 @@ export class TursoDriver extends SqlDriver {
20642095
objects.map((obj) => ({ object: obj.name, schema: obj })),
20652096
);
20662097
// Remote DDL bypasses SqlDriver.initObjects, which is what normally
2067-
// populates the boolean/json/date/numeric read-coercion registries.
2068-
// Register the field-type metadata explicitly (no DDL) so remote reads
2069-
// run the same formatOutput() coercion as local/replica mode — otherwise
2070-
// a boolean reads back as raw 0/1, JSON as a string, dates as raw text.
2098+
// populates the boolean/json/date/numeric read-coercion registries and
2099+
// runs the canonical temporal backfill. Without the registration a
2100+
// boolean reads back as raw 0/1, JSON as a string, dates as raw text.
20712101
// (Root cause of the 2026-07-06 case_escalation `1 != true` incident.)
2072-
for (const obj of objects) this.registerRemoteFieldMetadata(obj);
2073-
// #5770: the remote twin of the `backfillCanonicalDatetimes` /
2074-
// `backfillCanonicalTimes` calls `SqlDriver.initObjects` makes per table.
2075-
// One batched probe covers every column synced here, so the steady state
2076-
// (nothing to converge) costs a single round-trip for the whole boot.
2077-
await this.backfillRemoteCanonicalTemporalQuietly();
2102+
await this.completeRemoteSchemaSync(objects);
20782103
return;
20792104
}
20802105
return super.initObjects(objects);
@@ -2084,14 +2109,24 @@ export class TursoDriver extends SqlDriver {
20842109
* Batch-synchronize multiple schemas in a single round-trip.
20852110
*
20862111
* In remote mode, delegates to `RemoteTransport.syncSchemasBatch()` which
2087-
* uses `client.batch()` to submit all DDL as one network call.
2112+
* uses `client.batch()` to submit all DDL as one network call, then finishes
2113+
* exactly as the other two remote doors do (see
2114+
* {@link completeRemoteSchemaSync}). This is the door `ObjectQLPlugin`'s boot
2115+
* sync takes on this driver, so it is the one that decides what a booted
2116+
* remote app reads back.
20882117
* In local/replica mode, falls back to sequential `syncSchema()` calls
20892118
* (Knex + better-sqlite3 is already local, so batching has no benefit).
20902119
*/
20912120
async syncSchemasBatch(schemas: Array<{ object: string; schema: unknown }>, options?: DriverOptions): Promise<void> {
20922121
this.assertRemoteTransactionUnsupported(options, 'syncSchemasBatch');
20932122
if (this.isRemote) {
2094-
return this.remoteTransport!.syncSchemasBatch(schemas);
2123+
await this.remoteTransport!.syncSchemasBatch(schemas);
2124+
// Key strictly by `object`, as syncSchema() does: it is the name the
2125+
// engine hands every later read and write for this table.
2126+
await this.completeRemoteSchemaSync(
2127+
schemas.map(({ object, schema }) => ({ ...(schema as Record<string, any>), name: object })),
2128+
);
2129+
return;
20952130
}
20962131
// Local/replica fallback: sequential sync (already fast with local SQLite)
20972132
for (const { object, schema } of schemas) {

0 commit comments

Comments
 (0)