Skip to content

Commit 40626bd

Browse files
huangyiireneclaude
andauthored
fix(driver-sql): stop the canonical temporal backfill writing SQLite julian-day misreadings to disk (#19688)
Part of #6009 Clause-②: no The canonical temporal backfill no longer writes SQLite's **julian-day reading** of a bare-numeric cell over the bytes on disk. The read path is untouched, deliberately — see the fence below. ## The defect `sqliteCanonicalDatetimeSql`'s `else` arm is `coalesce(strftime('%Y-%m-%dT%H:%M:%fZ', col), col)`, and its documented contract is that a value SQLite cannot parse falls through the `coalesce` unchanged. SQLite's time-value grammar has one limb that defeats that reasoning: a **bare number** is a julian day (`DDDD.DDDD`, the last of its documented formats), and `now` is the wall clock. For those two shapes `strftime` does not return NULL — it returns a confident wrong answer — so `coalesce` never fires. Reproduced first, before any code was written, on better-sqlite3 13.0.3 / SQLite 3.53.4 against a TEXT-affinity column. The card's own table, reproduced exactly, plus the two shapes that reach the same limb: ```text '12' -> -4713-12-06T12:00:00.000Z '2026' -> -4707-06-11T12:00:00.000Z '0' -> -4713-11-24T12:00:00.000Z '86400' -> -4476-06-15T12:00:00.000Z '1999' -> -4707-05-15T12:00:00.000Z '1e5' -> -4439-09-09T12:00:00.000Z '2440587.5' -> 1970-01-01T00:00:00.000Z 'now' -> the wall clock, at boot '2026-08-06' -> 2026-08-06T00:00:00.000Z (correct — control) '2026-08-06T00:00:00.000Z' -> itself (fixed point — control) '1753660800000.0' -> itself (past the julian range, strftime NULL — control) ``` On a read that is a temporary misreading and the disk is untouched. `backfillCanonicalDatetimes` uses the same expression as its `SET` value, so on that side it is a **write**, and the original value is unrecoverable afterwards. ## What landed - **`sqliteNonTemporalTextSql`** — the structural **complement** of the parseable spellings, not a heuristic. Every other format SQLite's date parser accepts goes through `parseYyyyMmDd` (which needs a `-`) or `parseHhMmSs` (which needs a `:`), so a cell the parser answers for while containing **neither** character reached it through the julian-day limb or the `now` limb — there is no third way in. The predicate never inspects the magnitude of the number and never decides what the cell *means*; it decides only that the migration must not rewrite it. It probes with `julianday()` rather than `strftime()` because the two agree on NULL-ness by construction, which makes one spelling serve both the datetime expression and the time one. - **Rows it matches are withheld from the `UPDATE`** in `backfillCanonicalDatetimes` and `backfillCanonicalTimes`, and left exactly as they are — which is what the backfill's own contract already promises for every other uninterpretable value. - **A withheld row also blocks the canonical mark, and that half is what keeps every query answer identical.** Skipping the row costs nothing while the read-side repair still applies to it: it keeps reading as the same instant it always did. Marking the column clean is what would move an answer — `needsLegacyDatetimeRepair` would drop the repair and the raw `'2026'` would be compared as TEXT. Unlike the `coalesce` fixpoints, these rows are **not** invariant under the repair, so they cannot ride through the mark the way genuinely unparseable junk does. The column stays un-marked, reads keep their (unindexed) repair, and one `warn` names the count, the reason and the remedy. That is the degradation posture this module already has for a failed `UPDATE`, reused rather than invented. - **`previewDatetimeConvergence` / `previewTimeConvergence` inherit the same exclusion**, so `os migrate plan` still promises exactly the rows `apply` rewrites — the pair's stated construction. - **A `patch` changeset**, measured rather than assumed: `@objectstack/driver-sql` is not private, its `files[]` ships `dist`, and the new symbol reaches it (10 hits in `dist/index.js` against a positive control of 12 for `sqliteCanonicalDatetimeSql`). ### The fence, honoured > 倾向**不要**在共享表达式里加启发式 —— 维护者 2026-08-03 在 cloud#1005 已就近似的提案裁过否决(公共契约 + 每次读都跑 + 会误读合法的数字字符串列)。 The read expression is byte-for-byte unchanged, and the suite **pins** that it still misreads, so a later "improvement" to it fails this suite rather than slipping through it. Everything added here runs once, as a migration, only on a column the metadata declares `Field.datetime` / `Field.time`, and it only ever declines to write. No new gate is added or recommended: this is a guard inside an existing migration path. ### On `'2440587.5'` The card notes SQLite's answer there is genuinely the julian epoch, and the dispatch read that as a legitimate case. The guard treats it like every other bare number, and the reason is the card's own criterion — 「合法的 canonical 拼写一定带 `-` / `:` / `T` / `Z`」 — so a cell without any of them is dirty data whichever number it holds. **It costs the query nothing**: the read path still answers `1970-01-01T00:00:00.000Z` for it, before and after, which the suite asserts on a window that admits the row. Only the bytes survive that did not before. Discriminating by magnitude would be the guessing the sibling module's 「不可解残留如实记录」 posture exists to refuse. ## Evidence | what | result | |:--|:--| | Reproduction, before the fix | the table above, on real SQLite (better-sqlite3 13.0.3 / SQLite 3.53.4) | | `pnpm --filter @objectstack/driver-sql test` | **179 files passed, 11 skipped; 2645 tests passed, 168 skipped, 0 failed** | | `pnpm --filter @objectstack/driver-sql typecheck` | exit 0 | | New suite | 18 cases, all passing | | **Ablation** | guard predicate neutralised via `scripts/ablation-replace.mjs`: anchor 1 to 0, blob `bc27c909d4eb` to `8d264f7cf4ed`; the suite goes **6 red**, including `'12'` written to disk as `-4713-12-06T12:00:00.000Z` and `'2026'` written as `12:00:00`. Restored: blob back to `bc27c909d4eb` == HEAD, `git diff HEAD` empty, whole-tree `git status --porcelain` empty | | Gate families | derived with `scripts/pm/dispatch-gates.mjs` on the merged tree: **62 derived, 62 run, 0 NOT MEASURED, 0 UNRUN**, every one recording its exit code (captured before any pipe) | | `pnpm lint` | the whole repo-wide `eslint . --no-inline-config` run, exit 0, at `e9d0cf2bc` — no narrowing claimed | | `origin/main` merged | `e9d0cf2bc`, clean, no `os-regen` deferral; `pnpm install --frozen-lockfile`, full workspace build and `pnpm --filter @objectstack/spec check:generated` all clean afterwards | `check:doc-authoring` caught the first draft: the `warn` string carried `(#6009)`. Per the recorded ruling the id moved to the adjacent `//` comment, and the test now keys on the message's own words. ### Why the fixture rebuilds the column as TEXT A column knex creates for `Field.datetime` is declared `datetime`, which is **NUMERIC affinity**, so `'2026'` is converted to INTEGER 2026 on the way in and takes the expression's integer/real branch instead — the local write path mostly dodges this, exactly as the card's analysis says. The bare-numeric TEXT shape needs a TEXT-affinity column, which is what `RemoteTransport.mapFieldTypeToSQL` declares for every temporal column and what an externally-provisioned SQLite file can carry: `initObjects` adds missing columns to a table that already exists and never retypes one. The fixture therefore declares `at` / `tod` TEXT and keeps a second NUMERIC-affinity datetime column `ep` as the integer/real control — one table, both storage classes, which is the situation the repair expression exists for. ## What is NOT addressed here, and is why this is `Part of` rather than a close **The remote transport's backfill still writes the misreading.** `packages/drivers/driver-turso/src/remote-canonical-backfill.ts` builds its own statements and receives only `canonicalSqlFor`, so it does not inherit this guard. Its `convergeSql` is ```text update TABLE set COL = CANONICAL where rowid in (select rowid from TABLE where NOT_CANONICAL limit ?) ``` with `CANONICAL` being this same shared expression, over columns `RemoteTransport.mapFieldTypeToSQL` declares `TEXT` — which the card names as the one reasonably reachable side. That module is outside this dispatch's declared file surface (`packages/drivers/driver-sql/src/`), so it is reported rather than edited. The shape a follow-up needs is already available: `sqliteNonTemporalTextSql` is `protected` on `SqlDriver` exactly like `sqliteCanonicalDatetimeSql`, so it can be handed across the transport boundary the same way. Related, and measured while reading that module: its `unresolvedEpochTextRows` contract states that the rows it counts 「are fixpoints of the shared repair: reading them through it and reading them without it give the identical answer」, and therefore do not block the canonical mark. That holds for the rows **above** the recovery band (past the julian range, where `strftime` returns NULL) and **not** for the ones below it: `'2026'` and `'86400'` are digits-only, fall outside `1e12 ≤ v`, and are read as BC dates by the shared repair, so they are not fixpoints and dropping the repair does change what they match. **Answering the dispatch's question directly: `unresolvedEpochTextRows` is not the right place to count these.** Two reasons, either sufficient. It lives in `driver-turso`, outside the file surface. And its documented meaning is "recorded rather than guessed at, and harmless to the mark" — filing julian-misread rows under a name that asserts they are harmless would make a false statement true-looking. The local counterpart added here is deliberately a different thing: it is not a report, it is a **blocker** on the canonical mark, because that is what the rows require. ## Acceptance notes Observed while measuring, **not** filed and not fixed: - `'12:30'` in a `Field.datetime` column reads as `2000-01-01T12:30:00.000Z` and is converged to it. That is a genuine SQLite date format (`parseHhMmSs`), not the julian limb, so it is outside this card's fence and the guard deliberately leaves it alone. Noted, not filed: it is the read path's documented answer and changing it is a contract decision, not a defect fix. Taker: none identified. - SQLite's `now` keyword shares the julian limb's shape and is caught by the same predicate for the same reason — a stored literal `now` would otherwise be converged to whatever the clock said at boot. Recorded in the helper's docblock so it is read rather than rediscovered; no separate treatment. --- _Generated by [Claude Code](https://claude.ai/code/session_01NcPSwnmJHczmTu6FG7NMjE)_ --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 372931e commit 40626bd

3 files changed

Lines changed: 625 additions & 11 deletions

File tree

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
---
2+
"@objectstack/driver-sql": patch
3+
---
4+
5+
`backfillCanonicalDatetimes` / `backfillCanonicalTimes` no longer write SQLite's julian-day reading of a bare-numeric cell over the bytes on disk (#6009).
6+
7+
Both migrations use the read path's own repair expression as their `SET` value, and that expression's `else` arm is `coalesce(strftime(...), col)` — on the stated understanding that a value SQLite cannot parse falls through the `coalesce` unchanged. SQLite's time-value grammar has one limb that defeats it: a **bare number** is a julian day (`DDDD.DDDD`, the last of its documented formats), and `now` is the wall clock. For those two shapes `strftime` does not return NULL, so `coalesce` never fires and a confident wrong answer is what gets written. Measured on better-sqlite3 13.0.3 / SQLite 3.53.4 against a TEXT-affinity column:
8+
9+
```text
10+
'12' -> -4713-12-06T12:00:00.000Z '2026' -> -4707-06-11T12:00:00.000Z
11+
'86400' -> -4476-06-15T12:00:00.000Z '2440587.5' -> 1970-01-01T00:00:00.000Z
12+
'now' -> whatever the clock said '2026-08-06' -> 2026-08-06T00:00:00.000Z (correct)
13+
```
14+
15+
On a read that was a temporary misreading and the disk was untouched. On the `SET` side it is a write, and the original value is unrecoverable afterwards.
16+
17+
- **The guard is the structural complement of the parseable spellings, not a heuristic.** Every other format SQLite's date parser accepts goes through `parseYyyyMmDd` (which needs a `-`) or `parseHhMmSs` (which needs a `:`), so a cell the parser answers for while containing neither character reached it through the julian-day limb or the `now` limb — there is no third way in. `sqliteNonTemporalTextSql` is that predicate; it never inspects the magnitude of the number and never decides what the cell means, only that the migration must not rewrite it.
18+
- **A withheld row also blocks the canonical mark, and that is what keeps every query answer identical.** Skipping the row costs nothing while the read-side repair still applies to it — it keeps reading as the same instant it always did. Marking the column clean is what would move an answer: `needsLegacyDatetimeRepair` would drop the repair and the raw `'2026'` would be compared as TEXT. Unlike the `coalesce` fixpoints, these rows are not invariant under the repair, so they cannot ride through the mark. The column stays un-marked, reads keep their (unindexed) repair, and one `warn` names the count, the reason and the remedy.
19+
- **The read path is untouched, deliberately.** The maintainer's 2026-08-03 ruling on cloud#1005 refused teaching the shared read expression to recognise numeric-looking text: it is a public contract for every SQLite consumer, it runs on every read, and there it would misread a legitimate numeric-string column. Everything here runs once, as a migration, only on a column the metadata declares `Field.datetime` / `Field.time`, and it only ever declines to write.
20+
- **`previewDatetimeConvergence` / `previewTimeConvergence` inherit the same exclusion**, so `os migrate plan` still promises exactly the rows `apply` rewrites.
21+
22+
Reaching the julian branch needs a temporal column with TEXT affinity. A column knex creates for `Field.datetime` is declared `datetime`, which is NUMERIC affinity, so `'2026'` is converted to INTEGER on the way in and takes the epoch branch instead — the local write path mostly dodges this. A table that already existed when `initObjects` first saw it keeps whatever affinity it was created with, and `initObjects` adds missing columns without ever retyping one.

0 commit comments

Comments
 (0)