Repository navigation
Commit 40626bd
fix(driver-sql): stop the canonical temporal backfill writing SQLite julian-day misreadings to disk (#19688)
Part of #6009
Clause-②: no
The canonical temporal backfill no longer writes SQLite's **julian-day
reading** of a bare-numeric cell over the bytes on disk. The read path
is untouched, deliberately — see the fence below.
## The defect
`sqliteCanonicalDatetimeSql`'s `else` arm is
`coalesce(strftime('%Y-%m-%dT%H:%M:%fZ', col), col)`, and its documented
contract is that a value SQLite cannot parse falls through the
`coalesce` unchanged. SQLite's time-value grammar has one limb that
defeats that reasoning: a **bare number** is a julian day (`DDDD.DDDD`,
the last of its documented formats), and `now` is the wall clock. For
those two shapes `strftime` does not return NULL — it returns a
confident wrong answer — so `coalesce` never fires.
Reproduced first, before any code was written, on better-sqlite3 13.0.3
/ SQLite 3.53.4 against a TEXT-affinity column. The card's own table,
reproduced exactly, plus the two shapes that reach the same limb:
```text
'12' -> -4713-12-06T12:00:00.000Z '2026' -> -4707-06-11T12:00:00.000Z
'0' -> -4713-11-24T12:00:00.000Z '86400' -> -4476-06-15T12:00:00.000Z
'1999' -> -4707-05-15T12:00:00.000Z '1e5' -> -4439-09-09T12:00:00.000Z
'2440587.5' -> 1970-01-01T00:00:00.000Z 'now' -> the wall clock, at boot
'2026-08-06' -> 2026-08-06T00:00:00.000Z (correct — control)
'2026-08-06T00:00:00.000Z' -> itself (fixed point — control)
'1753660800000.0' -> itself (past the julian range, strftime NULL — control)
```
On a read that is a temporary misreading and the disk is untouched.
`backfillCanonicalDatetimes` uses the same expression as its `SET`
value, so on that side it is a **write**, and the original value is
unrecoverable afterwards.
## What landed
- **`sqliteNonTemporalTextSql`** — the structural **complement** of the
parseable spellings, not a heuristic. Every other format SQLite's date
parser accepts goes through `parseYyyyMmDd` (which needs a `-`) or
`parseHhMmSs` (which needs a `:`), so a cell the parser answers for
while containing **neither** character reached it through the julian-day
limb or the `now` limb — there is no third way in. The predicate never
inspects the magnitude of the number and never decides what the cell
*means*; it decides only that the migration must not rewrite it. It
probes with `julianday()` rather than `strftime()` because the two agree
on NULL-ness by construction, which makes one spelling serve both the
datetime expression and the time one.
- **Rows it matches are withheld from the `UPDATE`** in
`backfillCanonicalDatetimes` and `backfillCanonicalTimes`, and left
exactly as they are — which is what the backfill's own contract already
promises for every other uninterpretable value.
- **A withheld row also blocks the canonical mark, and that half is what
keeps every query answer identical.** Skipping the row costs nothing
while the read-side repair still applies to it: it keeps reading as the
same instant it always did. Marking the column clean is what would move
an answer — `needsLegacyDatetimeRepair` would drop the repair and the
raw `'2026'` would be compared as TEXT. Unlike the `coalesce` fixpoints,
these rows are **not** invariant under the repair, so they cannot ride
through the mark the way genuinely unparseable junk does. The column
stays un-marked, reads keep their (unindexed) repair, and one `warn`
names the count, the reason and the remedy. That is the degradation
posture this module already has for a failed `UPDATE`, reused rather
than invented.
- **`previewDatetimeConvergence` / `previewTimeConvergence` inherit the
same exclusion**, so `os migrate plan` still promises exactly the rows
`apply` rewrites — the pair's stated construction.
- **A `patch` changeset**, measured rather than assumed:
`@objectstack/driver-sql` is not private, its `files[]` ships `dist`,
and the new symbol reaches it (10 hits in `dist/index.js` against a
positive control of 12 for `sqliteCanonicalDatetimeSql`).
### The fence, honoured
> 倾向**不要**在共享表达式里加启发式 —— 维护者 2026-08-03 在 cloud#1005 已就近似的提案裁过否决(公共契约 +
每次读都跑 + 会误读合法的数字字符串列)。
The read expression is byte-for-byte unchanged, and the suite **pins**
that it still misreads, so a later "improvement" to it fails this suite
rather than slipping through it. Everything added here runs once, as a
migration, only on a column the metadata declares `Field.datetime` /
`Field.time`, and it only ever declines to write. No new gate is added
or recommended: this is a guard inside an existing migration path.
### On `'2440587.5'`
The card notes SQLite's answer there is genuinely the julian epoch, and
the dispatch read that as a legitimate case. The guard treats it like
every other bare number, and the reason is the card's own criterion —
「合法的 canonical 拼写一定带 `-` / `:` / `T` / `Z`」 — so a cell without any of
them is dirty data whichever number it holds. **It costs the query
nothing**: the read path still answers `1970-01-01T00:00:00.000Z` for
it, before and after, which the suite asserts on a window that admits
the row. Only the bytes survive that did not before. Discriminating by
magnitude would be the guessing the sibling module's 「不可解残留如实记录」 posture
exists to refuse.
## Evidence
| what | result |
|:--|:--|
| Reproduction, before the fix | the table above, on real SQLite
(better-sqlite3 13.0.3 / SQLite 3.53.4) |
| `pnpm --filter @objectstack/driver-sql test` | **179 files passed, 11
skipped; 2645 tests passed, 168 skipped, 0 failed** |
| `pnpm --filter @objectstack/driver-sql typecheck` | exit 0 |
| New suite | 18 cases, all passing |
| **Ablation** | guard predicate neutralised via
`scripts/ablation-replace.mjs`: anchor 1 to 0, blob `bc27c909d4eb` to
`8d264f7cf4ed`; the suite goes **6 red**, including `'12'` written to
disk as `-4713-12-06T12:00:00.000Z` and `'2026'` written as `12:00:00`.
Restored: blob back to `bc27c909d4eb` == HEAD, `git diff HEAD` empty,
whole-tree `git status --porcelain` empty |
| Gate families | derived with `scripts/pm/dispatch-gates.mjs` on the
merged tree: **62 derived, 62 run, 0 NOT MEASURED, 0 UNRUN**, every one
recording its exit code (captured before any pipe) |
| `pnpm lint` | the whole repo-wide `eslint . --no-inline-config` run,
exit 0, at `e9d0cf2bc` — no narrowing claimed |
| `origin/main` merged | `e9d0cf2bc`, clean, no `os-regen` deferral;
`pnpm install --frozen-lockfile`, full workspace build and `pnpm
--filter @objectstack/spec check:generated` all clean afterwards |
`check:doc-authoring` caught the first draft: the `warn` string carried
`(#6009)`. Per the recorded ruling the id moved to the adjacent `//`
comment, and the test now keys on the message's own words.
### Why the fixture rebuilds the column as TEXT
A column knex creates for `Field.datetime` is declared `datetime`, which
is **NUMERIC affinity**, so `'2026'` is converted to INTEGER 2026 on the
way in and takes the expression's integer/real branch instead — the
local write path mostly dodges this, exactly as the card's analysis
says. The bare-numeric TEXT shape needs a TEXT-affinity column, which is
what `RemoteTransport.mapFieldTypeToSQL` declares for every temporal
column and what an externally-provisioned SQLite file can carry:
`initObjects` adds missing columns to a table that already exists and
never retypes one. The fixture therefore declares `at` / `tod` TEXT and
keeps a second NUMERIC-affinity datetime column `ep` as the integer/real
control — one table, both storage classes, which is the situation the
repair expression exists for.
## What is NOT addressed here, and is why this is `Part of` rather than
a close
**The remote transport's backfill still writes the misreading.**
`packages/drivers/driver-turso/src/remote-canonical-backfill.ts` builds
its own statements and receives only `canonicalSqlFor`, so it does not
inherit this guard. Its `convergeSql` is
```text
update TABLE set COL = CANONICAL where rowid in (select rowid from TABLE where NOT_CANONICAL limit ?)
```
with `CANONICAL` being this same shared expression, over columns
`RemoteTransport.mapFieldTypeToSQL` declares `TEXT` — which the card
names as the one reasonably reachable side. That module is outside this
dispatch's declared file surface (`packages/drivers/driver-sql/src/`),
so it is reported rather than edited. The shape a follow-up needs is
already available: `sqliteNonTemporalTextSql` is `protected` on
`SqlDriver` exactly like `sqliteCanonicalDatetimeSql`, so it can be
handed across the transport boundary the same way.
Related, and measured while reading that module: its
`unresolvedEpochTextRows` contract states that the rows it counts 「are
fixpoints of the shared repair: reading them through it and reading them
without it give the identical answer」, and therefore do not block the
canonical mark. That holds for the rows **above** the recovery band
(past the julian range, where `strftime` returns NULL) and **not** for
the ones below it: `'2026'` and `'86400'` are digits-only, fall outside
`1e12 ≤ v`, and are read as BC dates by the shared repair, so they are
not fixpoints and dropping the repair does change what they match.
**Answering the dispatch's question directly: `unresolvedEpochTextRows`
is not the right place to count these.** Two reasons, either sufficient.
It lives in `driver-turso`, outside the file surface. And its documented
meaning is "recorded rather than guessed at, and harmless to the mark" —
filing julian-misread rows under a name that asserts they are harmless
would make a false statement true-looking. The local counterpart added
here is deliberately a different thing: it is not a report, it is a
**blocker** on the canonical mark, because that is what the rows
require.
## Acceptance notes
Observed while measuring, **not** filed and not fixed:
- `'12:30'` in a `Field.datetime` column reads as
`2000-01-01T12:30:00.000Z` and is converged to it. That is a genuine
SQLite date format (`parseHhMmSs`), not the julian limb, so it is
outside this card's fence and the guard deliberately leaves it alone.
Noted, not filed: it is the read path's documented answer and changing
it is a contract decision, not a defect fix. Taker: none identified.
- SQLite's `now` keyword shares the julian limb's shape and is caught by
the same predicate for the same reason — a stored literal `now` would
otherwise be converged to whatever the clock said at boot. Recorded in
the helper's docblock so it is read rather than rediscovered; no
separate treatment.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01NcPSwnmJHczmTu6FG7NMjE)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 372931e commit 40626bd
3 files changed
Lines changed: 625 additions & 11 deletions
File tree
- .changeset
- packages/drivers/driver-sql/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
0 commit comments