Repository navigation
Commit 8a399b2
fix(service-datasource): the admin door reads a datasource's origin from provenance, and a metadata-door write reaches it in the same boot (#21977)
Fixes #21923
Clause-②: no
## What changes
A runtime datasource is one record that two doors serve: the datasource
admin door (`/api/v1/datasources`) and the metadata door
(`/api/v1/meta/datasource`). They disagreed about it in three ways. All
three land in
`packages/services/service-datasource/src/datasource-admin-plugin.ts`.
1. **Origin comes from provenance, not from the record.**
`listDatasourceRecords` and `getDatasourceRecord` served `origin:
r.origin ?? 'code'`. A metadata-door body has no `origin`, or it asserts
`origin: 'code'`. So after a restart, the boot restore registered the
row and the admin door served it as code-defined and refused its
`PATCH`. A new `servedOrigin(ctx, name)` returns `code` only for a name
in the host's code-datasource set, and `runtime` for every other name,
whatever the record says. That set is the `code-datasource-names` kernel
service PR #21965 landed, which the boot restore and the metadata door's
refusal already read. Boot pool rehydration filters on the served
origin, so such a datasource also gets its live pool after a restart.
2. **Same-boot reach.** The metadata door persists to `sys_metadata` and
the SchemaRegistry. It never registers the record in the MetadataService
slot the admin door lists. At `start()`, the plugin now registers the
protocol's awaited `datasource` mutation projector (ADR-0094,
`registerMutationProjector`; no `metadata-protocol` edit). After each
metadata-door save, publish, revert, rollback or delete, and before that
door answers, the projector does three things:
- it re-reads the stored row (`sys_metadata`, the same read the boot
restore makes);
- it registers or unregisters the MetadataService slot to match that
row;
- it converges the live pool through the existing `convergePool`.
A name in the code set is skipped, because code wins. So the metadata
door's repair `DELETE` of a stored shadow row under a code name keeps
the code definition served.
3. **The reverse direction.** `persistDatasourceRow` wrote the
`sys_metadata` row with no `checksum`. The metadata door's reads serve
`row.checksum ?? hashSpec(body)` as the version, but its writes compare
the parent against the raw column. So every metadata-door `PUT` and
`DELETE` of an admin-created datasource answered `409
METADATA_CONFLICT`. The row now carries `hashSpec(record,
'datasource')`, imported from `@objectstack/metadata-core`. That is the
same computation `SysMetadataRepository.put` stamps (`hashSpec(body,
ref.type)`). `@objectstack/metadata-core` moves from devDependencies to
dependencies; in the lockfile only the importer entry moves.
`convergePool` is the receive half of the datasource cluster bridge, and
the projector's pool step. It now decides "code" from the same set
instead of `row.origin !== 'runtime'`. It pools every other stored row
as runtime, and stamps the record `origin: 'runtime'` before the record
reaches the connect context.
Editing a code-defined datasource is still refused at both doors.
### Why `Clause-②: no`
The `origin` docblock in `packages/spec/src/data/datasource.zod.ts`
reads: "`runtime` — created via the Studio wizard, persisted in the
runtime metadata store, environment-scoped, editable", and "Never
accepted from client input". The published contract already says such a
datasource is editable, so the admin door's refusal of a metadata-door
datasource as code-defined was a false refusal. No key, export or stored
shape moves.
`projectionApplied` now appears on the answer to a datasource write
through the metadata door. That key is already declared optional on the
protocol's write answer, so this adds no new key to a published payload.
## Measured on the base (`c9761cd2fb`): the new door pin is red
`datasource-meta-door-reaches-admin-door.dogfood.test.ts`, run on the
unmodified base, gave 4 failed and 2 passed:
- `:174` same boot, after `PUT /meta/datasource/dogfood_meta_none_21923`
answered 200: `expected undefined to match object { origin: 'runtime',
…(1) }`. The admin door did not list it.
- `:193` an admin-created datasource, then `PUT
/meta/datasource/dogfood_admin_rt_21923`: `409 METADATA_CONFLICT`,
"Expected parent hmac-sha256:… but current is null."
- `:206` after a restart: received `"origin": "code"`, expected
`"runtime"`.
## Mechanism hypotheses, measured
1. **Does any code registration reach the MetadataService without its
name in the set?** This was measured with `bootStack` on HEAD
`493c13dbb3`, comparing `metadata.list('datasource')` against
`code-datasource-names`:
| composition | listed at boot | code set | listed but outside the set |
|---|---|---|---|
| showcase | `default`, `showcase_external` | `default`,
`showcase_external` | none |
| crm | `crm_analytics`, `crm_primary`, `default` | `crm_analytics`,
`crm_primary`, `default` | none |
| multi-package | `default` | `default` | none |
A package installed after boot does not register datasources in the
MetadataService at all: `registerApp` and the `sys_packages` rehydrate
write only the engine registry. So the admin door never lists one. The
three residual paths, read and not measured on a boot, are under
Acceptance notes. The fix never falls back to `?? 'code'`.
2. **Same-boot reach.** Verified on main (the `:174` failure above). The
seam is the awaited projector, not `onMetadataMutation`: the metadata
door answers only after the admin door lists the record, and a
projection failure is reported on that answer as well as logged.
- **A live pool is needed.** The admin door's create calls
`registerPool`, which connects. The dogfood pin asserts `connected` for
a metadata-door save in the same boot and again after a restart. Before
the fix there was no pool, and after a restart the served `code` kept
the datasource out of pool rehydration.
3. **Checksum.** The fix lands inside `service-datasource` by importing
the repository's own computation. The residual, rows already stored
without a checksum, is under Acceptance notes.
4. **Carrier notes.**
- `convergePool` is covered by the provenance rule (above).
- The `restoreRuntimeDatasources` and `rehydratePools` warnings that go
only to `options.logger` are not changed. The bounded in-place-fix
condition "same defect class" does not hold for them; see Acceptance
notes.
## Tests (head `493c13dbb3`)
- `pnpm --filter @objectstack/service-datasource typecheck`: exit 0. The
package's `tsc --listFiles` includes the edited test file.
- `pnpm --filter @objectstack/service-datasource test`: 41 files and 760
tests pass.
- 7 new cases in `datasource-admin-plugin.test.ts`: the served origin,
the projector registration, a metadata-door save, an edit and a delete
reaching the admin door with their pool, a write under a code name
changing nothing, a peer signal pooling by provenance, and the checksum.
- One fixture re-judged: the "artefact (code) datasource" case now
carries the code set, as the runtime fills it, instead of relying on a
missing `origin`.
- `datasource-system-context.pin.test.ts` follows `convergePool`'s new
`ctx` parameter.
- `pnpm --filter @objectstack/dogfood typecheck`: exit 0.
- Four dogfood files pass, 24 tests (after `pnpm --filter
@objectstack/service-datasource build`, dist marker preflight present):
the new pin, `datasource-restore-code-wins`,
`meta-door-code-datasource`, and
`external-import-code-datasource-namespace`.
- The rejection pin asserts `code` plus `status`: the admin `PATCH` of
`showcase_external` still answers `400 DATASOURCE_ADMIN_ERROR`, with the
message's first sentence.
### Ablations (each mutation through `scripts/ablation-replace.mjs`,
rebuilt, `ablation-dist-preflight` on both legs, restored to the HEAD
blob `a7f28b52b967`, `git diff HEAD` empty, and the restore leg rebuilt)
| put back | unit (26 in file) | dogfood |
|---|---|---|
| `origin ?? 'code'` in list and get | 3 red | new pin 3 red (`:174`,
`:206`, `:222`) |
| the record's own `origin` first (`r.origin ?? servedOrigin(…)`) | 2
red | new pin 2 red (`:175`, the body asserting `code` served as code) |
| no projector (`void this.projectMetadataDoorWrites;`) | 4 red | new
pin 4 red (`:174`, `:194`, `:206`, `:222`) |
| `convergePool` reads `row.origin !== 'runtime'` | 3 red | new pin 2
red (`:178`, no pool) |
| no checksum (`const checksum = null`) | 1 red | new pin 3 red (`:193`
and `:221`, `409 METADATA_CONFLICT`) |
| projector without its code-name guard | 1 red |
`datasource-restore-code-wins` 1 red (`:271`, the repair `DELETE`
unregisters the code definition) |
Three first attempts did not run: the no-projector, `convergePool` and
no-checksum mutations each failed the DTS build with an unused symbol
(TS6133), so nothing was measured. Each was redone with a mutation that
compiles. In every void attempt the restore leg was proven the same way.
## Gates (head `493c13dbb3`)
`node scripts/pm/dispatch-gates.mjs --commands --repo
objectstack-ai/objectstack` re-derived 78 families on this change. That
is a superset of the 49 the dispatch listed; the additions come from the
changeset, the lockfile and `package.json`. All 78 exit 0. `--ran`
reconciles: 78 derived, 78 run, 0 NOT-MEASURED, each with a recorded
exit code.
One gate needed a rerun. `pnpm check:dual-build-cjs-loads` first exited
3 with PREREQUISITE NOT MET (8 unrelated packages had no `dist/`). After
those packages were built, it exited 0: 106 require entry points across
66 packages load.
Narrowed lint (`eslint --no-inline-config --format json`) over the 4
touched `.ts` files:
- every one resolves a config (`--print-config`);
- the JSON output counts 4 files, 0 errors and 0 warnings;
- `eslint.config.mjs` enables no type-aware linting, so this diff cannot
move a verdict on an untouched file.
The repo-wide `pnpm lint` is left to CI.
## Acceptance notes
- **Rows stored before this release.** An admin-created datasource row
written before this release has no `checksum`, so the metadata door
still answers it 409 until the admin door next writes it; one admin edit
is the remedy. The asymmetry itself is in `metadata-protocol`'s
`sys-metadata-repository.ts`: `rowToItem` serves `row.checksum ??
hashSpec(body)` while `put` and `delete` compare the raw column. It
holds for any type's null-checksum row, and is reported to the seat
rather than edited here.
- **Code datasources the code set does not cover** (read, not measured
on a boot):
- Dev artifact HMR (`artifactWatch`) re-registers the artifact's
datasources through the MetadataPlugin door. A datasource added to the
artifact after boot is not in the set, because `AppPlugin` memoizes its
owners, so the admin door serves it as runtime until a restart.
- The legacy `FilesystemLoader` (a `datasource/` directory under the
metadata root) lists datasources nothing adds to the set.
- A host that composes the artifact door without `AppPlugin` or
`DefaultDatasourcePlugin` registers no set, so nothing is code there;
the boot restore already treats such a host that way.
The remedy for each is on the producer side: contribute to the set. It
is not a consumer default.
- **Cluster.** The projector runs on the writing replica only; the
protocol's cluster channel replays mutation listeners, not projectors.
So a metadata-door datasource write does not converge peer replicas'
MetadataService or pools until they restart. Before this change no
replica converged. Cross-replica MetadataService coherence is a
separate, open measurement.
- **Lost warnings under `os serve`.** The existing
`restoreRuntimeDatasources` and `rehydratePools` warnings still go only
to `options.logger`, which `os serve` does not pass. This change widens
the population that reaches `rehydratePools`, because metadata-door
datasources now rehydrate. The new projector adds no log line of its
own: a projection failure is reported on the metadata door's answer
(`projectionApplied`) and logged by the protocol.
- **Not this card.** The metadata door's own `GET` serves a stored row
under a code-defined name (the overlay read); #21922 remains open for
that half.
## Files
- `packages/services/service-datasource/src/datasource-admin-plugin.ts`
-
`packages/services/service-datasource/src/__tests__/datasource-admin-plugin.test.ts`
-
`packages/services/service-datasource/src/__tests__/datasource-system-context.pin.test.ts`
- `packages/services/service-datasource/package.json`, `pnpm-lock.yaml`
(`@objectstack/metadata-core` becomes a dependency)
-
`packages/qa/dogfood/test/datasource-meta-door-reaches-admin-door.dogfood.test.ts`
(the door pin, declared on #6024)
- `.changeset/21923-datasource-origin-from-provenance.md`
(`@objectstack/service-datasource` patch)
---
_Generated by [Claude
Code](https://claude.ai/code/session_01WMQprn46CND82KmY8sZWBu)_
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 04e776b commit 8a399b2
7 files changed
Lines changed: 580 additions & 37 deletions
File tree
- .changeset
- packages
- qa/dogfood/test
- services/service-datasource
- src
- __tests__
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
Lines changed: 237 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
39 | 39 | | |
40 | 40 | | |
41 | 41 | | |
| 42 | + | |
42 | 43 | | |
43 | 44 | | |
44 | 45 | | |
| |||
62 | 63 | | |
63 | 64 | | |
64 | 65 | | |
65 | | - | |
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| |||
0 commit comments