Skip to content

Commit 381d735

Browse files
committed
Merge remote-tracking branch 'origin/main' into claude/issue-21068-macro-past-date-range
2 parents 1dbee5b + 9c8b65a commit 381d735

80 files changed

Lines changed: 4719 additions & 2398 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
---
2+
'@objectstack/driver-sql': patch
3+
---
4+
5+
fix(driver-sql): `os migrate plan` on a database that does not exist yet no longer prints `DATABASE_ERROR` for the tables whose DDL it deferred (#20821)
6+
7+
`os migrate plan` (and the boot of `os migrate apply`) runs with the SQL driver's DDL deferred: the driver records every table as pending `create_table` and creates none of them. The same boot then reads `sys_metadata`, `sys_metadata_activation` and `sys_migration`. On a new database those tables do not exist yet, so each read was refused, and each refusal printed a line like this on the driver's warn channel (stderr by default):
8+
9+
```text
10+
[sql-driver] DATABASE_ERROR — the backend refused a read on 'sys_metadata' (SQLITE_ERROR) ... no such table: sys_metadata
11+
```
12+
13+
Nothing was wrong: every reader already answers from the refusal, and the plan lists the same tables as pending creates. A dry run on a new database printed six of these lines.
14+
15+
The driver now sends such a refusal to the logger's `debug` channel instead of `warn`, when all three hold:
16+
17+
- this driver has DDL deferred;
18+
- the refused statement targets a table whose DDL this driver deferred;
19+
- the shared `isMissingTableError` predicate from `@objectstack/types` recognises the refusal as that table being missing.
20+
21+
The default logger has no `debug`, so the line is not printed. The refusal is still thrown to the caller with the same envelope (`DATABASE_ERROR`, status 500), and the plan's output is unchanged.
22+
23+
What still warns:
24+
25+
- every other refusal on a deferred driver, such as a malformed statement on a table that exists;
26+
- a missing table that the driver did not defer, such as a table nothing in the boot declares;
27+
- every refusal once the deferred DDL has been applied, or on a driver that never deferred any.
28+
29+
There is nothing to migrate.
Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
'@objectstack/spec': patch
3+
---
4+
5+
fix(spec): protocol 18's migration rationale now covers the form view's inline grid columns and the identity-only currency `scale` refusal (#20901)
6+
7+
**`@objectstack/spec`**
8+
9+
- **`MIGRATIONS_BY_MAJOR[18].rationale` gains one fragment, `form-view-subform-columns-closed`.** It is the paragraph `os migrate meta --step` shows for the protocol 17 → 18 hop. The new sentences say that a form view's `subforms[].columns` now takes the strict `InlineGridColumnSchema` a relationship field's `inlineColumns` takes, that the conversion `form-view-subform-columns-canonicalized` respells a `{ field }` column as `{ name }` in stored rows and assembled artifacts while an author writing `field` is refused, and that `defineStack` refuses `scale` on a column that declares no `type` when its `name` is a `currency` field of a child object declared in the same stack (`inline-grid-column-identity-only-currency-scale-refused`).
10+
- Text only: no schema, conversion or migration entry changes, and `conversionIds` and `semantic` for step 18 are unchanged.
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
'@objectstack/core': minor
3+
---
4+
5+
feat(core): the bulk-import runner, its row coercion, the mapping apply and the field-meta map now live in `@objectstack/core`, beside `bulkWrite` (#20919)
6+
7+
`runImport` (with `sanitizeRowError` and its option/result types), the cell
8+
coercion (`coerceRow`, `coerceFieldValue`, `parseDateCell`, `parseNumberCell`,
9+
`parseBooleanCell`, `matchOption`, `splitMulti`, `isBlank`), the `mapping`
10+
artifact pipeline (`applyMappingToRows`, `refuseUnknownMappingTargets`,
11+
`MappingArtifactLike`, `MappingFailure`, `ApplyMappingOptions`) and the field
12+
metadata map (`buildFieldMetaMap`, `ExportFieldMeta`) are exported from
13+
`@objectstack/core`. They moved here unchanged from `@objectstack/rest` so the
14+
connector sync executor in `@objectstack/service-automation` writes through the
15+
same runner as the HTTP import door without depending on the HTTP layer. Nothing
16+
to change for consumers: `@objectstack/rest` re-exports every name it exported
17+
before.
Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
---
2+
'@objectstack/rest': patch
3+
---
4+
5+
refactor(rest): the import runner, coercion, mapping apply, field-meta map and error classification moved to `@objectstack/core` / `@objectstack/types`; `rest` re-exports them (#20919)
6+
7+
`runImport`, `coerceRow`, `buildFieldMetaMap` and their types (from the package
8+
index), and `mapDataError` with its sibling classification exports, are now
9+
re-exported from their new homes — byte-identical code, the same names, the same
10+
behaviour at both import routes and at `plugin-auth`'s identity import. Nothing to
11+
change for consumers.
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
'@objectstack/service-automation': minor
3+
---
4+
5+
feat(service-automation): the connector sync executor pulls a `mapping`'s `connectorSource` and writes it through the import runner (#20919)
6+
7+
`AutomationServicePlugin.pullConnectorSource({ mapping, context })` (and the
8+
exported `pullConnectorSource(deps, opts)`) reads the mapping through the
9+
protocol's `getMetaItem`, resolves `connectorSource.connector` to a declared
10+
(`connectors[]`) `rest` or `openapi` instance, makes ONE call to its read action,
11+
takes the array at `recordsPath` (default `body`), projects it through the
12+
mapping's `fieldMapping` and writes it with `@objectstack/core`'s `runImport` —
13+
the import door's coercion, `mode` / `upsertKey` matching and per-row verdicts,
14+
with the door's defaults for every knob `connectorSource` does not declare.
15+
16+
- **Watermark, read from the target.** For `connectorSource.watermark`, the
17+
starting point sent as `query[watermark.param]` is the highest value already
18+
stored in the target field a `fieldMapping` entry copies `watermark.field` onto
19+
(transform `none`). Nothing else stores it.
20+
- **One response per pull.** The connector's paging is not followed.
21+
- **Loud refusals,** each a `ConnectorPullError` (`code`, `status`, `reason`)
22+
raised before anything is written: a plugin-registered or unregistered
23+
connector, a degraded instance, a provider other than `rest` / `openapi`, an
24+
undeclared action, `update` / `upsert` with an empty `upsertKey`, a
25+
`javascript` transform, an unmapped `watermark.field`, an `ok: false` answer,
26+
a non-array at `recordsPath` and a non-object record.
27+
- **Nothing schedules a pull** — a `job` will drive it; the caller supplies the
28+
execution context.
29+
30+
The plugin now records the provider of each declared connector instance it
31+
materializes, which the executor reads.
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
'@objectstack/spec': patch
3+
---
4+
5+
docs(spec): `mapping.connectorSource` is pulled when a job drives it — the describes say where the watermark is read from and that a pull reads one response (#20919)
6+
7+
The `connectorSource` describe no longer says the pull is not executed: the
8+
connector sync executor in `@objectstack/service-automation` reads the binding,
9+
and nothing schedules a pull until the `job` stage lands. `watermark.field` now
10+
states that the next pull's starting point is read from the TARGET field a
11+
`fieldMapping` entry copies it onto (an unmapped one is refused at pull time), and
12+
`watermark` states the one-response limit: the connector's paging is not followed,
13+
so a paged endpoint yields its first page only. The liveness ledger's
14+
`connectorSource` rows are `live` and keep `authorWarn` (nothing schedules a pull
15+
yet). The retired `connector.syncConfig` prescription and the
16+
`connector-sync-keys-retired` upgrade entry say the same, and the entry's
17+
acceptance criterion no longer claims the connector is validated at authoring.
18+
No key, value or default changed.
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
'@objectstack/types': minor
3+
---
4+
5+
feat(types): the data-error classification table (`mapDataError` and its judgements) is exported from `@objectstack/types` (#20919)
6+
7+
The classification half of `@objectstack/rest`'s error boundary — `mapDataError`
8+
(a thrown error → the `{ status, body }` ADR-0112 answer, without emitting it),
9+
`declaredHttpStatus`, `declaredServerFaultAnswer`, `sandboxBusinessMessage`,
10+
`boundedDeclaredUserMessage`, `boundedDeclaredRefusalMessage` and
11+
`isEngineDuplicateRecordEnvelope` — moved here unchanged, beside the primitives it
12+
composes, so the bulk-import runner in `@objectstack/core` judges a failed row with
13+
the same table the REST door answers with. `@objectstack/rest` keeps the emitters
14+
and re-exports every name it exported before. The module also exports the eight
15+
helpers the REST emitters compose (`truncateClientMessage`, `thrownCodeFields`,
16+
`withoutDeclaredCodePrefix`, `withDeclaredUserMessage`, `isSandboxOrigin`,
17+
`isSandboxCrash`, `fiveXxArmDisplacesDeclared4xx`, `structuredCodeAnswer`).
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
---
2+
'@objectstack/plugin-security': minor
3+
---
4+
5+
fix(plugin-security)!: the record an anonymous public-form submit echoes back passes the result masker, so a field whose masking rule applies is echoed masked (#21062)
6+
7+
Clause-②: no (narrowing)
8+
9+
<!-- adr-0087: not-required (no-migration-prescription) Nothing an author wrote is rewritten or changes meaning: every maskingRule already declared that it applies to every non-system caller who does not hold all of the field's requiredPermissions, and an anonymous form submitter is such a caller. The runtime now honours that declaration on the one door that skipped the masker. No stored metadata, schema key or published type moves. -->
10+
11+
**BREAKING for the anonymous public-form submit's response.**
12+
13+
**What changed.** A public form's submission is authorized by the ADR-0056
14+
declaration-derived grant, which admits the create and the read-back on the
15+
form's declared object and passes before any permission set is resolved. The
16+
grant handed the operation to the engine and returned before the result masker
17+
ran, so the record echoed in the `201` body carried every field whose
18+
`maskingRule` applies as stored: the field the form collects, and a field
19+
filled from its `defaultValue` that the form never shows.
20+
21+
The grant now hands what it returns to the same result masker the data plane
22+
uses, for the caller it stands in for: the permission sets resolved for the
23+
grant's context (the deployment's guest set when it registers one, otherwise
24+
none) and the object posture those sets read. A field whose masking rule
25+
applies is echoed masked. A field the caller's sets mark unreadable, or whose
26+
`requiredPermissions` they do not hold, is masked the way the data plane masks
27+
it for that caller. The read-backs the grant admits are masked the same way.
28+
29+
**What did not change.** The grant admits exactly what it admitted: the create
30+
and the read-back on the form's declared object, and nothing else. The
31+
server-managed fields are still stripped from the submitted row. The stored row
32+
is unchanged; only the echo is masked.
33+
34+
**One more refusal, by the same rule.** If the caller's permission sets or the
35+
object's security posture cannot be read, the submission is now refused with
36+
`403 PERMISSION_DENIED` before anything is written, as every other caller's
37+
request already is. The masking rules come from that posture, so the echo
38+
cannot be masked without it.
39+
40+
**What to do.** Nothing, unless a client reads a masked field's stored value
41+
back out of the submit response. The response now carries the masked value, as
42+
every other non-system read does. A field's `requiredPermissions` are the gate
43+
that lifts its mask, so a deployment whose guest set holds all of them is
44+
echoed the stored value; otherwise drop the `maskingRule`.

‎content/docs/references/data/mapping.mdx‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ const result = ImportFieldMappingSchema.parse(data);
4949
| **fieldMapping** | `{ source: string \| string[]; target: string \| string[]; transform: Enum<'none' \| 'constant' \| 'lookup' \| 'split' \| 'join' \| 'javascript' \| 'map'>; params?: object }[]` | ✅ | |
5050
| **mode** | `Enum<'insert' \| 'update' \| 'upsert'>` | optional (default: `"insert"`) | |
5151
| **upsertKey** | `string[]` | optional | Fields to match for upsert (e.g. email) |
52-
| **connectorSource** | `{ connector: string; action: string; input?: Record<string, any>; recordsPath?: string; … }` | optional | Pull binding: the rest/openapi connector this mapping pulls rows from (one-way, full or timestamp-incremental; a `job` sets the cadence). Declared; the pull is not executed yet |
52+
| **connectorSource** | `{ connector: string; action: string; input?: Record<string, any>; recordsPath?: string; … }` | optional | Pull binding: the rest/openapi connector this mapping pulls rows from (one-way, full or timestamp-incremental; a `job` sets the cadence). Pulled when a job drives it; nothing schedules it yet |
5353
| **_lock** | `Enum<'none' \| 'no-overlay' \| 'no-delete' \| 'full'>` | optional | Item-level lock — controls overlay & delete (ADR-0010). |
5454
| **_lockReason** | `string` | optional | Human-readable reason shown when a write is refused by _lock. |
5555
| **_lockSource** | `Enum<'artifact' \| 'package' \| 'env-forced'>` | optional | Layer that set _lock (artifact \| package \| env-forced). |
@@ -75,7 +75,7 @@ const result = ImportFieldMappingSchema.parse(data);
7575
| **action** | `string` | ✅ | Key of the connector action that reads the records: `request` on a `rest` connector, an operation key on an `openapi` connector |
7676
| **input** | `Record<string, any>` | optional | Fixed input passed to the action on every pull (e.g. `{ method: 'GET', path: '/contacts' }` for `rest`) |
7777
| **recordsPath** | `string` | optional | Dot path, within the action's result, to the array of records (e.g. `body.results`); omitted ⇒ the result's `body` is the array |
78-
| **watermark** | `{ field: string; param: string }` | optional | Timestamp-incremental pull; omitted ⇒ every pull reads the full set |
78+
| **watermark** | `{ field: string; param: string }` | optional | Timestamp-incremental pull; omitted ⇒ every pull reads the full set. Either way a pull makes ONE action call and reads ONE response: the connector's paging is not followed, so a paged endpoint yields its first page only — and an incremental pull over a newest-first paged endpoint moves its starting point past the pages it never read |
7979

8080

8181
---

0 commit comments

Comments
 (0)