Repository navigation
Commit 54c3ce1
Fixes #22332
Clause-②: no
## What was wrong
The chunk door (`PUT
/storage/upload/chunked/:uploadId/chunk/:chunkIndex`) read the session
row, merged its part into the parts list in memory (`recordChunk`) and
wrote the whole record back with an unconditional by-id update. Two PUTs
to one upload at once both read the same record, and the later write
erased the earlier part: both answered `200`, `sys_upload_session.parts`
kept one part, `uploaded_chunks` / `uploaded_size` counted one chunk,
and the completion door refused the upload `409` naming the lost chunk.
## The atomic route taken, and why
**A compare-and-set on the row's three progress columns, with a bounded
re-read-and-merge retry.** This is the route the store already supports;
it needs no new object, no new field and no `packages/spec` change.
- **On a wired engine** it is the engine's own conditional update:
`engine.update('sys_upload_session', progress, { where: { parts,
uploaded_chunks, uploaded_size, id }, multi: true, context })`. The
update-dispatch module (`resolveEngineUpdateDispatch`,
`@objectstack/metadata-core`) names exactly this shape, an id in `where`
beside further keys with `multi: true`, as "the compare-and-set
spelling". It routes to `driver.updateMany`, which evaluates the guard
in the same statement that writes and answers the matched-row count.
driver-sql issues one `UPDATE … WHERE` with the tenant scope applied;
driver-memory matches and writes in one synchronous step. Every driver's
`updateMany` (sql, memory, turso, mongodb) resolves a count.
- **On the engine-absent stand-in** it is one synchronous
compare-and-set on the Map.
- **Why these columns are the guard:** they are exactly the values the
merge read, so the comparison is exact. `updated_at` as a version has
millisecond grain, so two writes in one millisecond would still match. A
per-part row would be a new object, a schema change, and was not needed.
- ⛔ No process-local lock: a second server process would defeat it. ⛔
The completion door's guard is untouched.
The store side is `updateSessionProgressIfUnchanged` in
`metadata-store.ts`. It is a module export that is not re-exported from
the package entry, following the `organizationOutOfWriteReach` pattern,
so `StorageMetadataStore`'s public face does not grow.
The door merges and writes up to `CHUNK_RECORD_ATTEMPTS` (16) times. A
conditional write loses only to a write that landed between its read and
itself, so each lost attempt is another chunk recorded.
- **On exhaustion** the door answers `409 RESOURCE_CONFLICT` with
`error.details: { chunkIndex, attempts }`. The chunk's bytes are stored
but the record does not hold them, and re-sending the chunk replaces its
slot. It never answers a silent `200`.
- **When the conditional write cannot reach the row** (it misses, and
the row still holds the progress the write was conditioned on), the door
answers `500` at once. It does not loop into a `409` that would tell the
uploader to retry something that cannot succeed.
## Measured: what the store supports (H2)
A probe against the real `ObjectQL` over `SqlDriver` (better-sqlite3,
`:memory:`) at `e36ee5351b`:
| conditional update | answer |
| :--- | ---: |
| guard as read | 1 |
| stale guard | 0 |
| guard on an 8-part `parts` string | 1 |
| guard of NULL columns (lowered to `IS NULL`) | 1 |
| row stamped org_B, acting organization org_A | 0, no throw |
| the same row, acting organization org_B | 1 |
| missing row | 0 |
## Measured: the reproduction, before and after
`pnpm --filter @objectstack/service-storage exec vitest run
--maxWorkers=2 src/chunk-part-record-concurrency.test.ts`
**Before**, the pins on `main` at `e36ee5351b` (which carries the
completion guard): 8 failed and 2 passed; the 2 that passed are the
sequential controls.
- The stand-in and the real engine alike lost parts: `every concurrently
sent chunk is in the record: expected [ 1 ] to deeply equal [ +0, 1 ]`
(2 concurrent), and `expected [ +0 ] to deeply equal [ +0, 1, 2, 3, 4,
5, 6, 7 ]` (8 concurrent).
- The parallel upload's completion was refused `409` with
`missingChunks: [1]`.
- With a competing write landing between the read and the write, the
door answered `{"success":true,…}` (`expected 200 to be 409`), and the
record held `[ +0 ]`: the competitor's part was erased.
**After**, at `006344eb50`: 24 passed (24). Both stores pass the 2 and 8
concurrent pins (every part, every eTag and size, `uploaded_chunks`,
`uploaded_size`, and the progress door), the parallel completion `200`
with the whole file, the sequential control (the same answers and
record, one session write per chunk), and the exhaustion pin (`409
RESOURCE_CONFLICT` after exactly 16 reads, with every competitor's part
kept). The store-level pins also pass on both stores: the write lands on
an unchanged row, misses once any one progress column moved, and misses
on a gone row. On the real engine, the call shape is pinned (dispatch
verdict `multi`, the same `{ tenantId, isSystem }` context), another
organization's row is not reached, a non-count answer is refused loudly,
and an unreachable row answers `500` once.
**Over HTTP**, a real boot (`bootStack`, sqlite-wasm):
`packages/qa/dogfood/test/storage-chunked-parallel-parts.dogfood.test.ts`
passes 2 of 2 (2 concurrent PUTs, then progress and completion `200`; 8
concurrent PUTs, then progress). With the sibling
`storage-chunked-resume-integrity.dogfood.test.ts`, 6 passed (6) at
`006344eb50`.
## Ablations
Each mutation went through `scripts/ablation-replace.mjs`: the anchor
hit 1 to 0, the blob changed, and the restore was proven with blob ==
HEAD and an empty `git diff HEAD`. The runner also carried its own
`EXIT`/`INT`/`TERM` restore over absolute paths. The control run with no
mutation passed 24 of 24.
| mutation | pins turned red |
| :--- | :--- |
| M1: the stand-in's comparison removed | 7, all stand-in: 2 and 8
concurrent, the parallel completion, exhaustion, and 3 column-moved pins
|
| M2: the engine guard replaced by an unchanging column | 8, all engine:
2 and 8 concurrent, the parallel completion, exhaustion, 3 column-moved
pins, and the where-shape pin |
| M3: the door treats a lost write as landed | 9: on both stores, 2 and
8 concurrent, the parallel completion and exhaustion; plus the
unreachable-row `500` |
| M4: exhaustion falls through to `200` | 2: exhaustion, on both stores
|
| M5: a non-count engine answer accepted | 1 |
| M6: the unreachable-row check removed | 1 |
| M7: the stand-in lands on a missing row | 1 |
| M8: the organization scope dropped from the conditional write | 2 |
The dogfood pin reads `@objectstack/service-storage` from `dist/`, so
its ablation included a rebuild.
- **Mutation leg:** M3, then a rebuild. `ablation-dist-preflight` found
the marker in `dist/index.js` and `dist/index.cjs`, and 2 of 2 failed
(the progress fell short of `uploadedChunks`).
- **Restore leg:** a rebuild, after which the marker was absent from all
6 built files and the tree was clean against HEAD. 2 of 2 passed.
## Tests and gates (HEAD `006344eb50`)
- `@objectstack/service-storage`: 46 files and 773 tests passed.
`typecheck` passed (`tsc`, the scripts project, and
`check:test-typecheck: OK`). `--listFiles` shows the new test file is in
the test-layer program.
- `@objectstack/dogfood`: `typecheck` passed.
- Two fake engines now answer a declared predicate update with its
matched-row count, as the real engine does:
`tenant-audit-update-delete-half-repairs.test.ts` and
`storage-routes.metadata-outage.test.ts`. The chunk-door pin in the
first now expects the conditional `where` with `multi: true`, under the
same context.
- `check:tenant-audit-census`: the new conditional write is one more
engine write call site (236 to 237). The census was regenerated (`node
scripts/tenant-audit-census.mjs --write`), and the page's hand-written
prose figures were moved with it. The gate and its self-test pass.
- Lint, narrowed and stated as a measurement. ① Population: the 6
touched TypeScript files, all inside the `eslint.config.mjs` glob
`**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}`. ② `--format json` linted 6 files
with 0 errors and 0 warnings. ③ Invariance: the config sets no
`parserOptions.project` (`--print-config` gives
`{"ecmaVersion":"latest","sourceType":"module"}`), so linting is not
type-aware and this diff cannot move a verdict on an untouched file. The
repo-wide `pnpm lint` is CI's.
- The derived gate families (`dispatch-gates --commands`, derived at
`006344eb50`) were run after the last commit, with exit codes captured
before any pipe. `dispatch-gates --ran` reads `97 derived, 97 run, 0
NOT-MEASURED, 0 UNRUN` (a derived zero: every family recorded `exit 0`).
`pnpm check:error-status-conformance` was run by hand and exited 0: `✓
every derivable runtime status is documented, and every documented
status is reachable.` A few verdict lines:
- `check-engine-double-contract: OK — 983 pinned, 129 in the DEBT
ledger, 3 exempt.`
- `check-nul-bytes: OK (scanned 10310 text file(s) …; no raw ASCII
control bytes).`
- `✓ check-tenant-audit-census: OK -- 237 write call sites certified …`
- `check-test-source-alias OK — 73 packages with tests scanned …`
- `✓ check:dual-build-cjs-loads — 106 published require entry point(s)
across 66 package(s) load …`. Its first run answered `PREREQUISITE NOT
MET` (no `dist/` for 8 packages), which is not a measurement. It passed
after a full cache-backed `turbo run build`.
- ✓ This diff introduces no `major` bump. (`check-changeset-no-major`)
## Acceptance notes
- **Clause-② stays `no`, as the claim declared.** No accept set, export
or public signature moves. The one new answer, `409` after 16 lost
record writes, replaces a `200` that had recorded nothing.
- **The progress write is now a predicate update.** On an engine-backed
store, the chunk door's write publishes the bulk `data.records.updated`
event (a count) where it published `data.record.updated`, and its
after-hooks go through the bulk per-row path. Nothing in the tree
registers an update hook on `sys_upload_session` or subscribes to its
record events, and the config-change audit excludes the object.
- **Observation, read-only and not measured, not filed** (承接者:无). The
chunk door's upload-limit check judges the record as it was read.
Concurrent PUTs can therefore each pass it against the same stale total,
and the backend can briefly hold more bytes than `maxUploadBytes`. The
completion door still refuses any upload whose held bytes differ from
its declared size, and the init door bounds that size.
---
_Generated by [Claude
Code](https://claude.ai/code/session_01WkL6Eijt432S1Y7ekb6ovQ)_
---------
Co-authored-by: Claude <noreply@anthropic.com>
1 parent c6fc938 commit 54c3ce1
9 files changed
Lines changed: 912 additions & 46 deletions
File tree
- .changeset
- content/docs/permissions
- docs/audits
- packages
- qa/dogfood/test
- services/service-storage/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
86 | | - | |
| 86 | + | |
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| |||
122 | 122 | | |
123 | 123 | | |
124 | 124 | | |
125 | | - | |
| 125 | + | |
126 | 126 | | |
127 | 127 | | |
128 | 128 | | |
| |||
150 | 150 | | |
151 | 151 | | |
152 | 152 | | |
153 | | - | |
154 | | - | |
| 153 | + | |
| 154 | + | |
155 | 155 | | |
156 | 156 | | |
157 | 157 | | |
| |||
187 | 187 | | |
188 | 188 | | |
189 | 189 | | |
190 | | - | |
191 | | - | |
192 | | - | |
193 | | - | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
194 | 194 | | |
195 | 195 | | |
196 | 196 | | |
| |||
200 | 200 | | |
201 | 201 | | |
202 | 202 | | |
203 | | - | |
| 203 | + | |
204 | 204 | | |
205 | 205 | | |
206 | 206 | | |
207 | 207 | | |
208 | 208 | | |
209 | 209 | | |
210 | | - | |
| 210 | + | |
211 | 211 | | |
212 | 212 | | |
213 | 213 | | |
214 | | - | |
| 214 | + | |
215 | 215 | | |
216 | 216 | | |
217 | | - | |
| 217 | + | |
218 | 218 | | |
219 | 219 | | |
220 | 220 | | |
| |||
223 | 223 | | |
224 | 224 | | |
225 | 225 | | |
226 | | - | |
227 | | - | |
| 226 | + | |
| 227 | + | |
228 | 228 | | |
229 | | - | |
| 229 | + | |
230 | 230 | | |
231 | 231 | | |
232 | 232 | | |
233 | 233 | | |
234 | | - | |
235 | | - | |
| 234 | + | |
| 235 | + | |
236 | 236 | | |
237 | 237 | | |
238 | | - | |
| 238 | + | |
239 | 239 | | |
240 | 240 | | |
241 | 241 | | |
242 | 242 | | |
243 | | - | |
| 243 | + | |
244 | 244 | | |
245 | 245 | | |
246 | 246 | | |
247 | | - | |
| 247 | + | |
248 | 248 | | |
249 | 249 | | |
250 | 250 | | |
| |||
297 | 297 | | |
298 | 298 | | |
299 | 299 | | |
300 | | - | |
| 300 | + | |
301 | 301 | | |
302 | 302 | | |
303 | 303 | | |
304 | | - | |
| 304 | + | |
305 | 305 | | |
306 | 306 | | |
307 | | - | |
| 307 | + | |
308 | 308 | | |
309 | 309 | | |
Lines changed: 10 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
37 | | - | |
| 36 | + | |
| 37 | + | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
40 | 40 | | |
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
44 | | - | |
45 | | - | |
| 44 | + | |
| 45 | + | |
46 | 46 | | |
47 | 47 | | |
48 | | - | |
| 48 | + | |
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
| |||
90 | 90 | | |
91 | 91 | | |
92 | 92 | | |
93 | | - | |
| 93 | + | |
94 | 94 | | |
95 | 95 | | |
96 | 96 | | |
97 | | - | |
| 97 | + | |
98 | 98 | | |
99 | 99 | | |
100 | | - | |
| 100 | + | |
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
| |||
255 | 255 | | |
256 | 256 | | |
257 | 257 | | |
258 | | - | |
| 258 | + | |
Lines changed: 117 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
0 commit comments