Repository navigation
Commit 51b43a8
ci(board-snapshot): upload only the records that changed to R2, not all ~41k objects every run (#22930)
## Why
The R2 step of `board-snapshot.yml` runs `aws s3 sync archive/board/
s3://$R2_BUCKET/objectstack/board/ --delete`. `aws s3 sync` uploads a
file when its size differs from the object's, or when the local mtime is
newer than the object's LastModified (aws-cli
`SizeAndLastModifiedSync.compare_time`). A fresh CI checkout gives every
file a brand-new mtime, so **every run re-PUTs all ~41,000 objects**.
Measured on the `objectstack-board` bucket: 1.2M PutObject in 11 days
(~110k a day at 4 runs a day). That is ~$10 a month of R2 Class A
operations, against a free tier of 1M a month.
## What changes (`.github/workflows/board-snapshot.yml` only)
1. **Archive checkout:** `git fetch --depth=1` becomes `git fetch
--filter=blob:none`. The archive branch's whole history is fetched
without file contents. The worktree checkout still downloads the tip's
blobs in one batch, the same content the depth-1 fetch downloaded, so
the history adds only commits and trees: ~7 MB for 127 commits over
41,047 files, measured today with a blobless fetch of `board-archive`.
2. **R2 step:** before `aws s3 sync`, a short inline Node script sets
each file's mtime to the committer time of the **last commit on the
archive branch that changed it**. It uses one `git log --no-renames
--name-only -z --format=%x01%ct HEAD -- board` pass over trees only
(~0.2 s for today's history). The sync command itself is unchanged: same
source, same `--delete`, same target.
Why that is correct:
- An object's LastModified is the time it was last uploaded. Every
upload happens after the commit whose content it carries, because the
commit step runs before the R2 step. So a current object's LastModified
is newer than its file's restored mtime, and sync skips it.
- A file changed by a commit made after its object's last upload has the
newer mtime and is uploaded. That covers this run's changes **and
anything an earlier failed run did not upload**, with no state kept
between runs.
- Files removed from the archive are still deleted by `--delete`.
- `--size-only` was rejected, as required: an `updated_at` move keeps a
record's byte count.
- A truncated history (if `--depth` ever came back) only errs in the
safe direction. The boundary commit looks like the creator of every
older file, which stamps those files newer than their real change, so
they are re-uploaded and no stale object is skipped.
- If the history cannot be read at all, the step leaves the checkout
mtimes in place. That is the old full re-upload, announced in a
`::warning::` and in the run's R2 status line. It costs more Class A
operations and never gives a weaker copy.
Unchanged: the 4×/day schedule, the daily `snapshots/YYYY-MM-DD.tar.gz`
(its members now carry the restored mtimes), credentials read but never
echoed, unset secrets as a notice, a configured upload that fails goes
red, and the `--delete` target is never widened. Comments that the
change made untrue were updated: the "every later run moves a handful"
note was false before this PR and is true now. A new "Only changed
records are uploaded" section documents the mechanism and its two
assumptions:
- only this workflow writes the archive branch, so committer times
follow real time;
- nothing else writes under the mirror prefix. An out-of-band same-size
write with a newer LastModified would no longer be overwritten within
six hours.
## Validation
The R2 step cannot run in a PR (skipped on `pull_request`). So I ran the
**checkout, commit/push and R2 steps verbatim**, extracted from this
workflow file, against a throwaway git server (`uploadpack.allowFilter`)
and a fake `aws` that applies aws-cli's size+mtime rule to a JSON bucket
and records content hashes. After every successful run it checks that
the mirror equals the archive byte for byte:
| run | scenario | uploaded | result |
|---|---|---|---|
| baseline | old behaviour, fresh checkout | 60 of 60 | — |
| 1 | 2 edits (one same-size), 1 new, 1 deleted | exactly the 3 changed;
1 deleted | mirror == archive |
| 2 | upload fails | — | step red, as before |
| 3 | 1 more edit | run 2's 2 misses + its own 1 | mirror == archive |
| 4 | no change | 0 | mirror == archive |
| 5 | upload fails again | — | step red |
| 6 | **depth=1 history**, no change, R2 behind | 60 (over-upload, incl.
run 5's miss) | mirror == archive |
The push from the blobless clone landed every run's commit on the
server. **Ablation:** with the mtime restore replaced by `false` (one
line), runs 1, 3 and 4 each upload all 60 objects and the harness
reports 3 failures, so the test detects the old behaviour.
`actionlint` 1.7.12 with shellcheck 0.11.0 reports exactly the one
finding untouched `main` already has (SC2086 on the intentionally
unquoted `$SNAPSHOT_ARGS` in the snapshot step). The new script is
clean. The repo's workflow-scanning gates were run locally on this
branch and all exit 0: `check:pm-board-snapshot`,
`check:pm-dispatch-gates`, `check:pm-expected-skips`,
`check:node-version`, `check:pnpm-acquisition`,
`check:workflow-step-name-quoting`, `check:workflow-status-functions`,
`check:required-contexts`, `check:stall-guard-budget`,
`check:shard-attestation`, `check:agent-test-spelling`,
`check:pnpm-filter-targets`, `check-self-test-wired`,
`check-step-collectors`, `check-whole-set-label-write`,
`check-aggregator-roster` and `check-ci-filter-parity`.
## Expected effect
Per run: ~41,000 PutObject goes down to the records that changed since
the last upload, plus the ~41 ListObjectsV2 pages sync already reads.
The archive's last five commits touched 546 distinct files, so that is
roughly a hundred per run, or ~13k a month against the 1M free tier. The
first run after merge should log `mtimes from history: <n> file(s)
stamped from <m> commit(s)` and a small `synced N object(s)`, which is
the post-merge check.
## Notes
- Searched the open issues and PRs for this (`board-snapshot R2`, `s3
sync`) and found nothing. Earlier R2-mirror fixes in this file: #18049,
#19536.
- Releases nothing from any published package, so `skip-changeset`.
Clause-②: no
---
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude <noreply@anthropic.com>1 parent 4426d9d commit 51b43a8
1 file changed
Lines changed: 108 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
217 | 217 | | |
218 | 218 | | |
219 | 219 | | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
220 | 230 | | |
221 | 231 | | |
222 | 232 | | |
223 | 233 | | |
224 | | - | |
| 234 | + | |
225 | 235 | | |
226 | 236 | | |
227 | 237 | | |
| |||
319 | 329 | | |
320 | 330 | | |
321 | 331 | | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
322 | 369 | | |
323 | 370 | | |
324 | 371 | | |
| |||
378 | 425 | | |
379 | 426 | | |
380 | 427 | | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
381 | 481 | | |
382 | 482 | | |
383 | 483 | | |
| |||
392 | 492 | | |
393 | 493 | | |
394 | 494 | | |
395 | | - | |
396 | | - | |
397 | | - | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
398 | 500 | | |
399 | 501 | | |
400 | 502 | | |
| |||
414 | 516 | | |
415 | 517 | | |
416 | 518 | | |
417 | | - | |
418 | | - | |
| 519 | + | |
| 520 | + | |
419 | 521 | | |
420 | 522 | | |
421 | 523 | | |
| |||
0 commit comments