Skip to content

Commit 51b43a8

Browse files
hotlongclaude
andauthored
ci(board-snapshot): upload only the records that changed to R2, not all ~41k objects every run (#22930)
## Why The R2 step of `board-snapshot.yml` runs `aws s3 sync archive/board/ s3://$R2_BUCKET/objectstack/board/ --delete`. `aws s3 sync` uploads a file when its size differs from the object's, or when the local mtime is newer than the object's LastModified (aws-cli `SizeAndLastModifiedSync.compare_time`). A fresh CI checkout gives every file a brand-new mtime, so **every run re-PUTs all ~41,000 objects**. Measured on the `objectstack-board` bucket: 1.2M PutObject in 11 days (~110k a day at 4 runs a day). That is ~$10 a month of R2 Class A operations, against a free tier of 1M a month. ## What changes (`.github/workflows/board-snapshot.yml` only) 1. **Archive checkout:** `git fetch --depth=1` becomes `git fetch --filter=blob:none`. The archive branch's whole history is fetched without file contents. The worktree checkout still downloads the tip's blobs in one batch, the same content the depth-1 fetch downloaded, so the history adds only commits and trees: ~7 MB for 127 commits over 41,047 files, measured today with a blobless fetch of `board-archive`. 2. **R2 step:** before `aws s3 sync`, a short inline Node script sets each file's mtime to the committer time of the **last commit on the archive branch that changed it**. It uses one `git log --no-renames --name-only -z --format=%x01%ct HEAD -- board` pass over trees only (~0.2 s for today's history). The sync command itself is unchanged: same source, same `--delete`, same target. Why that is correct: - An object's LastModified is the time it was last uploaded. Every upload happens after the commit whose content it carries, because the commit step runs before the R2 step. So a current object's LastModified is newer than its file's restored mtime, and sync skips it. - A file changed by a commit made after its object's last upload has the newer mtime and is uploaded. That covers this run's changes **and anything an earlier failed run did not upload**, with no state kept between runs. - Files removed from the archive are still deleted by `--delete`. - `--size-only` was rejected, as required: an `updated_at` move keeps a record's byte count. - A truncated history (if `--depth` ever came back) only errs in the safe direction. The boundary commit looks like the creator of every older file, which stamps those files newer than their real change, so they are re-uploaded and no stale object is skipped. - If the history cannot be read at all, the step leaves the checkout mtimes in place. That is the old full re-upload, announced in a `::warning::` and in the run's R2 status line. It costs more Class A operations and never gives a weaker copy. Unchanged: the 4×/day schedule, the daily `snapshots/YYYY-MM-DD.tar.gz` (its members now carry the restored mtimes), credentials read but never echoed, unset secrets as a notice, a configured upload that fails goes red, and the `--delete` target is never widened. Comments that the change made untrue were updated: the "every later run moves a handful" note was false before this PR and is true now. A new "Only changed records are uploaded" section documents the mechanism and its two assumptions: - only this workflow writes the archive branch, so committer times follow real time; - nothing else writes under the mirror prefix. An out-of-band same-size write with a newer LastModified would no longer be overwritten within six hours. ## Validation The R2 step cannot run in a PR (skipped on `pull_request`). So I ran the **checkout, commit/push and R2 steps verbatim**, extracted from this workflow file, against a throwaway git server (`uploadpack.allowFilter`) and a fake `aws` that applies aws-cli's size+mtime rule to a JSON bucket and records content hashes. After every successful run it checks that the mirror equals the archive byte for byte: | run | scenario | uploaded | result | |---|---|---|---| | baseline | old behaviour, fresh checkout | 60 of 60 | — | | 1 | 2 edits (one same-size), 1 new, 1 deleted | exactly the 3 changed; 1 deleted | mirror == archive | | 2 | upload fails | — | step red, as before | | 3 | 1 more edit | run 2's 2 misses + its own 1 | mirror == archive | | 4 | no change | 0 | mirror == archive | | 5 | upload fails again | — | step red | | 6 | **depth=1 history**, no change, R2 behind | 60 (over-upload, incl. run 5's miss) | mirror == archive | The push from the blobless clone landed every run's commit on the server. **Ablation:** with the mtime restore replaced by `false` (one line), runs 1, 3 and 4 each upload all 60 objects and the harness reports 3 failures, so the test detects the old behaviour. `actionlint` 1.7.12 with shellcheck 0.11.0 reports exactly the one finding untouched `main` already has (SC2086 on the intentionally unquoted `$SNAPSHOT_ARGS` in the snapshot step). The new script is clean. The repo's workflow-scanning gates were run locally on this branch and all exit 0: `check:pm-board-snapshot`, `check:pm-dispatch-gates`, `check:pm-expected-skips`, `check:node-version`, `check:pnpm-acquisition`, `check:workflow-step-name-quoting`, `check:workflow-status-functions`, `check:required-contexts`, `check:stall-guard-budget`, `check:shard-attestation`, `check:agent-test-spelling`, `check:pnpm-filter-targets`, `check-self-test-wired`, `check-step-collectors`, `check-whole-set-label-write`, `check-aggregator-roster` and `check-ci-filter-parity`. ## Expected effect Per run: ~41,000 PutObject goes down to the records that changed since the last upload, plus the ~41 ListObjectsV2 pages sync already reads. The archive's last five commits touched 546 distinct files, so that is roughly a hundred per run, or ~13k a month against the 1M free tier. The first run after merge should log `mtimes from history: <n> file(s) stamped from <m> commit(s)` and a small `synced N object(s)`, which is the post-merge check. ## Notes - Searched the open issues and PRs for this (`board-snapshot R2`, `s3 sync`) and found nothing. Earlier R2-mirror fixes in this file: #18049, #19536. - Releases nothing from any published package, so `skip-changeset`. Clause-②: no --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4426d9d commit 51b43a8

1 file changed

Lines changed: 108 additions & 6 deletions

File tree

‎.github/workflows/board-snapshot.yml‎

Lines changed: 108 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -217,11 +217,21 @@ jobs:
217217
# The archive branch is an ORPHAN: it shares no history with `main`, so a
218218
# checkout of it carries the archive and nothing else. First run creates it
219219
# locally; the push below is what publishes it.
220+
#
221+
# It is fetched with its WHOLE history but without file contents
222+
# (`--filter=blob:none`), because the R2 step reads each record's
223+
# last-change time out of that history; see "Only changed records are
224+
# uploaded" there. The worktree checkout then fetches the tip's blobs in
225+
# one batch, which is the same content a `--depth=1` fetch downloaded, so
226+
# the history adds only commits and trees: ~7 MB for 127 commits over
227+
# 41,047 files, measured 2026-10-12. ⛔ Do not put `--depth` back: a
228+
# truncated history makes the R2 step re-upload every record older than
229+
# the cut on every run (safe, but the cost that step exists to avoid).
220230
- name: Check out or create the archive branch
221231
if: github.event_name != 'pull_request'
222232
run: |
223233
set -euo pipefail
224-
git fetch --depth=1 origin '+refs/heads/board-archive:refs/remotes/origin/board-archive' || true
234+
git fetch --filter=blob:none origin '+refs/heads/board-archive:refs/remotes/origin/board-archive' || true
225235
if git rev-parse --verify --quiet refs/remotes/origin/board-archive > /dev/null; then
226236
git worktree add archive -b board-archive refs/remotes/origin/board-archive
227237
echo "archive branch exists; checked out at $(git -C archive rev-parse --short HEAD)"
@@ -319,6 +329,43 @@ jobs:
319329
# rather than by the clock, so a missed 02:07 run, a re-run and a
320330
# `workflow_dispatch` all converge on one tarball per day.
321331
#
332+
# ## Only changed records are uploaded
333+
#
334+
# `aws s3 sync` uploads a file when its size differs from the object's,
335+
# or when the file's mtime is newer than the object's LastModified
336+
# (aws-cli `SizeAndLastModifiedSync`). A fresh checkout stamps every file
337+
# with the checkout time, so each run re-PUT all ~41,000 objects:
338+
# measured on the bucket, 1.2M PutObject in 11 days (~110k a day, ~$10 a
339+
# month of R2 Class A operations against a 1M-a-month free tier).
340+
#
341+
# So before syncing, every file's mtime is set to the committer time of
342+
# the last commit on the archive branch that changed it. That makes the
343+
# comparison mean what sync assumes it means:
344+
#
345+
# - an object's LastModified is the time it was last uploaded, and every
346+
# upload happens AFTER the commit whose content it carries (the commit
347+
# step runs before this one), so an object that is current has a
348+
# LastModified newer than its file's mtime and is skipped;
349+
# - a file changed by a commit made after its object was last uploaded
350+
# has the newer mtime and is uploaded. That covers this run's changes
351+
# AND anything an earlier run failed to upload, with no state kept
352+
# between runs;
353+
# - a file the archive no longer has is still removed by `--delete`.
354+
#
355+
# ⛔ Not `--size-only`: a record edit can keep its byte count (every
356+
# `updated_at` move does, and so does a count gaining no digit), and a
357+
# size-only comparison would leave the stale copy in R2 for good.
358+
#
359+
# This relies on two things being true. The archive branch is written
360+
# only by this workflow, so its committer times follow real time. And
361+
# nothing else writes under the mirror prefix: an out-of-band write
362+
# with the same size and a newer LastModified would not be overwritten,
363+
# where the old full re-upload replaced it within six hours. If the
364+
# history cannot be read, the step does not guess. It leaves the
365+
# checkout times in place, which re-uploads everything, and says so in
366+
# a warning and in the run's R2 status line: more Class A operations,
367+
# never a weaker copy.
368+
#
322369
# ## Unset secrets are a notice, not a failure
323370
#
324371
# The bucket and its scoped API token are the maintainer's to create.
@@ -378,6 +425,59 @@ jobs:
378425
# No `.git` filter is needed here: `archive/` is a linked worktree,
379426
# so its `.git` is a FILE at `archive/.git`, which is outside this
380427
# source (filters match paths relative to the source directory).
428+
#
429+
# mtimes first: see "Only changed records are uploaded" above. The
430+
# history is walked newest-first, so the first time a path appears
431+
# is the last commit that touched it; a path whose newest commit
432+
# deleted it has no file and is skipped. `-z` keeps every byte of a
433+
# path intact. A truncated history errs in the safe direction only:
434+
# its oldest commit looks like the one that created every file older
435+
# than it, which stamps those files NEWER than their real last
436+
# change, so they are uploaded again, and no stale object is ever
437+
# skipped.
438+
#
439+
# The script is written to a file rather than fed through a heredoc
440+
# inside the command substitution, which keeps bash's parser out of
441+
# the backticks and quotes it contains.
442+
cat > "$RUNNER_TEMP/archive-mtimes.cjs" <<'JS'
443+
const { execFileSync } = require('node:child_process');
444+
const { utimesSync } = require('node:fs');
445+
const { join } = require('node:path');
446+
const git = (...args) =>
447+
execFileSync('git', ['-C', 'archive', ...args], { encoding: 'utf8', maxBuffer: 1 << 30 });
448+
const log = git('log', '--no-renames', '--name-only', '-z', '--format=%x01%ct', 'HEAD', '--', 'board');
449+
const seen = new Set();
450+
let time = NaN;
451+
let stamped = 0;
452+
for (const token of log.split('\0')) {
453+
if (token.startsWith('\x01')) { time = Number(token.slice(1)); continue; }
454+
const path = token.replace(/^\n/, '');
455+
if (!path || seen.has(path)) continue;
456+
seen.add(path);
457+
if (!Number.isInteger(time)) throw new Error(`no commit time read before ${path}`);
458+
try {
459+
utimesSync(join('archive', path), time, time);
460+
stamped += 1;
461+
} catch (e) {
462+
if (e.code !== 'ENOENT') throw e;
463+
}
464+
}
465+
const tracked = git('ls-files', '-z', '--', 'board').split('\0').filter(Boolean).length;
466+
if (stamped !== tracked) throw new Error(`stamped ${stamped} file(s) but the archive tracks ${tracked}`);
467+
const commits = git('rev-list', '--count', 'HEAD').trim();
468+
console.log(`${stamped} file(s) stamped from ${commits} commit(s)`);
469+
JS
470+
if mtimes=$(node "$RUNNER_TEMP/archive-mtimes.cjs" 2>&1); then
471+
# The last line only: the status line below goes into
472+
# $GITHUB_OUTPUT, where a second line would break its format.
473+
mtime_note="mtimes from history: ${mtimes##*$'\n'}"
474+
echo "${mtime_note}"
475+
else
476+
printf '%s\n' "$mtimes" >&2
477+
echo "::warning::Could not set the archive's mtimes from its git history (output above), so this sync re-uploads every object. The copy stays complete, but every file costs a Class A operation."
478+
mtime_note="FULL re-upload, mtimes not restored"
479+
fi
480+
381481
# No pipe between the command and `$?` — see the snapshot step.
382482
set +e
383483
aws s3 sync archive/board/ "$mirror" \
@@ -392,9 +492,11 @@ jobs:
392492
echo "::error::aws s3 sync to R2 exited $code. This run IS archived on board-archive; the out-of-GitHub copy is now behind and every later run will stay behind until this is fixed."
393493
exit 1
394494
fi
395-
# One line per object moved. The first configured run moves the whole
396-
# board and every later run moves a handful, so the log is truncated
397-
# here and the counts below are taken over all of it.
495+
# One line per object moved. With the mtimes restored above, a run
496+
# moves the records that changed since the last upload; a run that
497+
# fell back (or the first one configured) moves the whole board, so
498+
# the log is truncated here and the counts below are taken over all
499+
# of it.
398500
transfers=$(wc -l < "$RUNNER_TEMP/r2-sync.log")
399501
head -n 50 "$RUNNER_TEMP/r2-sync.log"
400502
if [ "$transfers" -gt 50 ]; then
@@ -414,8 +516,8 @@ jobs:
414516
tarred="written for ${day}"
415517
fi
416518
417-
echo "status=synced ${uploaded} object(s), deleted ${deleted}; daily tarball ${tarred}" >> "$GITHUB_OUTPUT"
418-
echo "R2: synced ${uploaded} object(s), deleted ${deleted}; daily tarball ${tarred}"
519+
echo "status=synced ${uploaded} object(s), deleted ${deleted} (${mtime_note}); daily tarball ${tarred}" >> "$GITHUB_OUTPUT"
520+
echo "R2: synced ${uploaded} object(s), deleted ${deleted} (${mtime_note}); daily tarball ${tarred}"
419521
420522
- name: Publish the run to the summary
421523
if: always()

0 commit comments

Comments
 (0)