fix(storage): buffer unrewindable bodies so S3 uploads work over plain HTTP

[xe-nvdk]

Summary

• Uploads with an unrewindable body failed against every plain-HTTP S3 endpoint. S3Backend.WriteReader sent bodies under 100 MiB with a raw PutObject; aws-sdk-go-v2 must re-read the body to sign the payload and compute the default request checksum, and its rewind-free alternative (trailing checksums) exists only over TLS — so an io.Pipe or HTTP request body failed client-side, before a byte was sent, with compute input header checksum failed, unseekable stream is not supported without TLS and trailing checksum. Broken callers: the tiering migrator's streaming copy (every daily file under 100 MiB — fix(tiering): gate migration on the primary writer in shared-storage mode #951's live run saw errors=1 every cycle), edge-sync hub receive (stage and promote), and a peer-replication pull into an S3-backed node. Over TLS these worked, which is why AWS deployments never noticed. The repo was on a post-checksum-default SDK from the start; born broken, not regressed.
• Fix: isSeekable mirrors the SDK's own probe (an io.Seeker whose Seek works). An unrewindable body ≤ 16 MiB is read exactly into a right-sized buffer (spoolExact: io.ReadFull + a one-byte EOF probe, ctx-checked) and sent through the unchanged PutObject branch; larger ones go through the existing SDK Uploader path wrapped in exactLengthReader (declared-length enforcement, per-read ctx check, the source's own errors pass through verbatim so edge sync's errors.Is(err, errShortBody) still holds). Rewindable bodies (ingest flushes, compaction output, backups, the delete API's rewritten file) take exactly the path they took before.
• Consequences: a body whose length differs from the declared size now fails with storage.ErrBodyLength and commits nothing (single part: no request; multipart: aborted) — before it was truncated or rejected depending on TLS; every request body is now rewindable, so transient errors on these uploads are retried instead of failing with failed to rewind transport stream. The migrator sizes its copy from StatFile on the source rather than the metadata row, and prefers ErrBodyLength over the closed-pipe error it induces on the reading side.
• Why not "route everything through the Uploader": its part pool releases slices when no upload overlaps, so every 1 KiB migration would allocate and discard a fresh 16 MiB (twice per edge-sync receive). Spooling is proportional to the file.
• Found on the way, not fixed here: tiered_storage.cold.s3_storage_class (default GLACIER) is never applied — S3Config has no such field — so cold objects land in STANDARD. Honoring it would make cold data unreadable by DuckDB without a restore path; design decision, raised separately.

Design + matrix + adversarial-review history: docs/progress/2026-09-28-s3-unseekable-put-over-http.md (untracked).

Test plan

• go build, gofmt -l empty, go vet on internal/storage, internal/tiering, internal/api
• go test -race ./internal/storage/ ./internal/tiering/ green
• 9 new untagged tests against an httptest S3 stub (the SDK's failure is client-side, so the stub reproduces it): single PutObject, exactly-one-part spooled, 20 MiB multipart, short and long bodies on both paths (nothing committed, multipart aborted), inner-error passthrough on both paths, seek-probe routing, cancelled context, seekable path unchanged. Pre-fix: the two unseekable-success tests fail on the unmodified tree with the SDK error (recorded before implementing).
• //go:build objectstore contract test against real SeaweedFS 4.47 over plain HTTP (the configuration CI runs, .github/workflows/ci.yml:120-124): 1 KiB single PutObject and 20 MiB multipart with CRC32 part headers, read back byte-equal — the multipart path over plain HTTP had never been observed against a real store before this. Whole tagged suite green locally.
• Live: fix(tiering): gate migration on the primary writer in shared-storage mode #951's shared-storage rig (3 writers + reader, SeaweedFS over http://, cold.s3_prefix=archive/, */1 schedule) built from this branch. A daily file seeded in the shared hot bucket: on the next tick the leader logged File migrated successfully … size_bytes=1072 … to=cold and Migration cycle completed errors=0 migrated=1 — the migrator's own streaming PutObject, which fix(tiering): gate migration on the primary writer in shared-storage mode #951's run could only stage by hand. The object landed at archive/tier_smoke/cpu/1970/01/20/cpu_19700120_daily.parquet in the cold bucket, the hot copy is gone, and the reader (which never migrates) learned the cold row on its next sync with migrated_at = the object's timestamp (18:44:00Z) and answers SELECT COUNT(*) with hot + cold rows (10 = 5 + 5).
• Post-implementation deep review (matrix + diff): no Blockers, no Highs; every matrix row confirmed by trace. Its four Mediums are in this branch: the one-byte EOF probe uses io.ReadFull so a spurious (0, nil) cannot pass as end-of-stream; the spool path no longer logs (every caller does); a length mismatch is neither counted nor logged as a storage error on the multipart path either; the migration record and log carry the stat'd size. A 32 MiB (exact part multiple) case was added to the multipart test.