Fetch WARC records in coalesced ranged GETs

[tamnd]

Closes #48.

One record per HTTP request is fine for a few thousand locations and hopeless for a few million. --batch sorts the locations by file and offset, coalesces the ones that sit within --gap bytes of each other, and reads each run of them in a single ranged GET.

Each record is sliced back out of the coalesced span by its own offset and length and parsed on its own, so the batch path is byte identical to the one at a time path by construction rather than by resemblance.

Flags on ccrawl fetch

Flag	Description
--batch	Coalesce nearby records in the same WARC file into shared ranged GETs
--gap	Coalesce records at most this many bytes apart (default 1 MiB)
--max-span	Never read more than this in one GET (default 16 MiB)
--order	input or file: emit in the order given or the order on disk (default file)
--ledger	File of finished locations, to skip on a resume
--lookahead	Ranged GETs allowed to run ahead of the writer (default 64)

--dry-run groups the locations and reports what the run would ask for without asking for any of it, which is the free way to pick a gap.

Done when, checked against real Common Crawl

1M locations fetched with at least 3x fewer HTTP requests. A million real robotstxt locations from CC-MAIN-2026-30, 99997 distinct WARC files, median 10 records per file:

gap=4096     1000000 locations in 916205 requests, 1.1x fewer; 1.7 GB read for 1.6 GB of records, 1.1x amplification
gap=65536    1000000 locations in 612078 requests, 1.6x fewer; 10.1 GB read for 1.6 GB of records, 6.3x amplification
gap=262144   1000000 locations in 252748 requests, 4.0x fewer; 58.6 GB read for 1.6 GB of records, 36.1x amplification
gap=1048576  1000000 locations in 101238 requests, 9.9x fewer; 116.0 GB read for 1.6 GB of records, 71.5x amplification

9.9x at the default gap. The 71x byte amplification is the honest other half of the trade and it is in the docs, not hidden: a round trip to data.commoncrawl.org costs far more than a megabyte of transfer does. Measured on 486 real records packed into 20 files, both paths at 8 workers:

batch          20 requests, 29.2 MB read     6.6s
one at a time  486 requests, 864 KB read    97.4s

14.7x faster while reading 34x the bytes.

Record output is identical to the one at a time path. Those same 486 real records fetched both ways into two directories, diff -rq clean.

A SIGKILL mid run resumes without refetching. kill -9 after 3 seconds left 50 locations in the ledger and 50 files on disk. Rerunning the same command skipped those 50, fetched the remaining 436, and the finished directory diffs clean against the unbroken run.

Notes

• FileURL now passes an absolute URL through untouched, the same way resolvePartURL already treated a manifest entry. DataBaseURL is a const so a test cannot override it, and this is what lets the tests point the fetch path at a local server. No real Common Crawl path starts with a scheme, so it costs nothing.
• --order file streams under a hard --lookahead cap. --order input has to put back an ordering the grouping destroyed, so it holds finished records until their turn comes, and one slow group early in the input holds everything behind it. The docs say so rather than leaving it to be discovered.
• scripts/docs-drift.sh treated a third level prose heading as a reset to the whole page scope, so ### Batch mode under ## fetch reported every flag as missing. A third level heading now keeps the scope of the command above it.

Tests: grouping (span cap, unsorted input, duplicate records), batch output against FetchWARCRecord record for record with a request count assertion, both orderings, a resume with an exact request count, a single bad location among good ones, and the ledger itself.