crawl: write WARC to ISO 28500

[tamnd]

Closes #53.

The crawl side had a WARC writer in name only: a response record with five headers, no digests, no request record, no warcinfo, a Content-Length copied off the wire, and no command that could reach it. It was WARC shaped and nothing else.

What is here

ccrawl/warcwrite.go holds a WARCWriter that writes what the standard asks for.

• Each file opens with a warcinfo record naming the tool and the exact command, and every record in the file carries that record's ID in WARC-Warcinfo-ID.
• A fetch becomes a request and a response record, linked both ways with WARC-Concurrent-To and written as one unit, so a rotation never splits a pair across files.
• Every record carries a WARC-Block-Digest and every response also a WARC-Payload-Digest over the HTTP body, both sha1: and base32.
• WARC-IP-Address comes from an httptrace hook, which is the only place the remote address is observable, and the last connection in a redirect chain is the one that ends up in the record.
• The 10 MB body cap now flags the record with WARC-Truncated: length rather than storing a short page as if it were the whole thing.
• Rotation at a size target, one gzip member per record, and a file takes the first sequence number the directory does not already hold, so fetching into the same directory twice adds files rather than overwriting them.

CrawlURL keeps the request and response header blocks, the remote address and the truncation flag. Those headers are rebuilt rather than captured, because net/http hands back a decoded body and a parsed header and never the wire bytes. That is what makes the header rewriting mandatory: Content-Length is always recomputed against the stored body, Transfer-Encoding is always dropped since Go dechunks on the way in, and Content-Encoding is dropped when we decoded it. The record has to describe the body stored with it, and it does.

crawl fetch --warc-dir makes it reachable from the CLI and reports the path it wrote.

Checked against a tool that did not write it

warcio, ten live sites, one rotating writer:

3 files, 20 records, warcio check exit 0, every digest pass
10 complete request and response pairs
0 records whose partner is in another file
0 records carrying the wrong warcinfo id

Ten separate CLI fetches, one file each, all ten pass warcio check with both digests verified per file. Reading them back with warcio and recomputing gives 10 response payloads verified, 0 mismatched, where verified means the payload digest matches a fresh sha1 of the body and the stored Content-Length equals the stored body length. That set deliberately includes a gzipped response and a chunked one, which are the two cases the old writer could not have got right.

Records round trip through ccrawl parse: all three types come back with their URLs, statuses and payload digests intact.

TestWARCWriterAgainstRealSites is the live check, skipped unless CCRAWL_WARC_OUT points somewhere, in the same shape as the robots corpus test.

Removed

NewWARCRecord and WriteWARCResponse, with their two test call sites and the docs bullet that advertised them.

Gates

go test ./... green, golangci-lint run ./... 0 issues, scripts/docs-drift.sh clean.