E3: ccrawl crawl run, wire up the frontier that already compiles

[tamnd]

Spec: notes/Spec/2115/07-enhancements.md, item E3.

Problem

Frontier, RobotsCache, CrawlURL, ExtractOutLinks, NormalizeURL and WriteWARCResponse all exist and compile, and the only command that touches any of it is crawl fetch with a single url. The guide describing how they compose already ships. The parts are there, the composition is not.

What to do

Add one command:

ccrawl crawl run --seeds seeds.jsonl --out ./warc --workers 64 \
                 --delay 1s --max-depth 2 --max-pages 1000000 \
                 --state ./crawl.db --robots

It reads SeedRecords from crawl seed, drives the Frontier, enforces the RobotsCache and per host politeness, dispatches through ami's FetchBatch (already a dependency and already used by refetch, so it already handles the concurrency), writes WARC through WriteWARCResponse, expands the frontier from ExtractOutLinks up to --max-depth, and checkpoints into --state. Error classification uses the same taxonomy as RefetchStats.

Done when

• a 100k page crawl from a seed file completes
• output validates under warcio check
• a Disallow on a test host is respected
• SIGKILL and restart resumes without refetching
• per host request spacing never goes under --delay

Estimate: 5 days. Needs #51 and #52 first, and E19 should be in before we point this at the open web at volume.