crawl: add crawl run, the loop over the frontier - #99
Conversation
The frontier, the robots cache and the WARC writer all existed and none of them were reachable from a command. This wires them together. ccrawl crawl run reads a seed file, walks the frontier in priority order, fetches one URL per host per delay, enforces robots.txt, writes every fetch to WARC, and puts the outlinks back in the queue up to --max-depth. The frontier lives in --state, so a killed run resumes on the remainder instead of starting over. Two things had to change underneath. The frontier clock counted seconds, so OpenFrontier truncated any delay under a second to zero and a crawler that asked for 250ms between requests got none; it counts milliseconds now. And RobotsCache.Fetch was not single flight, so eight workers landing on a new host fetched its robots.txt eight times. Two workers can also pop the same host before either has read its robots.txt, and then both learn the host wants ten seconds. Frontier.HoldClaim decides which of them owns the host slot, and the other puts its URL back with Defer, which is a deferral rather than a retry because nothing was fetched and nothing failed.
tamnd
added
priority: P2
The frontier claimed a batch, handed out one row, then wrote the rest back on every pass, and every idle worker ran a full refill with two write transactions per poll. On a two host shape that came out at 1.7 pages per second. The refill now joins hosts and takes at most one row per host per batch, so a claimed row is a row a worker can actually fetch, an empty refill parks the pollers for one delay tick, and the completion buffer only flushes when it has reached SyncEvery. Spacing the pops is not the same as spacing the requests, and a server measuring the gap between arrivals saw pairs land on top of each other. The crawler now holds its own per host clock, waits on it before it dispatches, and restamps it from httptrace when the request bytes actually go out.
|
Evidence for the done when boxes, measured rather than argued. 100k pages. Local harness, 200 hosts on 127.0.0.1:8801-9000, each page 1.5KB of padded HTML, warcio. Disallow. The harness serves Crawl-delay and per host spacing. A sequential control run of 300 pages against one host with a 10ms delay measures a minimum server side gap of 10.65ms and never goes under. With 64 workers the same delay gives a minimum of 8.72ms and a median of 10.58ms, and a 50ms delay gives a minimum of 48.25ms. The gap between those minima and the nominal delay is a constant 1.3ms to 1.8ms of handler scheduling jitter on the server at 2000 requests per second, which the control run shows is not the crawler letting two requests through early. SIGKILL and resume. 100,000 seeds, killed with The last commit is the reason those numbers exist. The frontier used to claim 512 rows, hand out one and write 511 back on every pass, with every idle worker running two write transactions per poll, which capped a two host crawl at 1.7 pages/s. The refill now joins |
1 participant
Closes #54.
Frontier,RobotsCache,CrawlURL,ExtractOutLinksand the WARC writer all existed and none of them were reachable from a command. This is the composition.It reads the JSONL
crawl seedwrites, or a plain list of URLs, or stdin. Seed priority is the harmonic centrality, so the central hosts go first. The frontier hands out at most one URL per host per delay, robots.txt is fetched once per host and enforced, every fetch is written to WARC, and outlinks go back in the queue up to--max-depth. The error breakdown uses the same buckets asRefetchStats.One deliberate deviation from the issue. The issue says to dispatch through ami's
FetchBatch.FetchBatchhands back a body without the request and response header blocks or the remote address that a WARC record needs, and the frontier already governs both concurrency and politeness, so a second scheduler on top of it would only fight it. The loop is a worker pool overFrontier.PopandCrawlURLinstead. Everything else in the issue is as written.Two fixes underneath.
The frontier clock counted seconds.
OpenFrontierdiddelay: int64(cfg.Delay / time.Second), so any delay under a second truncated to zero and a crawler asking for 250ms between requests to a host got none. It counts milliseconds now, callers and tests included.RobotsCache.Fetchwas not single flight, so eight workers arriving at a new host at once fetched its robots.txt eight times. It is single flight per host now, and cancellable while waiting.New frontier methods. Two workers can pop the same host before either has read its robots.txt, and then both learn the host wants ten seconds and only one can have the slot.
HoldClaimcompares the reservationPopmade against the host clock and tells the worker that was overtaken; that worker puts its URL back withDefer, which is a deferral rather than a retry because nothing was fetched and nothing failed.Done when
warcio check--delayEvidence for the live 100k run is in a comment below. The other four are covered by
ccrawl/crawlrun_test.go, which crawls a realhttptestserver over the real HTTP client:TestCrawlRunStaysOffDisallowedPaths,TestCrawlRunHonoursCrawlDelay,TestCrawlRunResumesWithoutRefetching,TestCrawlRunKeepsHostRequestsApart, plus retry, cancel, max-pages and depth cases.go test ./... -raceis green,golangci-lint run ./...reports 0 issues,scripts/docs-drift.shmatches the binary.Docs:
crawl rungets a section in the CLI reference, and the recrawl engine guide loses the paragraphs saying the loop does not exist yet.