Bound the memory search spends on --at, --latest-only and --dedup

[tamnd]

Closes #55.

--at, --latest-only and --dedup each held a map keyed on every URL the query touched. A wildcard over a large domain across every crawl touches hundreds of millions of them, which is the most likely place ccrawl runs a machine out of memory.

What changed

A CDX response is sorted by urlkey and every capture of a URL sits inside one urlkey group, so the winner inside a crawl can be decided while the crawl is being read.

• --at reduces each urlkey group as it goes by, keeps the per crawl winners in sorted runs, and k-way merges the runs at the end.
• --latest-only writes the URLs it emitted for a crawl to a log in the same urlkey order, and checks the next crawl against every earlier log with a cursor that only moves forward.
• --dedup gets a bounded exact set instead, because payload digests arrive in no order and nothing about the index says otherwise. Past its ceiling it forgets the coldest digests and warns. That error lets a duplicate through rather than dropping a unique record, which is the direction that matters.

--at and --latest-only are exact at any result size. --max-buffer is the shared budget, 5,000,000 records by default, and it covers every crawl at once rather than each crawl on its own, which is the difference between the ceiling the flag names and a hundred times it. Past the budget the run writes to temporary files under TMPDIR, removes them however it exits, and warns once so the slowdown has an explanation.

One behaviour changes. --at sorts its result newest first, which needs the whole result in hand. It still does up to the buffer; past it the result goes out in index order and the command says so on stderr.

Real data

ccrawl search 'vi.wikipedia.org/wiki/*' -c CC-MAIN-2024-10,CC-MAIN-2023-14 -o jsonl was pulled live and replayed through both the old maps and the new code with a 1,000 record budget, so every disk path ran:

102434 records over 2 crawls
--at agrees with the map on 94747 URLs
--latest-only agrees with the map on 94747 records

That is TestCDXSpillAgainstRealIndexRecords, gated on CCRAWL_CDX_JSONL since the repository does not carry a 30 MB fixture.

Live against the index, at a budget small enough to force the temporary files, output matches the default budget exactly:

Query	Default	Small budget	Same records
--at 2021-06 over two crawls	240	240 at --max-buffer 100	yes
--latest-only over two crawls	240	240 at --max-buffer 50	yes
--dedup over two crawls	256	256 at --max-buffer 50	yes

Memory

TestCDXPickerScale and TestCDXURLLogScale, gated on CCRAWL_SCALE_CDX, stream synthetic records shaped like a domain wildcard across ten crawls and sample peak heap while each implementation runs. The replacements are held to a 100,000 record budget:

Records	Map peak heap	Replacement peak heap
2,000,000	100 MB	125 MB
20,000,000	555 MB	149 MB
100,000,000	2,484 MB	ran out of disk

The map grows with the result and the picker does not. Below a couple of million records the map is the cheaper of the two, which is why the default budget is 5M rather than something small.

The 100M run is the honest limit of this machine: the picker traded the memory for disk and wanted more than the 6.5 GB free, since spilling costs roughly one line per URL per crawl. The map got to 2.5 GB and was still climbing.

Not done here

ccrawl search '*.wikipedia.org' --at 20240101 -c all in bounded RSS is the first box on the issue and it did not complete today. index.commoncrawl.org is dropping pages, and a wide query fails the same way on both binaries:

Cdx page 3: unexpected EOF.

A single page failure aborting a query with no retry is worth its own issue. The box stays unticked until that query runs.

Checks

go test ./... -race, golangci-lint run ./... and scripts/docs-drift.sh are clean. --max-buffer is in the CLI reference with the ordering caveat and the note that the --at sort gets the same budget again.