E18: content deduplication, exact and near

[tamnd]

Spec: notes/Spec/2115/07-enhancements.md, item E18.

Problem

doc_id is the first 16 bytes of sha256 of the url, so two urls serving the same bytes give us two rows. CDX already carries a content digest that we parse and store and then never use. Near duplicate content, mirrors and syndicated news, is completely untouched.

What to do

• Add --dedup-digest to the pipelines, skipping records whose payload digest was already emitted in this run. Cheap, exact, and it should be the highest yield thing here.
• Add ccrawl/simhash.go, a 64 bit simhash over the extracted markdown, stored as a column. Keep near dup collapsing as a downstream query rather than a pipeline decision, that is the right place for it.
• Add ccrawl dedup <parquet-dir> to report duplicate clusters.

Done when

• --dedup-digest on one crawl shard drops the expected fraction, measure it first, with zero false drops on a manual sample
• the simhash column is present and stable across runs of the same input

Estimate: 3 days.