E18: content deduplication, exact and near

[tamnd]

Closes #50.

Two different problems live under one word, and this splits them.

Exact: --dedup-digest

Skips a record whose payload digest was already seen. The check runs before extraction, so a duplicate costs a hash lookup instead of an HTML parse, and export reuses the digest the WARC already carries rather than recomputing one.

The scope is one shard, not the whole run. Shards are converted in parallel, so a run wide set would make the choice of which copy survives depend on scheduling, and over --shards all the set would grow without a bound anyone set.

Refetch does the check in the single writer loop, using the digest ami already computes, and exempts failed fetches. Two dead hosts share an empty body and collapsing them would hide the failure.

Near: the simhash column and ccrawl dedup

Every row now carries a 64 bit fingerprint over overlapping three word shingles of the Markdown. It is stored, not acted on. Dropping near duplicates during a run means deciding once and irreversibly which copy is the good one, and that decision belongs to whoever knows what the dataset is for.

ccrawl dedup <parquet|dir>... reads it back and reports, never rewriting the input, so it is safe to point at a published dataset. It reads three columns and leaves the rest of the parquet alone, which is why a 20k row shard takes 0.68s.

parquet-go reads by name, so appending the column is backwards compatible and a file written before it existed reads as zero. ccrawl dedup fingerprints those rows on the fly, so older datasets still work.

Also in here

stats.Rows in refetch was incremented before the dedup and lang filters rather than after them, so the number the dataset card publishes did not match the file. The test caught it as "stats say 6 rows, parquet holds 4". This was already wrong for --lang before this change.

dedupSummary was rewritten for the same reason: its seen = dropped + kept arithmetic assumed the pre-filter count.

Done when

• --dedup-digest on one crawl shard drops the expected fraction, measure it first, with zero false drops on a manual sample
• the simhash column is present and stable across runs of the same input

Measured on CC-MAIN-2026-30 shard 0, unfiltered first and then with the flag.

Run	Rows	digest_dropped	md bytes	Parquet
without --dedup-digest	20861		96399420	49701703
with --dedup-digest	20819	83	96336958	48340832

Rather than a manual sample I scanned the raw WARC independently: 21278 HTML response records holding 21195 distinct payloads, so 83 duplicate payloads in 28 groups. That is the pipeline's number exactly, over the whole shard rather than a sample, with zero false drops. Only 42 of the 83 show up as fewer rows because the other 41 extract to nothing and would never have been rows.

One pair looked like a false drop at first, two URLs on btg-bestellservice.de whose Markdown differed. It is a true duplicate: Convert(body, pageURL) resolves relative links against the page URL, so byte identical HTML at two URLs produces different Markdown. Payload identity and text identity are not the same question.

For box two, two independent full exports of the same shard share 20819 doc_ids with 0 simhash mismatches, and only 2 rows in 20861 have no fingerprint. A unit test separately recomputes Simhash(markdown) from the stored text and checks it equals the stored column.

The mojibake finding

Worth recording because it changed the implementation. Charikar's simhash weights each feature by how often it occurs, and on web text that is a trap. A mojibake page, one where UTF-8 was served as Latin-1, extracts to a handful of replacement sequences repeated thousands of times, and a count weighted fingerprint is then decided almost entirely by those few features.

Two such pages measured here shared 6.5 percent of their shingles and still came out one bit apart, which chained 68 unrelated documents into a single cluster. The corpus was fine, the fingerprint was wrong.

Voting once per distinct shingle, however often it occurs, put that pair 30 bits apart and cut the shard from 67 near duplicates in 49 clusters to 26 in 25. A 512 byte floor on the near pass took it to 24, since a 64 bit fingerprint decided by fewer than 64 features is decided by noise. Short pages still get a fingerprint and still cluster as exact duplicates, which is the only claim worth making about a stub.

What survives looks like real template duplication: phpBB FAQ pages across two forums, WordPress tag archives on the same site, focus.de cancellation letter templates.

What it reports now

20,861 rows in 1 files
  exact duplicates          165 in 113 clusters, 139.7 kB
  near duplicates            24 in 23 clusters, 171.0 kB  (distance <= 3)
  redundant                 189  (0.9% of rows)
  no fingerprint              2  (too short to hash, left alone)

The largest cluster is 42 byte identical CONTENTdm digital collection pages.

Tests

ccrawl/dedup_test.go covers determinism over 50 repeats, near versus unrelated separation, the empty case, the digest drop through packStream, column presence and stability across two runs, the column matching text recomputed from the row, both cluster kinds with two uniques that must not be swept in, --distance 0, a file with no simhash column, empty input, and the 512 byte floor.

ccrawl/refetch_dedup_test.go runs the whole refetch path against an httptest server serving six pages of which three are identical, and asserts stats.Rows equals the parquet's content row count.

Fixtures generate prose from word banks rather than repeating one sentence, because strings.Repeat has too few distinct shingles to fingerprint once each shingle votes once. Sizes were measured, not guessed: at n=200 words the near pair sits at distance 0 and the unrelated pair at 36.

Gates

gofmt -l . clean, go vet ./... clean, go test ./... -count=1 green, golangci-lint run 0 issues, ./scripts/docs-drift.sh matches.