v0.10.1

Changelog

Other

• a6e0608: Check the requirements page against the crawl commands too (#123) (@tamnd)
• 664360e: Give a read that fails partway through a response exit 8 (#147) (@tamnd)
• 71d55f3: Give the WAT and WET records their JSON names (#126) (@tamnd)
• 5a0281b: Keep working from a cached crawl list when the index server is down (#139) (@tamnd)
• 6f39ddc: Leave out host counts nothing measured (#145) (@tamnd)
• 8c3a2b3: Let ccrawl version render as data (#151) (@tamnd)
• 5a1f832: Make --data-dir move the cache with it (#143) (@tamnd)
• c2007e0: Make crawls info an operation so -o works (#142) (@tamnd)
• b4898da: Make the content commands do what the guide said they did (#118) (@tamnd)
• 6fbaeab: Name the repo the card is committed to (#119) (@tamnd)
• a3dd06a: Refuse an output format nothing can encode (#148) (@tamnd)
• 30f40d5: Refuse crawl seed --max-tier 1 instead of scanning for nothing (#136) (@tamnd)
• 390dbf1: Stop printing flags that do not exist (#124) (@tamnd)
• 65d9c26: Stop printing help nobody wrote (#128) (@tamnd)
• f2ff80d: Write down that a misspelled subcommand exits 0 (#149) (@tamnd)
• 102a6e9: exit 8 when Common Crawl cannot be reached (#132) (@tamnd)
• 7fd895c: rank: find the table instead of demanding one (#130) (@tamnd)

v0.10.0

Changelog

Other

• 6210889: Bind the api server to loopback, and fail rather than serve half a table (#111) (@tamnd)
• f768f84: Bound the memory search spends on --at, --latest-only and --dedup (#101) (@tamnd)
• d4cab12: Check that a published shard is right, not only that it landed (#103) (@tamnd)
• 4586f2a: Check the writes in the CDX test server, for errcheck (#107) (@tamnd)
• b60e866: Give the dataset library a manifest, checksums, and a gc (#114) (@tamnd)
• 4449ef5: Make the search index do what it says, and write down where it stops (#109) (@tamnd)
• 5ffa5c5: Read a config file, with named profiles (#113) (@tamnd)
• 241b83b: Run the whole CLI end to end against a fake Common Crawl, and floor the coverage (#108) (@tamnd)
• a5cac48: Share the request rate limit across every ccrawl process on the host (#104) (@tamnd)
• 6bb946b: Stop losing a whole search to one index page (#115) (@tamnd)
• 0ddfa98: Stop wrapping the CDX callback's error with the page it came from (#106) (@tamnd)

v0.9.0

Changelog

Other

• 65768c9: Write the v0.9.0 release notes (#100) (@tamnd)
• 02c02bf: crawl: add crawl run, the loop over the frontier (#99) (@tamnd)
• 214752f: crawl: put the frontier on disk (#95) (@tamnd)
• 3e2cffa: crawl: rewrite robots.txt to RFC 9309 (#97) (@tamnd)
• f627822: crawl: stop asserting throughput floors on CI hardware (#96) (@tamnd)
• fbd258c: crawl: write WARC to ISO 28500 (#98) (@tamnd)

v0.8.0

Changelog

Features

• 9e32c9e: feat(pipeline): content deduplication, exact and near (#90) (@tamnd)

Fixes

• abcd914: fix(deps): ami v0.3.1 for the fetch trace data race (#91) (@tamnd)

Other

• af1f8a8: Add negated and set valued columnar filters (#83) (@tamnd)
• b83ed4d: Detect language in the markdown pipelines and filter on it (#86) (@tamnd)
• 25b3f80: Drop a stray blank line in batchfetch.go (#85) (@tamnd)
• d88ab19: Fetch WARC records in coalesced ranged GETs (#84) (@tamnd)
• 637a784: Let the caller pick the extractor, and read WET files as a source (#87) (@tamnd)
• 5d0523c: Move whatlanggo to the direct requires (#89) (@tamnd)
• f093201: Write the v0.8.0 release notes (#94) (@tamnd)
• 1dc0cd0: markdown export: convert a location set instead of whole shards (#92) (@tamnd)
• 687b446: markdown: keep the local parquet when there was no push (#93) (@tamnd)

v0.7.0

Changelog

Other

• f1a75cb: Add a run journal, progress modes and metrics for long runs (#79) (@tamnd)
• 50f22fa: Answer the columnar filter queries without duckdb (#80) (@tamnd)
• eb0d9bb: Delete the Python HuggingFace commit path (@tamnd)
• a8524b6: Make --source s3 actually read over S3 (@tamnd)
• ee9e8f3: Write the v0.7.0 release notes (#81) (@tamnd)

v0.6.0

Changelog

Other

• 194112e: Add domains diff to count added, removed, and shared domains between releases (@tamnd)
• 9cf870b: Clear the lint findings in the new commit path (@tamnd)
• 5202cfc: Commit to HuggingFace from Go instead of shelling out to Python (@tamnd)
• d1bdd03: Document domains diff in the CLI reference (@tamnd)
• d058fcf: Document the exit code contract (@tamnd)
• f7a758f: Make the docs describe the binary that ships (@tamnd)
• ef37c21: Stop sched diff putting every noisy host on a 24h recrawl (@tamnd)
• d2b49dd: Stop the markdown help advertising a flag that does not exist (@tamnd)
• 0bc8715: Write the v0.6.0 release notes (@tamnd)

v0.5.3

Changelog

Other

• 8b6235b: Add v0.5.3 release notes (@tamnd)
• 67ab0e7: Pick the newest domain graph that actually has ranks (@tamnd)

v0.5.2

Changelog

Other

• 440c40a: Add a live build-metrics section to both dataset cards (@tamnd)
• 72872ea: Add domains recount to repair release totals from the hub (@tamnd)
• faf31a3: Add urls recount to repair drifted URL and byte totals from the hub (@tamnd)
• c66a6aa: Add v0.5.2 release notes and document the publish pipelines (@tamnd)
• 5ccb27a: Bring the two dataset cards up to arctic depth (@tamnd)
• 35bee3e: Fix CI: gofmt, tidy go.sum, bump x/text past GO-2026-5970 (@tamnd)
• 8a0feba: Mark domain releases complete only at end of stream (@tamnd)
• f1af863: Move the dataset cards into embedded Markdown templates (@tamnd)
• 9c1802f: Publish the dataset card with every batch, not just at the end (@tamnd)
• 8550e19: Republish the Common Crawl URL index and domain ranks to HuggingFace (@tamnd)
• 36d887c: Retry urls publish until the crawl is whole (@tamnd)
• dca0874: Stop the resume probe from treating a rate limit as "nothing exists" (@tamnd)

v0.5.1

Changelog

Other

• f56afa2: Add dataset publish and pull to cache a processed corpus on HuggingFace (#34) (@tamnd)
• 248820a: Bump go toolchain to 1.26.5 (#35) (@tamnd)

v0.5.0

Changelog

Other

• 8c358c2: Add seed cc and seed publish for full-crawl URL seeds (#32) (@tamnd)
• e786013: Add v0.5.0 release notes (#33) (@tamnd)