Milestone: v0.7.0, self contained

[tamnd]

Tracking issue for the v0.7.0 milestone. Plan lives in notes/Spec/2115/08-roadmap.md section 3.

Goal

Drop the second external dependency and make long runs legible.

Checklist

• E7: --source s3 does not fetch anything over S3 #44 E7, --source s3 does not fetch anything over S3
• E9: native columnar engine so duckdb is optional #45 E9, native columnar engine so duckdb is optional
• E12: structured run journal and metrics for long jobs #46 E12, structured run journal and metrics for long jobs

Ship gate

• the six columnar subcommands return identical results under --engine native and --engine duckdb across three crawls
• a markdown export run leaves a queryable run.jsonl
• RSS reporting works on macOS

Sequencing

E9 is the biggest single item in the whole plan and the one most likely to overrun. It also has a working precedent in the repo, projectURLPart already does ranged remote parquet reads. Do columnar count alone first as a spike and confirm row group statistics pruning actually works against real CC parquet files before committing to all six subcommands.

Process

Every PR that lands against this milestone gets a comment on this issue with the PR number and one line on what changed, and the matching checklist box is ticked when the linked issue closes. Automated by .github/workflows/milestone-tracker.yml.

[github-actions]

Merged #77, Delete the Python HuggingFace commit path.

Closes:

• E1 cleanup: delete the Python HuggingFace commit path #75 E1 cleanup: delete the Python HuggingFace commit path

[github-actions]

Merged #78, Make --source s3 actually read over S3.

Closes:

• E7: --source s3 does not fetch anything over S3 #44 E7: --source s3 does not fetch anything over S3

[github-actions]

Merged #79, Add a run journal, progress modes and metrics for long runs.

Closes:

• E12: structured run journal and metrics for long jobs #46 E12: structured run journal and metrics for long jobs

[github-actions]

Merged #80, Answer the columnar filter queries without duckdb.

Closes:

• E9: native columnar engine so duckdb is optional #45 E9: native columnar engine so duckdb is optional

[tamnd]

All three ship gates are met, on real data rather than fixtures.

Both engines agree across three crawls. Every subcommand run under --engine native and --engine duckdb against CC-MAIN-2026-30, CC-MAIN-2026-25 and CC-MAIN-2026-21, robotstxt subset, sorted output compared byte for byte. 18 of 18 identical. The --domain vnexpress.net cases are the ones worth having, because those 370 to 395 rows are an apex plus a dozen subdomains and they are exactly what a single surtkey prefix would have got wrong. Full list in #80 (comment).

The comparison paid for itself. It caught toInt64 zeroing every warc offset the native engine produced, which meant 370 locations with the right filename and URL and an offset of 0, a record nobody can fetch that looks completely fine until the fetch comes back empty. It also caught langs and mimes writing a different JSON key per engine. Neither was visible from the unit tests, which is the argument for insisting on this gate rather than trusting the suite.

A markdown export run leaves a queryable run.jsonl. One real shard of CC-MAIN-2026-30, 962 MB of WARC in, 20861 rows out:

$ jq -r 'select(.event=="shard") | "shard \(.shard) rows=\(.rows) warc=\(.warc_bytes) parquet=\(.parquet_bytes)"' run.jsonl
shard 0 rows=20861 warc=962786231 parquet=50304299

$ jq -r 'select(.event=="shard" and .status!="ok") | .shard' run.jsonl
(nothing, no shard failed)

$ jq -r 'select(.event=="tick" or .event=="end") | "\(.elapsed_s|floor)s inflight=\(.inflight // 0) rate=\(.rate_per_hour // 0)"' run.jsonl
30s inflight=1 rate=0
60s inflight=1 rate=0
90s inflight=1 rate=0
93s inflight=0 rate=38.487014778552634

Which shards failed and when the rate dropped are both one filter away, which is what the gate was asking for.

RSS reporting works on macOS. rss_bytes in that same journal climbs from 407552000 to 548126720 across the run, on darwin. It read 0 before v0.7.0.

Release notes are in #81. Tagging v0.7.0 next.

[tamnd]

v0.7.0 is out: https://github.com/tamnd/ccrawl-cli/releases/tag/v0.7.0

Everything on the checklist landed and all three ship gates are ticked with real data behind them. The release carries the full set of artifacts, archives for Linux, macOS, Windows and FreeBSD, deb, rpm and apk packages, the multi-arch image on GHCR, and the apt and dnf repositories are refreshed.

The one thing left open from this milestone is not a code gap. #44 shipped with the S3 signing verified against botocore and verified live as far as a credential error, but nobody has confirmed a successful signed read that moves bytes, because there are no AWS credentials on the box this was built on. Worth doing from an EC2 instance in us-east-1 when somebody has one to hand.

Closing this out.