Milestone: v0.10.0, durability and scope

[tamnd]

Tracking issue for the v0.10.0 milestone. Plan lives in notes/Spec/2215/08-roadmap.md section 6.

Goal

Make the long runs trustworthy and stop half committing to the side projects.

Checklist

Correctness under scale:

• E8: --at and --dedup buffer the whole result set in memory #55 E8, --at and --dedup buffer the whole result set in memory
• E11: publish verify, nothing checks that an uploaded shard is right #56 E11, publish verify, nothing checks that an uploaded shard is right
• E19: the politeness throttle is per process, so running three pipelines triples the rate #61 E19, the politeness throttle is per process, so running three pipelines triples the rate

Confidence:

• E13: get coverage to a defensible floor and enforce it #57 E13, get coverage to a defensible floor and enforce it

Scope decisions, both recommended as document the ceiling and stop, both reversible later:

• E14: decide what the BM25 index is #58 E14, decide what the BM25 index is
• E15: decide what the api server is #59 E15, decide what the api server is

Ergonomics:

• E16: push search filters to the CDX server #60 E16, push search filters to the CDX server, moved to v0.10.1 because it cannot be measured while the CDX server is down
• E20: config file and profiles, --profile currently selects from nothing #62 E20, config file and profiles
• E21: dataset library has no manifest, checksums or gc #63 E21, dataset library has no manifest, checksums or gc

Ship gate for v0.10.0

• coverage floors enforced in CI
• every documented flag exists
• every command's runtime dependencies are stated and true
• the exit codes documented, which turned out to be seven and not four
• no subsystem is sitting between demo and production without saying which it is

Process

Every PR that lands against this milestone gets a comment on this issue with the PR number and one line on what changed, and the matching checklist box is ticked when the linked issue closes. Automated by .github/workflows/milestone-tracker.yml.

[github-actions]

Merged #101, Bound the memory search spends on --at, --latest-only and --dedup.

Closes:

• E8: --at and --dedup buffer the whole result set in memory #55 E8: --at and --dedup buffer the whole result set in memory

[github-actions]

Merged #103, Check that a published shard is right, not only that it landed.

Closes:

• E11: publish verify, nothing checks that an uploaded shard is right #56 E11: publish verify, nothing checks that an uploaded shard is right

[github-actions]

Merged #104, E19: share the request rate limit across every ccrawl process on the host.

Closes:

• E19: the politeness throttle is per process, so running three pipelines triples the rate #61 E19: the politeness throttle is per process, so running three pipelines triples the rate

[github-actions]

Merged #106, Stop wrapping the CDX callback's error with the page it came from.

Closes:

• ccrawl search --limit N fails with "Cdx page 0: stop." once the limit is hit #105 ccrawl search --limit N fails with "Cdx page 0: stop." once the limit is hit

[github-actions]

Merged #107, Check the writes in the CDX test server, for errcheck.

[github-actions]

Merged #108, Run the whole CLI end to end against a fake Common Crawl, and floor the coverage.

Closes:

• E13: get coverage to a defensible floor and enforce it #57 E13: get coverage to a defensible floor and enforce it

[github-actions]

Merged #109, E14: the search index is a reference implementation, and now says so.

Closes:

• E14: decide what the BM25 index is #58 E14: decide what the BM25 index is

[github-actions]

Merged #111, E15: the api server is a localhost tool, and now behaves like one.

Closes:

• E15: decide what the api server is #59 E15: decide what the api server is

[github-actions]

Merged #113, Read a config file, with named profiles.

Closes:

• E20: config file and profiles, --profile currently selects from nothing #62 E20: config file and profiles, --profile currently selects from nothing

[github-actions]

Merged #114, Give the dataset library a manifest, checksums, and a gc.

Closes:

• E21: dataset library has no manifest, checksums or gc #63 E21: dataset library has no manifest, checksums or gc