Tracking Issue for serving an index over HTTP

[ehuss]

Summary

RFC: #2789
Implementation: #10470
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#sparse-registry
Issues: A-sparse-registry Area: http sparse registries

This is a tracking issue for RFC #2789, an experimental extension to serve the index over HTTP instead of via git.

Unresolved issues

• Cache invalidation
  • Currently 1 minute TTL. Too long for publish, too short for effective caching.
  • Minor concern about how long an invalidation takes (Cloudfront says 10 to 100 seconds)
  • The cost is expected to negligible.
  • Send invalidations for index.crates.io when updating crates crates.io#4913
• Make sparse+ and registry+ more consistent. HTTP registry implementation #10470 (comment) (resolved Add new SourceKind::SparseRegistry to differentiate sparse registries #11209)
• Find a better name (and structure) for index_version HTTP registry implementation #10470 (comment)
• Handle user urls that start with http. They currently imply registry+. But if sparse+ are more common then it would be good to auto detect. RFC: Serve crates-io registry over HTTP as static files rfcs#2789 (comment)
• Have a plan for lockfiles.
• Implement support in crates.io Upload index metadata to index/ when publishing new crates crates.io#4661
• Check that crates.io is willing to maintain their current system at production levels of traffic.
• Teach cargo that crates.io has two equivalent indexes. Teach cargo that crates.io has two equivalent indexes. #10722
• Make sure we are future compatible with a merkle tree design. Sparse registry indexes should be viewable atomically #10928
• The RFC is intentionally sparse on details, it is almost an eRFC. Do we need another RFC/PR to discuss the details we have chosen?
• Does not support fuzzy queries. Is that a blocker? Only used by error messages. This was a misunderstanding, indexes don't support name-based typo suggestions at all. Sparse supports the same case-folding and -/_ fuzzy comparison as git indexes. Sparse fundamentally won't be able to support typo suggestions, but since cargo hasn't handled that in the past, this isn't a regression, and more like a "can't do that in the future" thing.
• Investigate publish issues. 1 minute update is quite slow if publishing multiple packages (and cargo does not yet have retry logic).

Current best way to test

To try it out, add the -Z sparse-registry flag on nightly-2022-06-20 or newer build
of Cargo. For example, to update dependencies:

rustup update nightly
cargo +nightly -Z sparse-registry update

The feature can also be enabled by setting the environment variable
CARGO_UNSTABLE_SPARSE_REGISTRY=true. Setting this variable will have no effect on stable
Cargo, making it easy to opt-in for CI jobs.

If you see any issues please report them in new tickets here in the Cargo repo. The output of Cargo
with the environment variable CARGO_LOG=cargo::sources::registry::http_remote=trace set
will be helpful in debugging.

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

Implementation history

• Registry functions return Poll to enable parallel fetching of index data #10064

• HTTP registry implementation #10470

• Refactor RegistryData::load to handle management of the index cache #10482

• Require http-registry URLs to end with a '/' #10698

• Make -Z http-registry use index.crates.io when accessing crates-io #10725

• Improve testing framework for http registries #10738

• Make is_yanked return Poll<> #10830

• Fix publishing to crates.io with -Z sparse-registry #10831

• Refactor check_yanked to avoid some duplication #10835

• Change progress indicator for sparse registries #11068

• Add retry support to sparse registries #11069

• Fix sparse registry lockfile urls containing 'registry+sparse+' #11177

• Add new SourceKind::SparseRegistry to differentiate sparse registries #11209

• Add configuration option for controlling crates.io protocol #11215

• Upload index metadata to index/ when publishing new crates crates.io#4661

• Drop index/ prefix when uploading crate metadata. crates.io#4826

[mightyiam]

Hey, would this mean that when cargo publish exits with 0 the published crate version is actually available (which is not the case now, due to CDN)?

[kornelski]

It doesn't address this problem. There is still going to be a delay between cargo publish and the crate being available globally. It would be nice to improve this, but it's more of an issue for server-side crates-io implementation than the registry protocol.

[epage]

@mightyiam #9507 is the issue you want to be following.

For background, publishing a crate used to be a blocking operation but to speed up crates.io, they made cargo publish just put the crate into a queue for later publishing, making the operation asynchronous.

Tools like cargo-release poll the server. We could easily do similar in cargo until an improved registry protocol is made. I was looking at implementing it but there was a large hurdle for writing the relevant tests. The new sparse registry code has improved the test infrastructure so I could now write the relevant tests.