Add API support additional to Git as a Crate registry index

[secana]

Status Quo

Currently, cargo expects a crate index in the form of a git repository where all crates are listed. See Cargo Book: Registries. This has some disadvantages:

• The git index needs to be pulled and updated before a build. Especially on build servers, where the build starts "fresh", this adds multiple minutes to each build.
• Switching the implementation behind the index is not possible. While the registry HTTP API defines a clear API and the implementation is hidden (doesn't matter of written in rust, python, ...), the index has to be a git repository. Using a DB, S3 or similar, to store the crate metainformation, is not easily possible.
• Authentication is limited. While the HTTP API to publish crates can easily support enterprise authentication like LDAP, the git index cannot. This is problematic for private registry, where authentication is needed to pull crates.

Possible solution

Add a feature to cargo that handles the pull of a crate or the search for a crate the same way the push to a registry is handled. Define a HTTP API which allows cargo to request if the registry has a crate in version x instead of cloning the git index and then search for the crate.

This would improve the performance of builds, as cargo can start directly to request only needed crates instead of cloning the whole index. The crate metadata store itself would be a hidden implementation detail, e.g. a DB or S3

It would allow to use the same token authorization approach to pull samples a it's used to push samples to a registry. Where the token is stored and how the authorization is done is a hidden implementation detail, which would support enterprise authentication mechanisms for private registries.

Notes

This is not a demand that someone implements such a feature, I would use this issue for a discussion and if there is interest create a proper RFC. As I whish for such a feature, I would like to help it implement, too. So please, let me know what you think!

Furthermore to make my motivation clear: I work for a company (no name, as I don't want to advertise here) which sells a private crate registry and in talks with our customers we got multiple requests for enterprise authentication and faster build times, especially on build servers.

[ehuss]

This sounds a bit like RFC 2789, have you checked that out? There has been some preliminary work on it over at #8890.

[secana]

Hi @ehuss, thanks for the RFC link. I did not find that issue before. Looks exactly like what I need. I'll close this issue and add my comments to the RFC.