What is the significance of "specification" of index format/API?

[withoutboats]

In the alt registries RFC, @carols10cents specified the format that a registry index takes. I remember that we made a conscious choice to specify the index format, but not to specify the API that crates.io uses for publishing, yanking, etc.

I'm unclear, however, what it really means to specify these things. It seems to be about a level of service we provide and a level of expectation we put on maintainers of alt registries, but we haven't really nailed down what that means.

Here are some things I think this could maybe mean:

• Does this mean we will maintain accurate documentation of the parts of the interface we've specified (but not necessarily other parts)?
• Does this mean we require that all registries conform to the specified parts, but not the unspecified parts (i.e. if this were an IETF spec we would be using MUST for the index, but only SHOULD or MAY for the api parts).
• Does this mean we should not implement features that depend on the "unspecified" parts of the interface (e.g. we should not make it easier to cargo publish to alternative registries, since that requires them to support the unspecified API)?
• Or rather, restating the previous bullet, specifying a part of the interface is a necessary prerequisite toward generalizing a workflow that is built on top of that interface?

Probably, there are other significances this distinction could have to people that haven't occurred to me.

[carols10cents]

I'm going to think more on these questions but in the meantime I wanted to tag in @wycats for his thoughts on this.

[alexcrichton]

To me it does seem like the high-order bit is getting cargo build working rather than publication/browsing of packages, that seems to be the real enabler. In that sense it makes sense to me to start off by specifying the index (which cargo build downloads and consumes) without the API (which is very rarely used only by cargo publish)

[withoutboats]

I had overlooked before that rust-lang/rfcs#2141 specified that the publish key in a Cargo.toml could take a list of registries that cargo publish will publish this to. @cswindle has implemented this in #4568.

I have some concerns about this:

First, the highest priority issue - cargo currently stores a single auth token in the config as registry.token. The RFC doesn't mention anything about this, and #4568 would have us send that auth token to every registry. We definitely can't have users sharing a single auth token among all registries they publish to.

But this raises a sort of deeper question. Clearly not all registries will use GitHub as the auth mechanism; how much do we expect them to share? The request made by cargo publish contains a number of details, some of them very particular:

• It communicated with the server using a single HTTP PUT request to the path api/v1/crates/new (appended to the URL in the index's api member).
• It sets an Authorization header, currently with that single token received from GitHub.
• The body it sends has a very particular format:
  • A JSON blob containing package metadata preceeded by a little endian u32 of the length of that blob.
  • A tarball of the package preceded by a little endian u32 of the length of the tarball.
• It expects to receive a JSON blob in response, from which it looks for warnings from crates.io, listing the invalid categories and badges.

I could imagine alternative registries wanting to make very different choices around this, especially around how they handle authentication & authorization. And if the servers and robust in how they handle the metadata JSON (e.g. not being extensible to new fields), we could have frustrating problems with breaking peoples' registries in the future as we extend the features of cargo.

What I especially don't want is a situation where you interact with 3 or 4 alternative registries, and half of them support cargo publish and half of them don't, because then I'll have to look up every time whether or not this registry supports cargo publish. Right now, I feel a lot more comfortable with saying that cargo publish et al only speak to the default registry, which is crates.io, and expecting that out of band tools will be built for mutating other registries (with our support, e.g. the metadata and index stuff).

[alexcrichton]

Oh I was sort of expecting that for now a custom registry is just running literally the crates.io software with different configuration or perhaps a patch or two, in which case we wouldn't have to worry too too much about the implementation details. You're definitely right though in that we'd have to worry about the publish key here especially wrt tokens. The good news though is that it's all unstable!

My thinking is that if we enable publishing to different registries then we'd probably change the format of the credentials file in a backwards-compatible fashion to be able to store tokens for multiple registries.

[cswindle]

Regarding the issue with sending a single authentication token, you would need to override this as part of the publish, until the related issue of saving multiple tokens in .cargo/credentials is solved (which is referenced in the RFC).

The last point I agree is potentially an issue, but the key thing that the publish key provides is ensuring that we don't publish to a server that we don't want to (which I think is required to stop private crates getting out onto crates.io as there is no easy way to remove them once they have been published).

[withoutboats]

Oh I was sort of expecting that for now a custom registry is just running literally the crates.io software with different configuration or perhaps a patch or two, in which case we wouldn't have to worry too too much about the implementation details.

See to me this seems like overkill for a private registry - and also ties them to using GitHub for auth. I could imagine tooling existing to maintain a private registry just based on a hosted git repo and an Amazon S3 bucket, using the out of the box auth mechanisms of those two systems, with no HTTP API of its own to speak of.

This would not use cargo publish but some other tool, probably with a similar CLI but doing a totally different transaction under the hood.

[cswindle]

We went the route of running a private crates.io, although we made some changes to crates.io to allow integration with gitlab OAuth, this way we have a nice way to search through private crates in the same way that we can for public crates.

[withoutboats]

@cswindle I think that's a viable choice, but my concern is that we're going to lock people into requiring that for their tooling to work how they expect it to.

---

To try to frame my concern, I'd like to think in terms of a "minimum viable registry." What is the least a registry must support to be considered a "conformant" alternative registry?

Then, I'd like to only extend workflows to suppot alternative registries if that minimum registry would satisfice that workflow. If we want to extend a workflow to alternative registries that requires more, the minimum viable registry needs to include that support.

What I don't want is a mix of "opportunistic" workflows, each of which require different levels of support from the registry, and so each of which may or may not be supported by any particular registry.

Lastly, I want to consider what support to alternative registries we're committing to providing. That is, how extensively will we document/spec that "minimum registry" requirement, vs things beyond that? I don't want you to have to clone crates.io or read the source to create a registry that supports a workflow we consider "supported for alternate registries."

[cswindle]

I acknowledge that concern, but if people do have the option of using publish to push the crates to an alternate registry (which they do as they can run their own crates.io), then I really think that we need a method of saying what registries the crate is permitted to be pushed to, otherwise we run the risk of code ending up on crates.io, or the wrong alternate registry, when it shouldn't.

[withoutboats]

So I talked today to André Arko, who is a maintainer of bundler, and he had an interesting proposal.

We could add a key to the config.json in the index called commands, which is an array of strings. For example:

{
    "api": "https://crates.io",
    "crates": "https://crates.io/api/crates",
    "commands": ["publish", "yank", "owner"] // etc
}

If a user attempts to use such a command on an alternative registry, if that registry's index does not claim to support the command, instead of attempting and barfing, cargo could report that the command is not supported by that registry. This way, registries with less capabilities don't just throw random errors, they report that they don't support those commands.

Additionally, we assume that the default registry supports all commands, and we assume that all alternative registries support fetching & building.