Offline mode

[matklad]

Implementation PR: #4770
Original issue: #4686

Summary:

Running Cargo with -Z offline flag changes the resolution process such that it only considered crates, already downloaded locally.

Steps:

• currently, there's no way to populate local crate storage, see Add an "airplane" flag to Cargo, altering it's dependency resolution behavior #4686 for details

• write precise documentation about effects and intended use-case for this flag. See https://github.com/rust-lang/cargo/blob/5845464cf92a348f95dc6b43133c017c37fcb684/src/doc/src/reference/unstable.md#offline-mode for current docs

Stabilization TODO:

• change -Z offline unstable cli option to a stable one, which works the same way as --frozen and --locked flags do.
• -Z offline actively breaks building crate offline #6014

[matklad]

cc @Eh2406

[Eh2406]

Thanks for the write up. I think "no way to populate local crate storage" is overstated. There are pretty good, approximate, ways to do it like having used that dep in any previous project. Nead an ecosystem, build stdx, or equivalent and you have most of it. And complete, manually ways to do it, download all the crates and unpack them into carcos cache directory. It just takes a lot of disk space ~12 GB. Both of these the community can develop and improve once people are using offline. What is missing is a way to store a local compressed copy of crates and only uncompressed them as cargo needs them. That seems like it needs a lot of design work and could be an add on feature.

Given that, I think we should work to stabilize the existing offline, and come back to populateting local crate storage.

[alexcrichton]

One TODO item I might add as well is to audit the error messages and make sure they're understandable. For example if resolution fails due to the inability to update the registry then the error message should state that -Z offline was passed and removing it may allow the resolution to be fixed (if there's a missing version or something like that)

Otherwise I do personally still feel that population of the local storage is somewhat important. I see this as a sort of "what do I do before I get on a plane?" question. Or similarly what does Crater do before building all its crates? I think we'd want it along the lines of cargo cache stdx or something like that (aka just download a list of crates and all their transitive dependencies from crates.io) but I think we'd want to build that out in some command (the implementation shouldn't be too hard) before stabilizing.

[Eh2406]

An audit of the error messages is a good idea. I think the "-Z offline may be your problem" is in already. But the opposite would be good as well. "errore: no internet access; you can try -Z offline to try with what you have cashed"

I think that an initial version of a cargo cache can be an out of tree cargo-subcommand. That would allow the community to experiment in the design space, before we have to commit to an implementation.

[alexcrichton]

But the opposite would be good as well

An excellent idea!

I think that an initial version of a cargo cache can be an out of tree cargo-subcommand

I'd be totally on board with that

[luser]

I'm sure I mentioned this in a previous issue but I've used cargo-cacher for offline work before. It has a few options but I usually just run it with --all to fetch everything it can from crates.io. That always feels like overkill to me, though. Ideally I think I'd want some options like "fetch these specific crates and all their dependencies", "fetch the top N crates and all their dependencies", "fetch the newest version of every crate and all their dependencies".

[ehuss]

Is it intentional that offline mode will fail if an optional dependency is missing? It seems like it shouldn't because in online mode it doesn't fetch anything (AFAIK).

Example:

[package]
name = "off"
version = "0.1.0"

[dependencies]
# A very old version you are unlikely to have in your cache.
bitflags = {version="=0.1.0", optional=true}

> cargo +nightly build -Z offline
error: no matching version `= 0.1.0` found for package `bitflags`
location searched: registry `https://github.com/rust-lang/crates.io-index`
versions found: 1.0.3, 1.0.2, 1.0.1, ...
required by package `off v0.1.0 (file:///Users/eric/Proj/rust/cargo/scratch/off)`
As a reminder, you're using offline mode (-Z offline) which can sometimes cause surprising 
resolution failures, if this error is too confusing you may with to retry without the offline flag.

[alexcrichton]

@ehuss currently, yes, unfortunately. That's because lockfile generation requires the assumption that all possible features are enabled. Generating a lockfile without optional features is covered by #5133

[Jasper-Bekkers]

A similar situation seems to happen with target specific dependencies. Something with this in the Cargo.toml will fail to build on windows with -Z offline:

[target.x86_64-unknown-linux-gnu.dependencies]
x11-dl = "~2.14"

Giving

error: no matching package named `x11-dl` found

[alexcrichton]

@Jasper-Bekkers is that the full error message? Does Cargo print any other information?

[Jasper-Bekkers]

It's pretty much the full error message.

>cargo build -Z offline
error: no matching package named `x11-dl` found
location searched: registry `https://github.com/rust-lang/crates.io-index`
required by package `minifb v0.10.7`
    ... which is depended on by `voxelize-rs v0.1.0 (file:///...)`
As a reminder, you're using offline mode (-Z offline) which can sometimes cause surprising resolution failures, if this error is too confusing you may with to retry without the offline flag.