src tarballs are vendored without respecting lockfile

[infinity0]

This is with 1.48.0-beta.8, and previously all passed in 1.47.0. On Debian I see this failure on all architectures.

test [ui] ui/feature-gates/feature-gate-cfg-version.rs ... FAILED

------------------------------------------
diff of stderr:

118	   = note: see issue #64796 <https://github.com/rust-lang/rust/issues/64796> for more information
119	   = help: add `#![feature(cfg_version)]` to the crate attributes to enable
120	
+	error: invalid version literal
+	  --> $DIR/feature-gate-cfg-version.rs:30:15
+	   |
+	LL | #[cfg(version("1.65536.2"))]
+	   |               ^^^^^^^^^^^
+	
121	error[E0658]: `cfg(version)` is experimental and subject to change
122	  --> $DIR/feature-gate-cfg-version.rs:40:18
123	   |

127	   = note: see issue #64796 <https://github.com/rust-lang/rust/issues/64796> for more information
128	   = help: add `#![feature(cfg_version)]` to the crate attributes to enable
129	
-	error: aborting due to 16 previous errors
+	error[E0425]: cannot find function `version_check_bug` in this scope
+	  --> $DIR/feature-gate-cfg-version.rs:37:5
+	   |
+	LL |     version_check_bug();
+	   |     ^^^^^^^^^^^^^^^^^ not found in this scope
131	
-	For more information about this error, try `rustc --explain E0658`.
+	error: aborting due to 18 previous errors
+	
+	Some errors have detailed explanations: E0425, E0658.
+	For more information about an error, try `rustc --explain E0425`.
133	

It seems the failure is due to version_check being upgraded from 0.9.1 to 0.9.2 which now treats 1.65536.2 as malformed for some reason, even though I can't see this mentioned as part of https://semver.org/

SergioBenitez/version_check@v0.9.1...v0.9.2

Shouldn't this have been caught by rust CI, or is this not run on beta releases?

[infinity0]

Apparently they made a design decision to reject numbers above 65536, but again I can't see this in the semver spec which allows any integer. SergioBenitez/version_check#11

[Mark-Simulacrum]

Hm, version_check is at 0.9.1 on beta branch AFAICT -- can you say more about where you're getting 0.9.2 from? Maybe our vendoring is somehow buggy?

That said this is not really a beta regression since it's for an unstable feature so not marking as such.

[infinity0]

I downloaded https://static.rust-lang.org/dist/rustc-beta-src.tar.xz it's on 0.9.2 there.

tar xf rustc-beta-src.tar.xz rustc-beta-src/vendor/version_check/Cargo.toml

[Mark-Simulacrum]

That is indeed concerning. Looking at the logs for the dist-x86_64-linux builder, it uses version_check 0.9.1 for the main steps but the src tarball does indeed download 0.9.2, which is unexpected. I suspect it might be because of the additional Cargo.toml's included with --sync?

cc @ehuss @alexcrichton, I guess we should be passing --locked/--frozen to cargo vendor? I am surprised to see it altering the set of dependencies though

[infinity0]

That said this is not really a beta regression since it's for an unstable feature so not marking as such.

Not sure if this matters for your release policy, but the test is run at least and is expected to contain compiler errors saying "cfg(version) is experimental and subject to change", but the presence of these extra errors makes the test fail.

[Mark-Simulacrum]

Right, all tests run independent of the channel being targeted, and in this case the feature affected is not usable on stable regardless. (Edit: Our tests always run with RUSTC_BOOTSTRAP or equivalent, so they can always enable features if they want).

[ehuss]

I'm a bit uncertain what the issue is here. Because the vendor directory is including multiple workspaces, it is to be expected that there are multiple versions of different dependencies like version_check. From the beta tarball, there is a vendor/version_check-0.9.1/ directory which contains the version used by the root workspace. The root Cargo.lock contains the correct version, and should "just work" even with source replacement.

What are the commands that I can run after downloading the source tarball to reproduce the issue? I tried running ./configure --set build.vendor=true && ./x.py test src/test/ui/feature-gates/feature-gate-cfg-version.rs and it seemed to run fine.

[infinity0]

Ah right, this is an issue with how the Debian packaging works. The typical rustc tarball contains a lot of other tools and all of their dependencies as well. If we retained all of that, we would have to review all of this stuff every update to check it adheres to Debian policy and fix it if it doesn't (e.g. does not embed new C code & instead make it link to system shared libraries instead). This is a lot of effort for me to do every 6 weeks (at this stage it's just me maintaining this package). So instead of doing all of that, we delete these extra tools and regenerate Cargo.lock by running x.py build src/bootstrap. Otherwise the build complains because Cargo.lock contains references to the things we deleted. However this of course also refreshes the dependencies (within semver) without actually running the tests to ensure the new dependencies still work.

OTOH if this process does cause problems as described in the OP, it is still a bug in that crate since semver is supposed to retain this sort of compatibility.