Move `std_detect` into stdlib #143412

Kobzol · 2025-07-04T07:52:58Z

This PR moves the std_detect crate from stdarch to be a part of rust-lang/rust instead.

The first commit actually moves the whole directory from the stdarch Josh subtree, so that git blame history is kept intact. Then I had to make a few changes to appease tidy.

The most complex thing here is porting the tests. We can't have std_detect both in r-l/r and stdarch, because they could get desynchronized, so we have to perform the move more or less "atomically", which means that we also have to port all the existing std_detect tests from the stdarch repository.

The stdarch repo runs the following std_detect tests:

Build

The build-std-detect.sh script (https://github.com/rust-lang/stdarch/blob/e2b6512aed87df45294ae680181eeef7a802cd95/ci/build-std-detect.sh) builds std_detect using the nightly compiler for several targets. This will be subsumed by normal x build library on our Tier 1/2 targets. However, the stdarch repository also tests the following targets:

aarch64-unknown-freebsd
armv6-unknown-freebsd
powerpc-unknown-freebsd
powerpc64-unknown-freebsd
aarch64-unknown-openbsd

Which we don't build/test on our CI currently. I think we have mostly two options here:

Ignore these targets
Create a special CI job that will build stage 1 rustc and then cross-compile std (or just the std_detect crate?) for these targets.

Documentation

The dox.sh script (https://github.com/rust-lang/stdarch/blob/3fec5adcd52a815f227805d4959a25b6402c7fd5/ci/dox.sh) builds and documents std_detect for several targets. All of them are Tier 2/we have dist- jobs for them, so I think that we can just skip this and let our normal CI subsume it?

Tests

The run.sh script (https://github.com/rust-lang/stdarch/blob/1b201cec2cca7465602a65ed6ae60517224b15f3/ci/run.sh) runs cargo test on std_detect with a bunch of variations of feature flags. This will be subsumed by x test library in our CI. The only problem is that stdarch runs these tests for a ludicrous number of targets:

        - tuple: i686-unknown-linux-gnu
        - tuple: x86_64-unknown-linux-gnu
        - tuple: arm-unknown-linux-gnueabihf
        - tuple: armv7-unknown-linux-gnueabihf
        - tuple: aarch64-unknown-linux-gnu
        - tuple: aarch64_be-unknown-linux-gnu
        - tuple: riscv32gc-unknown-linux-gnu
        - tuple: riscv64gc-unknown-linux-gnu
        - tuple: powerpc-unknown-linux-gnu
        - tuple: powerpc64-unknown-linux-gnu
        - tuple: powerpc64le-unknown-linux-gnu
        - tuple: s390x-unknown-linux-gnu
        - tuple: i586-unknown-linux-gnu
        - tuple: nvptx64-nvidia-cuda
        - tuple: thumbv6m-none-eabi
        - tuple: thumbv7m-none-eabi
        - tuple: thumbv7em-none-eabi
        - tuple: thumbv7em-none-eabihf
        - tuple: loongarch64-unknown-linux-gnu
        - tuple: wasm32-wasip1
        - tuple: x86_64-apple-darwin
        - tuple: x86_64-apple-ios-macabi
        - tuple: aarch64-apple-darwin
        - tuple: aarch64-apple-ios-macabi
        - tuple: x86_64-pc-windows-msvc
        - tuple: i686-pc-windows-msvc
        - tuple: aarch64-pc-windows-msvc
        - tuple: x86_64-pc-windows-gnu
        - tuple: aarch64-unknown-linux-gnu
        - tuple: aarch64_be-unknown-linux-gnu
        - tuple: armv7-unknown-linux-gnueabihf
        - tuple: loongarch64-unknown-linux-gnu
        - tuple: powerpc-unknown-linux-gnu
        - tuple: powerpc64-unknown-linux-gnu
        - tuple: powerpc64le-unknown-linux-gnu
        - tuple: riscv32gc-unknown-linux-gnu
        - tuple: riscv64gc-unknown-linux-gnu
        - tuple: s390x-unknown-linux-gnu
        - tuple: x86_64-unknown-linux-gnu
        - tuple: aarch64-apple-darwin
        - tuple: aarch64-apple-ios-macabi

We definitely do not run tests for all of these targets on our CI.

Outcome

We have decided to just subsume std_detect tests by our normal test suite for now, and not create a separate CI job. Therefore, this PR performs the following changes in target testing for std_detect:

The following T3 targets would go from "build" to "nothing":

aarch64-unknown-freebsd (T3)
armv6-unknown-freebsd (T3)
powerpc-unknown-freebsd (T3)
powerpc64-unknown-freebsd (T3)
aarch64-unknown-openbsd (T3)

The following T3 targets would go from "test" to "nothing":

aarch64_be-unknown-linux-gnu (T3)
riscv32gc-unknown-liux-gnu (T3)

The following T2 targets would go from "test" to "build":

arm-unknown-linux-gnueabihf (T2)
armv7-unknown-linux-gnueabihf (T2)
riscv64gc-unknown-linux-gnu (T2)
powerpc-unknown-linux-gnu (T2)
powerpc64-unknown-linux-gnu (T2)
powerpc64le-unknown-linux-gnu (T2)
s390x-unknown-linux-gnu (T2)
i586-unknown-linux-gnu (T2)
loongarch64-unknown-linux-gnu (T2)
wasm32-wasip1 (T2)
x86_64-apple-ios-macabi (T2)
aarch64-apple-ios-macabi (T2)
aarch64-pc-windows-msvc (T2)
armv7-unknown-linux-gnueabihf (T2)
loongarch64-unknown-linux-gnu (T2)
powerpc-unknown-linux-gnu (T2)

I have confirmed in rust-lang/stdarch#1873 that the current version of this PR would pass stdarch's CI testsuite.

r? @ghost

rustbot · 2025-07-08T08:45:13Z

These commits modify the library/Cargo.lock file. Unintentional changes to library/Cargo.lock can be introduced when switching branches and rebasing PRs.

If this was unintentional then you should revert the changes before this PR is merged.
Otherwise, you can ignore this comment.

There are changes to the tidy tool.

cc @jieyouxu

Kobzol · 2025-07-08T08:47:11Z

This is not yet ready for review, but it will need some input from stdarch maintainers regarding the test changes, as they have to happen in this PR, otherwise we either lose testing for std_detect for some time, or we will have two duplicated sources for std_detect, neither is great. Not sure who to ping exactly about the CI changes though.

CC @rust-lang/libs

cuviper · 2025-07-08T18:14:14Z

One of my peers is looking at implementing aarch64 outlined atomics in compiler-builtins, which needs to check for the LSE feature. Will it still be possible for it to use is_aarch64_feature_detected! if this is moved in-tree?

(That said, I'm not sure if that can work either way since the feature caching also uses atomics, so that might create a chicken-and-egg situation...)

Kobzol · 2025-07-08T21:25:08Z

How would it work before? std_detect wasn't published to crates.io in ~6 years. So I assume that the code just accessed std_detect through std? That should still continue working.

tgross35 · 2025-07-08T21:40:37Z

One of my peers is looking at implementing aarch64 outlined atomics in compiler-builtins, which needs to check for the LSE feature. Will it still be possible for it to use is_aarch64_feature_detected! if this is moved in-tree?

Do you mean LSE would need to be checked within compiler-builtins? I don't think that is easily possible since AIUI std_detect for aarch64 requires filesystem access and builtins has to be dependency-free. It might be possible to do some things with weak symbols or a separate .o that gets linked only if std is available, but that gets tricky. We do some detection on x86, which is a clone of core_detect, but that doesn't require access to privileged registers https://github.com/rust-lang/compiler-builtins/blob/8aba4c899ee89eef7fe688cdfa6629ddd56908f9/libm/src/math/arch/x86/detect.rs.

(Amanieu would know more if you want to zulip this)

cuviper · 2025-07-08T21:55:48Z

How would it work before? std_detect wasn't published to crates.io in ~6 years. So I assume that the code just accessed std_detect through std?

The code is currently not doing this, only implementing the more basic LL/LC mode that doesn't need feature checks. That kind of defeats the point of "outlined atomics", but it's functional.

https://github.com/rust-lang/compiler-builtins/blob/8aba4c899ee89eef7fe688cdfa6629ddd56908f9/compiler-builtins/src/aarch64_linux.rs

Do you mean LSE would need to be checked within compiler-builtins?

Yes, I think so, but I'm open to suggestions.

I don't think that is easily possible since AIUI std_detect for aarch64 requires filesystem access and builtins has to be dependency-free.

Note that compiler-builtins recommends to just use compiler-rt if you care, i.e. setting bootstrap build.optimized-compiler-builtins, but that implementation is also not dependency-free. On Linux, it uses getauxval in a constructor here:

https://github.com/llvm/llvm-project/blob/6a993264ee0105da32a6a57fb077796076cf6bf4/compiler-rt/lib/builtins/cpu_model/aarch64.c#L49-L51
https://github.com/llvm/llvm-project/blob/main/compiler-rt/lib/builtins/cpu_model/aarch64/lse_atomics/getauxval.inc

cuviper · 2025-07-08T22:15:08Z

Anyway, as I said, I'm not even sure it's possible for that to work, given that std_detect caching also uses atomics. I just wanted to raise this possibility in case it affects our choices here, to move or not.

tgross35 · 2025-07-08T23:44:58Z

Rather than checking auxvec in compiler-builtins itself, could we check a weakly defined symbol? Then if std is available it provides this symbol, and if it's not available then compiler-builtins will fall back to the default implementation. That keeps the dependency tree clean and all of std (mostly fs and alloc) can be used to implement the feature detection. I think the proposed way of doing feature detection in core would work along those lines rust-lang/rfcs#3469 (comment).

LLVM indeed puts everything into compiler-rt, but I don't believe we need to follow exact suit here (#138944 is similar).

cuviper · 2025-07-09T01:30:12Z

If we allow a constructor like LLVM is doing, then we could have a static symbol like __aarch64_have_lse_atomics (not necessarily that same name) in compiler-builtins that's initialized false, and a constructor in std that checks for LSE to set it true.

bors · 2025-07-11T05:13:21Z

☔ The latest upstream changes (presumably #143762) made this pull request unmergeable. Please resolve the merge conflicts.

Kobzol · 2025-07-11T06:54:58Z

r? @Amanieu

Not sure who is responsible for maintaining the stdarch CI test suite :)

Amanieu

I'm happy with the test changes. r=me

Amanieu · 2025-07-11T13:36:59Z

library/std_detect/src/detect/os/ristc/tests.rs

Seems like a typo in the path.

Kobzol · 2025-07-11T15:26:37Z

Just to clarify, I haven't implemented any of these test changes yet 😅 Just wanted to get a vibe check on the general direction. I'll work on it then.

rustbot · 2025-07-13T11:14:34Z

stdarch is developed in its own repository. If possible, consider making this change to rust-lang/stdarch instead.

cc @Amanieu, @folkertdev, @sayantn

Kobzol · 2025-07-13T13:56:30Z

The stdarch part of this has green CI (rust-lang/stdarch#1873), and I documented the targets whose coverage we lose for std_detect.

This should be ready for another review.

rustbot added the T-libs Relevant to the library team, which will review and decide on the PR/issue. label Jul 4, 2025

This comment has been minimized.

Sign in to view

rustbot added A-tidy Area: The tidy tool T-bootstrap Relevant to the bootstrap subteam: Rust's build system (x.py and src/bootstrap) labels Jul 4, 2025

This comment has been minimized.

Sign in to view

Kobzol force-pushed the std-detect-in-stdlib branch from 9086668 to 277837d Compare July 4, 2025 13:25

Kobzol mentioned this pull request Jul 5, 2025

Remove std_detect dev dependency in core_arch rust-lang/stdarch#1843

Merged

Kobzol marked this pull request as ready for review July 8, 2025 08:45

rustbot added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Jul 8, 2025

Kobzol changed the title ~~Move std_detect into stdlib~~ [do not merge] Move std_detect into stdlib Jul 8, 2025

rustbot assigned Amanieu Jul 11, 2025

Amanieu approved these changes Jul 11, 2025

View reviewed changes

library/std_detect/src/detect/os/ristc/tests.rs

Copy link

Member

Amanieu Jul 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a typo in the path.

Kobzol added 6 commits July 12, 2025 15:54

Move std_detect from library/stdarch to library

22e9ad1

Make std_detect a direct dependency of std

782a16f

Allow platform-specific code in std_detect

f93ac1c

Reformat std_detect

93d3d6a

Update stdarch README

081e277

Move tests around to appease tidy

0c16ed8

Fix warning

21c01a1

Kobzol force-pushed the std-detect-in-stdlib branch from 277837d to 92d9d7f Compare July 13, 2025 11:14

Kobzol mentioned this pull request Jul 13, 2025

[do not merge] Remove std_detect from CI rust-lang/stdarch#1873

Closed

Remove std_detect from stdarch examples

38130d7

Kobzol force-pushed the std-detect-in-stdlib branch from 92d9d7f to dc93317 Compare July 13, 2025 11:20

Kobzol mentioned this pull request Jul 13, 2025

Allow configuring upstream repository for pull rust-lang/josh-sync#12

Open

Remove std_detect from stdarch's CI

42d602b

Kobzol force-pushed the std-detect-in-stdlib branch from dc93317 to 42d602b Compare July 13, 2025 11:41

Kobzol changed the title ~~[do not merge] Move std_detect into stdlib~~ Move std_detect into stdlib Jul 13, 2025

Bless bootstrap tests

c0d26cc

Move std_detect into stdlib #143412

Are you sure you want to change the base?

Move std_detect into stdlib #143412

Conversation

Kobzol commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Build

Documentation

Tests

Outcome

Uh oh!

This comment has been minimized.

This comment has been minimized.

rustbot commented Jul 8, 2025

Uh oh!

Kobzol commented Jul 8, 2025

Uh oh!

cuviper commented Jul 8, 2025

Uh oh!

Kobzol commented Jul 8, 2025

Uh oh!

tgross35 commented Jul 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cuviper commented Jul 8, 2025

Uh oh!

cuviper commented Jul 8, 2025

Uh oh!

tgross35 commented Jul 8, 2025

Uh oh!

cuviper commented Jul 9, 2025

Uh oh!

bors commented Jul 11, 2025

Uh oh!

Kobzol commented Jul 11, 2025

Uh oh!

Amanieu left a comment

Choose a reason for hiding this comment

Uh oh!

Amanieu Jul 11, 2025

Choose a reason for hiding this comment

Uh oh!

Kobzol commented Jul 11, 2025

Uh oh!

rustbot commented Jul 13, 2025

Uh oh!

Kobzol commented Jul 13, 2025

Uh oh!

Uh oh!

Move `std_detect` into stdlib #143412

Move `std_detect` into stdlib #143412

Kobzol commented Jul 4, 2025 •

edited

Loading

tgross35 commented Jul 8, 2025 •

edited

Loading