CI segfaults on FreeBSD

[gnzlbg]

CI has started to segfault on FreeBSD since the latest nightly.

This build passed with nightly-x86_64-unknown-freebsd 1.39.0 (760226733 2019-08-22)
This build failed with nightly-x86_64-unknown-freebsd 1.39.0 (9eae1fc0e 2019-08-23)

The segfault is triggered by the build.rs of rand_pcg v0.1.2 and rand_chacha v0.1.1 (cc @dhardy), which AFAICT only call autocfg (cc @cuviper ). There was an update of autocfg recently so initially I thought that might be it, but it seems that the builds were working correctly with that update, so AFAICT neither autocfg nor rand are at fault here (cc @mati865 @asomers ). Looking at the recently merged PRs in rust-lang/rust, I don't see any suspicious one.

[mati865]

I'm mostly unavailable until Monday/Tuesday.
Without backtrace it'll be hard to find the reason. Maybe you could run GDB on that CI or even locally?

[gnzlbg]

I have a minimal small-ish reproducer now: #1490

# Cargo.toml
[package]
name = "test_failure"
version = "0.1.0"
authors = ["The Rust Project Developers"]
license = "MIT OR Apache-2.0"

[build-dependencies]
autocfg = "0.1"

and build.rs:

extern crate autocfg;

fn main() {
    println!("cargo:rerun-if-changed=build.rs");
    let ac = autocfg::new();
    ac.emit_rustc_version(1, 26);
}

---

This fails on FreeBSD12, but not on FreeBSD11. A backtrace of the segfault would be nice.

[asomers]

I'll take a look.

[asomers]

Something is calling fstatat, and it's calling the FreeBSD 12 version but probably expecting the FreeBSD 11 structure definitions. It screws up the stack so the core file's stack trace is useless, but I think this is the culprit:

#0  0x00000008012df594 in stat () from /lib/libc.so.7
#1  0x0000000001065f60 in std::sys::unix::fs::stat ()
    at /checkout/src/libstd/sys/unix/fs.rs:736
#2  0x0000000001054d47 in std::fs::metadata (path=0x7fffffffe1a8)
    at /checkout/src/libstd/fs.rs:1398
#3  0x000000000104609d in autocfg::AutoCfg::with_dir (dir=...)
    at /usr/home/somers/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-0.1.6/src/lib.rs:143
#4  0x0000000001045d8a in autocfg::AutoCfg::new ()
    at /usr/home/somers/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-0.1.6/src/lib.rs:123
#5  0x0000000001045c92 in autocfg::new ()
    at /usr/home/somers/.cargo/registry/src/github.com-1ecc6299db9ec823/autocfg-0.1.6/src/lib.rs:109
#6  0x0000000001045696 in build_script_build::main () at build.rs:5

Why isn't libc using the right link_name?

[gnzlbg]

@asomers

Why isn't libc using the right link_name?

Can you submit a PR fixing that ?

[asomers]

I'm still trying to figure out what's going on. I verified that the 8-22 nightly compiler works and the 8-23 nightly doesn't, but I don't see any obviously relevant changes in that date range. rust-lang/rust's liblibc does use the correct link_name. The problem is something more subtle. I'll keep investigating .

[cuviper]

The only thing in rust-lang/rust@7602267...9eae1fc that looks relevant is the libc upgrade, 0.2.61...0.2.62 0.2.60...0.2.61, which includes the FreeBSD changes of #1467.

The new match which_freebsd() here is emitting cargo:rustc-cfg=freebsd11 even for the unknown case, but Rust CI is building for FreeBSD 10.3.

https://github.com/rust-lang/rust/blob/eeba189cfb2cfc5c5898513352d4ca8f1df06e05/src/ci/docker/scripts/freebsd-toolchain.sh#L8

[cuviper]

Since fstatat was specifically mentioned earlier:

libc/src/unix/mod.rs

Lines 632 to 638 in 9af04ce

	#[cfg_attr(target_os = "macos", link_name = "fstatat$INODE64")]
	#[cfg_attr(
	all(target_os = "freebsd", freebsd11),
	link_name = "fstatat@FBSD_1.1"
	)]
	pub fn fstatat(dirfd: ::c_int, pathname: *const ::c_char,
	buf: *mut stat, flags: ::c_int) -> ::c_int;

So forcing freebsd11 also forces that link name, but what happens when this is built against FreeBSD 10 on Rust CI? Does it actually end up linking without the specific version?

I think the reason this hit autocfg in particular is just that it's used in a build script, and those are built with -Cprefer-dynamic, so it will link to libstd.so that was itself linked in Rust CI's environment. (Versus regular builds using libstd.rlib that will link locally.)

[gnzlbg]

Does it actually end up linking without the specific version?

Isn't the C library dynamically linked here ? As in, shouldn't it pick the FreeBSD12 C library and try to find there the fstatat@FBSD_1.1 symbol ? I'm not sure what happens if in the C library using during the Rust CI build doesn't have this symbol.

So forcing freebsd11 also forces that link name, but what happens when this is built against FreeBSD 10 on Rust CI?

So this is the bug. We should add a freebsd10 cfg macro until Rust CI is upgraded to FreeBSD 11, and only enable that when building as part of libstd. Cirrus-CI has a freebsd-10-4-release-amd64 image, we should add that to CI to reproduce this and test that no regressions are introduced.

[asomers]

So this is the bug. We should add a freebsd10 cfg macro until Rust CI is upgraded to FreeBSD 11, and only enable that when building as part of libstd. Cirrus-CI has a freebsd-10-4-release-amd64 image, we should add that to CI to reproduce this and test that no regressions are introduced.

No, I don't think that's it. Very few symbols changed between FreeBSD 10 and 11, and fstatat was not among them. Could LIBC_CI be getting set somehow?

[gnzlbg]

I don't know how - searching rust-lang/rust for LIBC_CI doesn't reveal anything.

I set up a freebsd10 build job here (#1491) and tried to use the different link names, but that doesn't appear to make a difference.

[asomers]

I can reproduce the problem without using autocfg or build.rs. But I still don't know why Rust is using the wrong symbol version. Could it be building libc without using the build.rs at all? That would do it.

use std::path::PathBuf;

#[test]
fn t() {
    let pb = PathBuf::from(".");
    println!("{:?}", std::fs::metadata(&pb).unwrap());
}

[gnzlbg]

I can reproduce the problem without using autocfg or build.rs. But I still don't know why Rust is using the wrong symbol version.

First, what's the wrong symbol ? I suppose that the bug is that instead of picking the symbol that we use for FreeBSD11, the fstatat symbol (generic one) gets used, and that happens to point to a different symbol than the FreeBSD11 one in FreeBSD12. Is that it?

If so, then #1491 should fix that. It detects FreeBSD10, and uses the same symbols as FreeBSD11. It's not a great fix, but might do.