cross-compiling regression

[classabbyamp]

When cross-compiling openssl-sys from x86_64-unknown-linux-gnu to aarch64-unknown-linux-gnu (and other host/target combinations) with cc 1.2.15 or higher, cc fails to use the correct compiler, leading to compilation failures because it uses the host's compiler with the target's cflags.

By bisecting, I have determined the issue was caused by #1401.

Build log showing failure

warning: openssl-sys@0.9.108: cc1: error: bad value 'armv8-a' for '-march=' switch
warning: openssl-sys@0.9.108: cc1: note: valid arguments to '-march=' switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 cannonlake icelake-client rocketlake icelake-server cascadelake tigerlake cooperlake sapphirerapids emeraldrapids alderlake raptorlake meteorlake graniterapids graniterapids-d arrowlake arrowlake-s lunarlake pantherlake bonnell atom silvermont slm goldmont goldmont-plus tremont gracemont sierraforest grandridge clearwaterforest knl knm x86-64 x86-64-v2 x86-64-v3 x86-64-v4 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 lujiazui yongfeng k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 znver2 znver3 znver4 znver5 btver1 btver2 native
error: failed to run custom build command for `openssl-sys v0.9.108`

Caused by:
  process didn't exit successfully: `/builddir/sequoia-sq-1.3.1/target/release/build/openssl-sys-03b8be8f4cdf3f69/build-script-main` (exit status: 101)
  --- stdout
  cargo:rustc-check-cfg=cfg(osslconf, values("OPENSSL_NO_OCB", "OPENSSL_NO_SM4", "OPENSSL_NO_SEED", "OPENSSL_NO_CHACHA", "OPENSSL_NO_CAST", "OPENSSL_NO_IDEA", "OPENSSL_NO_CAMELLIA", "OPENSSL_NO_RC4", "OPENSSL_NO_BF", "OPENSSL_NO_PSK", "OPENSSL_NO_DEPRECATED_3_0", "OPENSSL_NO_SCRYPT", "OPENSSL_NO_SM3", "OPENSSL_NO_RMD160", "OPENSSL_NO_EC2M", "OPENSSL_NO_OCSP", "OPENSSL_NO_CMS", "OPENSSL_NO_COMP", "OPENSSL_NO_SOCK", "OPENSSL_NO_STDIO", "OPENSSL_NO_EC", "OPENSSL_NO_SSL3_METHOD", "OPENSSL_NO_KRB5", "OPENSSL_NO_TLSEXT", "OPENSSL_NO_SRP", "OPENSSL_NO_RFC3779", "OPENSSL_NO_SHA", "OPENSSL_NO_NEXTPROTONEG", "OPENSSL_NO_ENGINE", "OPENSSL_NO_BUF_FREELISTS", "OPENSSL_NO_RC2"))
  cargo:rustc-check-cfg=cfg(openssl)
  cargo:rustc-check-cfg=cfg(libressl)
  cargo:rustc-check-cfg=cfg(boringssl)
  cargo:rustc-check-cfg=cfg(awslc)
  cargo:rustc-check-cfg=cfg(libressl250)
  cargo:rustc-check-cfg=cfg(libressl251)
  cargo:rustc-check-cfg=cfg(libressl252)
  cargo:rustc-check-cfg=cfg(libressl261)
  cargo:rustc-check-cfg=cfg(libressl270)
  cargo:rustc-check-cfg=cfg(libressl271)
  cargo:rustc-check-cfg=cfg(libressl273)
  cargo:rustc-check-cfg=cfg(libressl280)
  cargo:rustc-check-cfg=cfg(libressl281)
  cargo:rustc-check-cfg=cfg(libressl291)
  cargo:rustc-check-cfg=cfg(libressl310)
  cargo:rustc-check-cfg=cfg(libressl321)
  cargo:rustc-check-cfg=cfg(libressl332)
  cargo:rustc-check-cfg=cfg(libressl340)
  cargo:rustc-check-cfg=cfg(libressl350)
  cargo:rustc-check-cfg=cfg(libressl360)
  cargo:rustc-check-cfg=cfg(libressl361)
  cargo:rustc-check-cfg=cfg(libressl370)
  cargo:rustc-check-cfg=cfg(libressl380)
  cargo:rustc-check-cfg=cfg(libressl381)
  cargo:rustc-check-cfg=cfg(libressl382)
  cargo:rustc-check-cfg=cfg(libressl390)
  cargo:rustc-check-cfg=cfg(libressl400)
  cargo:rustc-check-cfg=cfg(libressl410)
  cargo:rustc-check-cfg=cfg(ossl101)
  cargo:rustc-check-cfg=cfg(ossl102)
  cargo:rustc-check-cfg=cfg(ossl102f)
  cargo:rustc-check-cfg=cfg(ossl102h)
  cargo:rustc-check-cfg=cfg(ossl110)
  cargo:rustc-check-cfg=cfg(ossl110f)
  cargo:rustc-check-cfg=cfg(ossl110g)
  cargo:rustc-check-cfg=cfg(ossl110h)
  cargo:rustc-check-cfg=cfg(ossl111)
  cargo:rustc-check-cfg=cfg(ossl111b)
  cargo:rustc-check-cfg=cfg(ossl111c)
  cargo:rustc-check-cfg=cfg(ossl111d)
  cargo:rustc-check-cfg=cfg(ossl300)
  cargo:rustc-check-cfg=cfg(ossl310)
  cargo:rustc-check-cfg=cfg(ossl320)
  cargo:rustc-check-cfg=cfg(ossl330)
  cargo:rustc-check-cfg=cfg(ossl340)
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_LIB_DIR
  OPENSSL_LIB_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_INCLUDE_DIR
  OPENSSL_INCLUDE_DIR unset
  cargo:rerun-if-env-changed=X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR
  X86_64_UNKNOWN_LINUX_GNU_OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_DIR
  OPENSSL_DIR unset
  cargo:rerun-if-env-changed=OPENSSL_NO_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=OPENSSL_STATIC
  cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=SYSROOT
  cargo:rerun-if-env-changed=OPENSSL_STATIC
  cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rustc-link-lib=ssl
  cargo:rustc-link-lib=crypto
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG
  cargo:rerun-if-env-changed=PKG_CONFIG
  cargo:rerun-if-env-changed=OPENSSL_STATIC
  cargo:rerun-if-env-changed=OPENSSL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_STATIC
  cargo:rerun-if-env-changed=PKG_CONFIG_ALL_DYNAMIC
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_PATH
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_LIBDIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64-unknown-linux-gnu
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR_x86_64_unknown_linux_gnu
  cargo:rerun-if-env-changed=HOST_PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-env-changed=PKG_CONFIG_SYSROOT_DIR
  cargo:rerun-if-changed=build/expando.c
  OPT_LEVEL = Some(0)
  OUT_DIR = Some(/builddir/sequoia-sq-1.3.1/target/release/build/openssl-sys-fba1018ce15cddbc/out)
  TARGET = Some(x86_64-unknown-linux-gnu)
  HOST = Some(x86_64-unknown-linux-gnu)
  cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-gnu
  CC_x86_64-unknown-linux-gnu = None
  cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_gnu
  CC_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=HOST_CC
  HOST_CC = Some(gcc)
  cargo:rerun-if-env-changed=CC_KNOWN_WRAPPER_CUSTOM
  CC_KNOWN_WRAPPER_CUSTOM = None
  RUSTC_WRAPPER = None
  cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
  cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some(true)
  CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
  cargo:rerun-if-env-changed=CFLAGS
  CFLAGS = Some(-fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv8-a   -I/usr/aarch64-linux-gnu/usr/include -ffile-prefix-map=/builddir/sequoia-sq-1.3.1=.)
  cargo:rerun-if-env-changed=CC_SHELL_ESCAPED_FLAGS
  CC_SHELL_ESCAPED_FLAGS = None
  cargo:rerun-if-env-changed=HOST_CFLAGS
  HOST_CFLAGS = Some(-O2)
  cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
  CFLAGS_x86_64_unknown_linux_gnu = None
  cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
  CFLAGS_x86_64-unknown-linux-gnu = None
  CARGO_ENCODED_RUSTFLAGS = Some()
  cargo:warning=cc1: error: bad value 'armv8-a' for '-march=' switch
  cargo:warning=cc1: note: valid arguments to '-march=' switch are: nocona core2 nehalem corei7 westmere sandybridge corei7-avx ivybridge core-avx-i haswell core-avx2 broadwell skylake skylake-avx512 cannonlake icelake-client rocketlake icelake-server cascadelake tigerlake cooperlake sapphirerapids emeraldrapids alderlake raptorlake meteorlake graniterapids graniterapids-d arrowlake arrowlake-s lunarlake pantherlake bonnell atom silvermont slm goldmont goldmont-plus tremont gracemont sierraforest grandridge clearwaterforest knl knm x86-64 x86-64-v2 x86-64-v3 x86-64-v4 eden-x2 nano nano-1000 nano-2000 nano-3000 nano-x2 eden-x4 nano-x4 lujiazui yongfeng k8 k8-sse3 opteron opteron-sse3 athlon64 athlon64-sse3 athlon-fx amdfam10 barcelona bdver1 bdver2 bdver3 bdver4 znver1 znver2 znver3 znver4 znver5 btver1 btver2 native

  --- stderr

  thread 'main' panicked at /host/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/openssl-sys-0.9.108/build/main.rs:315:13:

  Header expansion error:
  Error { kind: ToolExecError, message: "command did not execute successfully (status code exit status: 1): LC_ALL=\"C\" \"gcc\" \"-O0\" \"-ffunction-sections\" \"-fdata-sections\" \"-fPIC\" \"-gdwarf-4\" \"-fno-omit-frame-pointer\" \"-m64\" \"-I\" \"/usr/aarch64-linux-gnu/usr/include\" \"-fstack-clash-protection\" \"-D_FORTIFY_SOURCE=2\" \"-O2\" \"-pipe\" \"-march=armv8-a\" \"-I/usr/aarch64-linux-gnu/usr/include\" \"-ffile-prefix-map=/builddir/sequoia-sq-1.3.1=.\" \"-O2\" \"-E\" \"build/expando.c\"" }

  Failed to find OpenSSL development headers.

  You can try fixing this setting the `OPENSSL_DIR` environment variable
  pointing to your OpenSSL installation or installing OpenSSL headers package
  specific to your distribution:

      # On Ubuntu
      sudo apt-get install pkg-config libssl-dev
      # On Arch Linux
      sudo pacman -S pkgconf openssl
      # On Fedora
      sudo dnf install pkgconf perl-FindBin perl-IPC-Cmd openssl-devel
      # On Alpine Linux
      apk add pkgconf openssl-dev

  See rust-openssl documentation for more information:

      https://docs.rs/openssl

  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...

[madsmtm]

The problem is that setting CFLAGS="-fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv8-a -I/usr/aarch64-linux-gnu/usr/include -ffile-prefix-map=/builddir/sequoia-sq-1.3.1=." is not enough for cc-rs to know which target those flags apply to. So we have to assume that it applies to all configurations. Your setup only worked before because you happened to set HOST_CFLAGS, but that was brittle.

I will close this as intended behaviour, you should use one of the following workarounds instead:

1. Set HOST_CFLAGS in a manner that "unset"s the flags from before. Something like HOST_CFLAGS="-fno-stack-clash-protection -U_FORTIFY_SOURCE -march=x86-64", depending on exact compiler errors you're seeing. Might not always be possible.
2. Set CFLAGS_aarch64_unknown_linux_gnu instead of CFLAGS.

[classabbyamp]

HOST_CFLAGS was already being set to -O2

if CFLAGS_$target is set (and CFLAGS is also set), it also does not work. Why does the more specific variable not override CFLAGS? That would make sense, instead of requiring that it be unset.

is this behaviour documented somewhere other than the build.rs scripts?

[madsmtm]

Rethinking a bit, maybe I am wrong here? The old behaviour somewhat matched build.rustflags in Cargo, though that has quite confusing behaviour too (it seems to be dependent on the --target flag).

For a bit more prior art, we have Bazel's --copt and --host_copt flags, I'm pretty sure that --copt only applies to target binaries.

WDYT @NobodyXu? Reopening for discussion.

[madsmtm]

Let me outline various scenarios, might make it easier to talk about what the desired behaviour is.

There are effectively three ways of invoking Cargo:

1. With --target $TARGET.
2. With --target $HOST.
3. Without target flag.

And cc-rs can be used in a few places:

1. In build.rs script for a crate that will be used in the application itself.
2. In build.rs script for a crate compiled for the host (such as a proc-macro dependency).
3. Outside build.rs, as part of the application itself.

[madsmtm]

Perhaps the desired behaviour would be:

let cflags = if cargo_target_flag_is_specified {
    if cross_compile {
        CFLAGS + CFLAGS_$TARGET
    } else {
        CFLAGS_HOST + CFLAGS_$TARGET
    }
} else {
    CFLAGS + CFLAGS_HOST + CFLAGS_$TARGET
};

But I don't think that is implementable with the information that Cargo provides us.

[NobodyXu]

Hmm so I see that the old behavior is more reasonable for some use cases, so maybe we can introduce CFLAGS_$target_OVERRIDE to override default CFLAGS? I don't want to change the semantics of CFLAGS_$target again as it might break someone again.

[NobodyXu]

But I don't think that is implementable with the information that Cargo provides us.

Is that because cargo_target_flag_is_specified is hard to probe?

We do have code for probing cross compilation

cargo_target_flag_is_specified

[madsmtm]

Yeah, only Cargo knows whether the --target flag has been specified (and that's probably for the best).

Hmm so I see that the old behavior is more reasonable for some use cases, so maybe we can introduce CFLAGS_$target_OVERRIDE to override default CFLAGS? I don't want to change the semantics of CFLAGS_$target again as it might break someone again.

I'd really like to avoid introducing yet another obscure env var. I'd rather revert to the old behaviour if we decide that's how we want it, I'm not too worried about the breakage from that, it's recent enough that I'm fairly sure it's only rustc's bootstrap that relies on it (and I'll fix that before landing any such revert).

[classabbyamp]

In our (Void Linux's) build system, when cross-compiling, the CFLAGS variable is set to the target's CFLAGS, as that is the primary CFLAGS programs and libraries should be using.

For various kinds of build systems, we also set:

CFLAGS_FOR_BUILD=-O2 -pipe
CFLAGS_target=-O2 -pipe -march=armv8-a  
BUILD_CFLAGS=-O2 -pipe
HOST_CFLAGS=-O2
CFLAGS=-fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv8-a   -I/usr/aarch64-linux-gnu/usr/include -ffile-prefix-map=/builddir/sequoia-sq-1.3.1=.
CFLAGS_host=-O2 -pipe

but CFLAGS contains all the flags that should be used for the target.

It's probably reasonable to assume that CFLAGS are for the target if something like HOST_CFLAGS or CFLAGS_host exists

[leleliu008]

I also face this problem. I am a package manager maintainer. I need to build some packages that use both trodinational build systems (e.g. cmake, autotools, etc) and cargo, these trodinational systems use CFLAGS, so we must set it. but if we set it. both target and host use it. it might break the host build.

I think it's better to choose only one in order:

https://docs.rs/cc/latest/cc/

Each of these variables can also be supplied with certain prefixes and suffixes, in the following prioritized order:

<var>_<target> - for example, CC_x86_64-unknown-linux-gnu
<var>_<target_with_underscores> - for example, CC_x86_64_unknown_linux_gnu
<build-kind>_<var> - for example, HOST_CC or TARGET_CFLAGS
<var> - a plain CC, AR as above.

[weihanglo]

From #1401

This has the slight chance of breaking builds where users assume that CFLAGS_$TARGET overwrites CFLAGS. I will argue that that is likely to be the minority, relative to users that do want flags from all CFLAGS* env vars.

This also broke our build system.

We have a std-bootstrap package that holds the rust-lang/rust std bootstrap logic. In order to compiling bootstrap itself, it sets CFLAGS for host. Then, to build std for each target platform, a platform-specific package would call the std-bootstrap package with CFLAGS_<target> and associated boostrap.toml values. Merging CFLAGS and CFLAGS_<target> is undesired to us because it leaks -march to target platform.

[arielb1]

@weihanglo

Are you sure it's not better to set HOST_CFLAGS for the host flags? Maybe make bootstrap do export CFLAGS="${HOST_CFLAGS:-CFLAGS}" when compiling itself?

[briansmith]

I think the change should be reverted and a new release should be made. Then we can have a design decision.

(This has broken ring's CI as well.)

[Edit: I am sorry I left this comment. I was mistaken.]