Bootstrap failure: /usr/bin/ld is wrongly invoked from /opt/rh/.../gcc under Bazel

[nodakai]

Chances are that my environment is too non-standard... but where can I find a config not to use it? git grep -- -plugin didn't find anything. Is it hard-coded in a pre-built JAR or something? Moreover, my toolkit should accept -plugin. Anything wrong with my CROSSTOOL ?

$ bash compile.sh
INFO: You can skip this first step by providing a path to the bazel binary as second argument:
INFO:    compile.sh build /path/to/bazel
�  Building Bazel from scratch...........
�  Building Bazel with Bazel.
.WARNING: Output base '/home/nodakai/.cache/bazel/_bazel_nodakai/b9f9d0bfe2c30f0de6e9fab9bee6c1a2' is on NFS. This may lead to surprising failures and undetermined behavior.
Extracting Bazel installation...
Sending SIGTERM to previous Bazel server (pid=9423)... done.
......
INFO: Found 1 target...
INFO: From Linking src/main/tools/namespace-sandbox:
/usr/bin/ld: unrecognized option '-plugin'
/usr/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status
ERROR: /home/nodakai/src/bazel-HEAD/src/main/tools/BUILD:14:1: Linking of rule '//src/main/tools:namespace-sandbox' failed: gcc failed: error executing command /opt/rh/devtoolset-3/root/usr/bin/gcc -o bazel-out/local_linux-fastbuild/bin/src/main/tools/namespace-sandbox ... (remaining 10 argument(s) skipped): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.
Target //src:bazel failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 3.963s, Critical Path: 1.72s

Building output/bazel

$ echo 'int main(void){return 5;}' | strace -f -e execve /opt/rh/devtoolset-3/root/usr/bin/gcc -xc - && ./a.out; echo $?                                   (git)-master
execve("/opt/rh/devtoolset-3/root/usr/bin/gcc", ["/opt/rh/devtoolset-3/root/usr/bi"..., "-xc", "-"], [/* 59 vars */]) = 0
Process 29145 attached
[pid 29145] execve("/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.1/cc1", ["/opt/rh/devtoolset-3/root/usr/li"..., "-quiet", "-", "-quiet", "-dumpbase", "-", "-mtune=generic", "-march=x86-64", "-auxbase", "-", "-o", "/tmp/ccQuUjFT.s"], [/* 62 vars */]) = 0
[pid 29145] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=29145, si_status=0, si_utime=0, si_stime=0} ---
Process 29146 attached
[pid 29146] execve("/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.1/as", ["/opt/rh/devtoolset-3/root/usr/li"..., "--64", "-o", "/tmp/ccWsjGDv.o", "/tmp/ccQuUjFT.s"], [/* 62 vars */]) = 0
[pid 29146] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=29146, si_status=0, si_utime=0, si_stime=0} ---
Process 29147 attached
[pid 29147] execve("/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.1/collect2", ["/opt/rh/devtoolset-3/root/usr/li"..., "-plugin", "/opt/rh/devtoolset-3/root/usr/li"..., "-plugin-opt=/opt/rh/devtoolset-3"..., "-plugin-opt=-fresolution=/tmp/cc"..., "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_"..., "-plugin-opt=-pass-through=-lc", "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_"..., "--build-id", "--no-add-needed", "--eh-frame-hdr", "--hash-style=gnu", "-m", "elf_x86_64", ...], [/* 64 vars */]) = 0
Process 29148 attached
[pid 29148] execve("/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.1/ld", ["/opt/rh/devtoolset-3/root/usr/li"..., "-plugin", "/opt/rh/devtoolset-3/root/usr/li"..., "-plugin-opt=/opt/rh/devtoolset-3"..., "-plugin-opt=-fresolution=/tmp/cc"..., "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_"..., "-plugin-opt=-pass-through=-lc", "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_"..., "--build-id", "--no-add-needed", "--eh-frame-hdr", "--hash-style=gnu", "-m", "elf_x86_64", ...], [/* 64 vars */]) = 0
[pid 29148] +++ exited with 0 +++
[pid 29147] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=29148, si_status=0, si_utime=0, si_stime=0} ---
[pid 29147] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=29147, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++
5

$ git di tools/
diff --git a/tools/cpp/CROSSTOOL b/tools/cpp/CROSSTOOL
index 25368e9..89e6bac 100644
--- a/tools/cpp/CROSSTOOL
+++ b/tools/cpp/CROSSTOOL
@@ -71,12 +71,12 @@ toolchain {
   target_system_name: "local"
   toolchain_identifier: "local_linux"

-  tool_path { name: "ar" path: "/usr/bin/ar" }
-  tool_path { name: "compat-ld" path: "/usr/bin/ld" }
-  tool_path { name: "cpp" path: "/usr/bin/cpp" }
+  tool_path { name: "ar" path: "/opt/rh/devtoolset-3/root/usr/bin/ar" }
+  tool_path { name: "compat-ld" path: "/opt/rh/devtoolset-3/root/usr/bin/ld" }
+  tool_path { name: "cpp" path: "/opt/rh/devtoolset-3/root/usr/bin/cpp" }
   tool_path { name: "dwp" path: "/usr/bin/dwp" }
-  tool_path { name: "gcc" path: "/usr/bin/gcc" }
-  cxx_flag: "-std=c++0x"
+  tool_path { name: "gcc" path: "/opt/rh/devtoolset-3/root/usr/bin/gcc" }
+  cxx_flag: "-std=c++14"
   linker_flag: "-lstdc++"
   linker_flag: "-B/usr/bin/"

@@ -84,21 +84,21 @@ toolchain {
   # used by gcc. That works because bazel currently doesn't track files at
   # absolute locations and has no remote execution, yet. However, this will need
   # to be fixed, maybe with auto-detection?
-  cxx_builtin_include_directory: "/usr/lib/gcc/"
+  cxx_builtin_include_directory: "/opt/rh/devtoolset-3/root/usr/include"
   cxx_builtin_include_directory: "/usr/local/include"
   cxx_builtin_include_directory: "/usr/include"
   tool_path { name: "gcov" path: "/usr/bin/gcov" }

   # C(++) compiles invoke the compiler (as that is the one knowing where
   # to find libraries), but we provide LD so other rules can invoke the linker.
-  tool_path { name: "ld" path: "/usr/bin/ld" }
+  tool_path { name: "ld" path: "/opt/rh/devtoolset-3/root/usr/bin/ld" }

-  tool_path { name: "nm" path: "/usr/bin/nm" }
-  tool_path { name: "objcopy" path: "/usr/bin/objcopy" }
+  tool_path { name: "nm" path: "/opt/rh/devtoolset-3/root/usr/bin/nm" }
+  tool_path { name: "objcopy" path: "/opt/rh/devtoolset-3/root/usr/bin/objcopy" }
   objcopy_embed_flag: "-I"
   objcopy_embed_flag: "binary"
-  tool_path { name: "objdump" path: "/usr/bin/objdump" }
-  tool_path { name: "strip" path: "/usr/bin/strip" }
+  tool_path { name: "objdump" path: "/opt/rh/devtoolset-3/root/usr/bin/objdump" }
+  tool_path { name: "strip" path: "/opt/rh/devtoolset-3/root/usr/bin/strip" }

   # Anticipated future default.
   unfiltered_cxx_flag: "-no-canonical-prefixes"

[nodakai]

48029 execve("/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.1/collect2", ...
...
48032 execve("/usr/bin/ld", ["/usr/bin/ld", "-plugin", "/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.1/liblto_plugin.so", ...
48029 <... vfork resumed> )             = 48032

/opt/rh/devtoolset-3/.../collect2 execves /usr/bin/ld only under Bazel. What is happening?

[damienmg]

The line that does add -plugin to collect2 is the as call. It seems like an installation problem. Can you tell a bit more about your environment?

[nodakai]

Indeed I have both of /usr/bin/gcc AND /opt/rh/devtoolset-3/root/usr/bin/gcc but why does that matter? Does Bazel look up a command in $PATH by its own rule?

I have this in my ~/.bash_profile:

PATH=/opt/rh/devtoolset-3/root/usr/bin:$PATH

So I always invoke /opt/rh/.../ld except under Bazel:

$ bash -l -c 'which g++'
/opt/rh/devtoolset-3/root/usr/bin/g++
$ bash -l -c 'which ld'
/opt/rh/devtoolset-3/root/usr/bin/ld
$ /usr/bin/ld --version
GNU ld version 2.20.51.0.2-5.42.el6 20100205
Copyright 2009 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.
$ bash -l -c 'g++ -Wl,--version'
collect2 version 4.9.1 20140922 (Red Hat 4.9.1-10)
/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.1/ld -plugin /opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.1/liblto_plugin.so -plugin-opt=/opt/rh/devtoolset-3/root/usr/libexec/gcc/x86_64-redhat-linux/4.9.1/lto-wrapper -plugin-opt=-fresolution=/tmp/ccPIPYCs.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --build-id --no-add-needed --eh-frame-hdr --hash-style=gnu -m elf_x86_64 -dynamic-linker /lib64/ld-linux-x86-64.so.2 /usr/lib/../lib64/crt1.o /usr/lib/../lib64/crti.o /opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.1/crtbegin.o -L/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.1 -L/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.1/../../../../lib64 -L/lib/../lib64 -L/usr/lib/../lib64 -L/opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.1/../../.. --version -lstdc++ -lm -lgcc_s -lgcc -lc -lgcc_s -lgcc /opt/rh/devtoolset-3/root/usr/lib/gcc/x86_64-redhat-linux/4.9.1/crtend.o /usr/lib/../lib64/crtn.o
GNU ld version 2.24
Copyright 2013 Free Software Foundation, Inc.
This program is free software; you may redistribute it under the terms of
the GNU General Public License version 3 or (at your option) a later version.
This program has absolutely no warranty.

[damienmg]

Good point, what do you have in your environment variable? I think gcc is finding ld using environment variable and the environment is rewritten for action running in Bazel

[nodakai]

When I set my logging wrapper of GCC to tool_path,

  tool_path { name: "gcc" path: "/home/nodakai/local/bin/mygcc.sh" }

mygcc.sh:

#! /bin/sh

{
    printenv
    echo
    echo "$0" "$@"
} >> /tmp/mygcc-$$.log
/opt/rh/devtoolset-3/root/usr/bin/gcc "$@"

I got something like this:

PWD=/home/nodakai/.cache/bazel/_bazel_nodakai/b9f9d0bfe2c30f0de6e9fab9bee6c1a2/bazel-HEAD
SHLVL=1
_=/usr/bin/printenv

/home/nodakai/local/bin/mygcc.sh -o bazel-out/local_linux-fastbuild/bin/src/main/tools/build-runfiles bazel-out/local_linux-fastbuild/bin/src/main/tools/_objs/build-runfiles/src/main/tools/build-runfiles.pic.o -lstdc++ -B/usr/bin/ -pie -Wl,-z,relro,-z,now -no-canonical-prefixes -pass-exit-codes -Wl,--build-id=md5 -Wl,--hash-style=gnu -Wl,-S

PATH=/opt/rh/devtoolset-3/root/usr/bin:/home/nodakai/local/zsh-rh6/bin:/home/nodakai/local/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/nodakai/local/bin
PWD=/home/nodakai/.cache/bazel/_bazel_nodakai/b9f9d0bfe2c30f0de6e9fab9bee6c1a2/bazel-HEAD
SHLVL=1
INTERCEPT_LOCALLY_EXECUTABLE=1
_=/usr/bin/printenv

/home/nodakai/local/bin/mygcc.sh -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1 -fstack-protector -fPIE -Wall -Wunused-but-set-parameter -Wno-free-nonheap-object -fno-omit-frame-pointer -std=c++14 -Wpedantic -DBLAZE_OPENSOURCE -iquote . -iquote bazel-out/local_linux-fastbuild/genfiles -isystem tools/cpp/gcc3 -Wno-sign-compare -DBLAZE_JAVA_CPU="k8" -no-canonical-prefixes -Wno-builtin-macro-redefined -D__DATE__="redacted" -D__TIMESTAMP__="redacted" -D__TIME__="redacted" -frandom-seed=bazel-out/local_linux-fastbuild/bin/src/main/cpp/_objs/client/src/main/cpp/blaze_util_linux.pic.o -MD -MF bazel-out/local_linux-fastbuild/bin/src/main/cpp/_objs/client/src/main/cpp/blaze_util_linux.pic.d -fPIC -c src/main/cpp/blaze_util_linux.cc -o bazel-out/local_linux-fastbuild/bin/src/main/cpp/_objs/client/src/main/cpp/blaze_util_linux.pic.o

That is, all the invocations of mygcc.sh as a linker had almost empty environment variable whereas those as a C++ compiler had a few entries (but much fewer than what I see on my console.) Does this have something to do with shell/Command.java?

[hanwen]

all commands get a sanitized environment so invocations are portable across machines. If you need special tweaks, you could modify mygcc.sh to add the relevant varialbes.

[nodakai]

For the linking phase,

linker_flag: "-B/opt/rh/devtoolset-3/root/usr/bin"

was essential rather than

tool_path { name: "ld" path: "/opt/rh/devtoolset-3/root/usr/bin/ld" }