Bazel's use of -isystem gives incorrect search path on macOS

[xjia1]

Description of the problem:

Bazel's use of -isystem gives incorrect search path on macOS, specifically 10.15.2 (19C57) which is the system I have right now.

The observed behavior is that /usr/local/include is incorrectly searched before the directories local to external dependencies. If a dependency has two distinct versions both installed under /usr/local/include and specified in WORKSPACE as an external dependency, it probably won't compile nicely, or even if it compiles, the resulted binary is probably not what we intend it to be.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

1- Install latest version of protobuf with homebrew. This isn't specific to protobuf, but it happened to be one of the dependencies that I encountered.

% brew install protobuf
% brew info protobuf
protobuf: stable 3.11.2 (bottled), HEAD
...

2- Check out the protobuf examples. Again, this isn't specific to this example. I'm just too lazy to prepare and upload some code snippet somewhere.

% git clone https://github.com/protocolbuffers/protobuf.git
% cd protobuf/examples

3- Update WORKSPACE to something as follows. I'm just using an older version of protobuf to make the problem evident.

workspace(name = "com_google_protobuf_examples")
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "com_google_protobuf",
    strip_prefix = "protobuf-3.10.1",
    url = "https://github.com/protocolbuffers/protobuf/archive/v3.10.1.zip",
    sha256 = "678d91d8a939a1ef9cb268e1f20c14cd55e40361dc397bb5881e4e1e532679b1",
)
load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")
protobuf_deps()

4- Build the example. The extra flags to bazel are to make this output reproducible as much as I can.

% bazel build --jobs 1 --verbose_failures --sandbox_debug :add_person_cpp
Starting local Bazel server and connecting to it...
INFO: Analyzed target //:add_person_cpp (16 packages loaded, 560 targets configured).
INFO: Found 1 target...
ERROR: /private/var/tmp/_bazel_xjia/701b89b752af7feca00b1102287553da/external/com_google_protobuf/BUILD:164:1: C++ compilation of rule '@com_google_protobuf//:protobuf' failed (Exit 1) sandbox-exec failed: error executing command
  (cd /private/var/tmp/_bazel_xjia/701b89b752af7feca00b1102287553da/sandbox/darwin-sandbox/5/execroot/com_google_protobuf_examples && \
  exec env - \
    PATH=/usr/local/opt/bison/bin:/usr/local/lib/ruby/gems/2.6.0/bin:/usr/local/opt/ruby/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin \
    PWD=/proc/self/cwd \
    TMPDIR=/var/folders/tq/nxwms1d13d57l4ky75sp9fh80000gn/T/ \
  /usr/bin/sandbox-exec -f /private/var/tmp/_bazel_xjia/701b89b752af7feca00b1102287553da/sandbox/darwin-sandbox/5/sandbox.sb /var/tmp/_bazel_xjia/install/4f12cbbb3a294bfa314e578758559d54/process-wrapper '--timeout=0' '--kill_delay=15' external/local_config_cc/cc_wrapper.sh -U_FORTIFY_SOURCE -fstack-protector -Wall -Wthread-safety -Wself-assign -fcolor-diagnostics -fno-omit-frame-pointer '-std=c++0x' -MD -MF bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/_objs/protobuf/descriptor.pb.pic.d '-frandom-seed=bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/_objs/protobuf/descriptor.pb.pic.o' -fPIC -iquote external/com_google_protobuf -iquote bazel-out/darwin-fastbuild/bin/external/com_google_protobuf -iquote external/zlib -iquote bazel-out/darwin-fastbuild/bin/external/zlib -isystem external/com_google_protobuf/src -isystem bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/src -isystem external/zlib/zlib/include -isystem bazel-out/darwin-fastbuild/bin/external/zlib/zlib/include -DHAVE_PTHREAD -DHAVE_ZLIB -Woverloaded-virtual -Wno-sign-compare -Wno-unused-function -Wno-write-strings -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc -o bazel-out/darwin-fastbuild/bin/external/com_google_protobuf/_objs/protobuf/descriptor.pb.pic.o)
external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc:1489:52: error: out-of-line definition of 'InternalSerializeWithCachedSizesToArray' does not match any declaration in 'google::protobuf::FileDescriptorSet'
::PROTOBUF_NAMESPACE_ID::uint8* FileDescriptorSet::InternalSerializeWithCachedSizesToArray(
                                                   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc:1498:25: error: cannot initialize a parameter of type 'google::protobuf::uint8 *' (aka 'unsigned char *') with an rvalue of type '::google::protobuf::uint8 **' (aka 'unsigned char **')
    stream->EnsureSpace(&target);
                        ^~~~~~~
/usr/local/include/google/protobuf/io/coded_stream.h:688:54: note: passing argument to parameter 'ptr' here
  PROTOBUF_MUST_USE_RESULT uint8* EnsureSpace(uint8* ptr) {
                                                     ^
external/com_google_protobuf/src/google/protobuf/descriptor.pb.cc:1500:7: error: no member named 'InternalWriteMessageToArray' in 'google::protobuf::internal::WireFormatLite'; did you mean 'InternalWriteMessage'?
      InternalWriteMessageToArray(1, this->_internal_file(i), target, stream);
      ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      InternalWriteMessage
/usr/local/include/google/protobuf/wire_format_lite.h:1710:31: note: 'InternalWriteMessage' declared here
inline uint8* WireFormatLite::InternalWriteMessage(
                              ^
...

The exact errors are not significant to this issue. The important part is that the dependency-local files are #include-ing their corresponding header files with angle brackets <...> and the headers are incorrectly resolved to be under /usr/local/include.

What operating system are you running Bazel on?

macOS Catalina 10.15.2 (19C57)

What's the output of bazel info release?

release 2.0.0

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/protocolbuffers/protobuf.git
6263268b8c1b78a8a9b65acd6f5dd5c04dd9b0e1
6263268b8c1b78a8a9b65acd6f5dd5c04dd9b0e1

Have you found anything relevant by searching the web?

protocolbuffers/protobuf#5376 is definitely the same issue.

I did searches with "is:issue -isystem /usr/local/include" in the Bazel issues, but the results seemed too noisy for me to correlate any of them to this issue.

Any other information, logs, or outputs that you want to share?

I debugged this for a while, and my best guess so far is that there are two issues:

1- Bazel puts a space between -isystem and the path, which Clang doesn't seem like to take it that way. It also applies to -iquote.

2- The Clang I have here somehow adds "-I/usr/local/include" automatically. And since -I has higher precedence over -isystem, the search path will be wrong as long as we use -isystem for the external dependency directories.

To verify these two items, I simplified the bazel command line and tried these:

xjia@mba examples % cd bazel-examples
xjia@mba bazel-examples % pwd
/Users/xjia/protobuf/examples/bazel-examples

xjia@mba bazel-examples % /Library/Developer/CommandLineTools/usr/bin/cpp -v -iquote external/com_google_protobuf -isystem external/com_google_protobuf/src /dev/null -o /dev/null
Apple clang version 11.0.0 (clang-1100.0.33.16)
Target: x86_64-apple-darwin19.2.0
...
clang: warning: argument unused during compilation: '-traditional' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-iquote external/com_google_protobuf' [-Wunused-command-line-argument]
Apple clang version 11.0.0 (clang-1100.0.33.16)
Target: x86_64-apple-darwin19.2.0
...
clang -cc1 version 11.0.0 (clang-1100.0.33.16) default target x86_64-apple-darwin19.2.0
ignoring nonexistent directory "-isystem"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/include
 /Library/Developer/CommandLineTools/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
error: error reading 'external/com_google_protobuf/src'
1 error generated.

xjia@mba bazel-examples % /Library/Developer/CommandLineTools/usr/bin/cpp -v -iquote external/com_google_protobuf -iquote bazel-out/darwin-fastbuild/bin/external/com_google_protobuf -isystem external/com_google_protobuf/src /dev/null -o /dev/null
...
clang: warning: argument unused during compilation: '-traditional' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-iquote external/com_google_protobuf' [-Wunused-command-line-argument]
...
ignoring nonexistent directory "-iquote"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/include
 /Library/Developer/CommandLineTools/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
error: error reading 'bazel-out/darwin-fastbuild/bin/external/com_google_protobuf'
1 error generated.

xjia@mba bazel-examples % /Library/Developer/CommandLineTools/usr/bin/cpp -v -iquote'external/com_google_protobuf' -isystem'external/com_google_protobuf/src' /dev/null -o /dev/null
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/Library/Frameworks"
#include "..." search starts here:
 external/com_google_protobuf
#include <...> search starts here:
 /usr/local/include
 external/com_google_protobuf/src
 /Library/Developer/CommandLineTools/usr/lib/clang/11.0.0/include
 /Library/Developer/CommandLineTools/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/System/Library/Frameworks (framework directory)
End of search list.
# 1 "/dev/null"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 361 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "/dev/null" 2

Apple clang version 11.0.0 (clang-1100.0.33.16)
Target: x86_64-apple-darwin19.2.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
clang: warning: argument unused during compilation: '-traditional' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-iquote external/com_google_protobuf' [-Wunused-command-line-argument]
clang: warning: argument unused during compilation: '-isystem external/com_google_protobuf/src' [-Wunused-command-line-argument]
xjia@mba bazel-examples % echo $?
0

I used /Library/Developer/CommandLineTools/usr/bin/cpp in those commands because I was also suspecting https://github.com/Homebrew/brew/blob/master/Library/Homebrew/shims/super/cc but it looks innocent to me now.

[xjia1]

FWIW this happened to me with previous bazel versions as well.

[rupertks]

Possibly relevant: https://groups.google.com/d/msg/bazel-discuss/V6ZsSsDwS9w/LMnBxiWaCgAJ

[susinmotion]

Closing as a duplicate of #8984, please reopen if necessary.

[evanj]

See also issue #8053 which includes the workaround of using build --sandbox_block_path=/usr/local in a .bazelrc to get bazel to exclude those includes.

[cpsauer]

Guys, could we reopen this?

(The duplicate-ish issue is closed as stale; it's breaking google/boringssl on bazelci (we just spent a bunch of time on this over at rules_boost) ; and IMO the discussion here better captures the issue from the user perspective -> the implementation problem.)

Some more context: boringssl currently fails on bazelci macOS x86_64, similar to protobuf before. Logs look like this, with header inclusion tearing between what I presume is brew installed openssl and //external/boringssl (named as openssl in the paths).

In file included from external/boringssk/src/crypto/x509/t_crl.c:61:
external/boringssl/src/include/openssl/mem.h:87:22: error: conflicting types for 'CRYPTO_zalloc'
OPENSSL_EXPORT void *OPENSSL_zalloc(size_t size);
                     ^
/usr/local/include/openssl/crypto.h:99:9: note: expanded from macro 'OPENSSL_zalloc'
        CRYPTO_zalloc(num, OPENSSL_FILE, OPENSSL_LINE)
        ^
/usr/local/include/openssl/crypto.h:346:25: note: previous declaration is here
OSSL_CRYPTO_ALLOC void *CRYPTO_zalloc(size_t num, const char *file, int line);
                        ^

IMO this really shouldn't happen. As y'all say, Bazel-included external repos should really take precedence over system-installed libraries.

And they would, except that Apple clang auto inserts -I/usr/local/include after the ones specified, as above.
(You can double check this with, e.g., clang -c whatever.cpp -v -Iexternal/boringssl -o /dev/null)

As in the discussion linked above, I find myself thinking we should probably have includes produce -I on Apple toolchains to work around this, so it takes precedence but is still in the system includes list, similar to what it sounds like is already done with MSVC. What do you think?

(cc'ing @philwo and @hlopko and @rikbrown and @thundergolfer from the other issue in an attempt to unify. Also @keith bc, well, he's so fantastically great and helpful on all things Apple. Wishing all a great holiday in the meantime.)

[cpsauer]

Thanks, Fabian! :)

[keith]

You can double check this with, e.g., clang -c whatever.cpp -v -Iexternal/boringssl -o /dev/null

With this command I see -I external/boringssl -I/usr/local/include, is this the incorrect order? or which flag specifically causes today's issue?

[cpsauer]

^ The issue is that Apple clang adds that -I/usr/local/include regardless of flags, since it takes prescidence over the -isystems added by Bazel's includes attribute.

I think the fixes--assuming we want to support Intel Apple or /usr/local/include are:
(1) Make includes emit -I instead of -isystem, parallel to Windows (but not linux) as above.
(2) [New and maybe better.] I think if we can insert a -isystem/usr/local/include at the end of the invocation, that'll cause the -I/usr/local/include to be ignored and demoted to just the -isystem/usr/local/include (clang docs, and checked out in a quick experiment.)

[keith]

Did that work? (for a quick workaround folks could do that locally in their projects)

[cpsauer]

(I'm assuming you're asking about (2), i.e. adding build --copt=-isystem/usr/local/include to .bazelrc, right Keith?)

I just did some tests against the original rules_boost case that brought me here, since bazelci made it easy to test on both Mac architectures :) (Backlinked above)

The principle behind suppressing with -isystem/usr/local/include works; as hypothesized, it works around the issue on the macOS x86_64 build where /usr/local/include exists, causing it to correctly prefer the bazel headers to those in /usr/local/include.

However on arm64 (where it's unneeded in that test environment, since there's no /usr/local/include anyway) it gets prevented by Bazel (The include path '/usr/local/include' references a path outside of the execution root.) Not sure why just there; you probably know better from working on the toolchains. So it seems like the fix would need to get woven into bazel.

Does that seem like the right way to go, Keith? Do you know where and how to weave this in from your prior work? If so, would you be down to just do?

[As an aside, for me personally, the build --sandbox_block_path=/usr/local-in-a-.bazelrc workaround is really the right permanent move; I want the Mac software I build to be portable, rather than dependent on brew installs.]
[Also, could someone with the power to do so update the labels if that's important?]

[thii]

^ The issue is that Apple clang adds that -I/usr/local/include regardless of flags, since it takes prescidence over the -isystems added by Bazel's includes attribute.

You could add -nostdinc and -nostdinc++ to your toolchain and only include what you want.

[keith]

I looked at that too but it also removes some important paths we'd have to recreate

[mering]

Having the some problem on BCR. Workaround worked in bazelbuild/bazel-central-registry#1190.