grpc Mac bazel build fails because of thread-local, but make passes

[vjpai]

Description of the problem / feature request / question:

I'm on the grpc/grpc project. We recently switched to using thread-local on one of our key structs. Although this project builds fine with make on Mac and Linux, it stopped building with bazel on Mac after we made this switch. Looking into it, it seems like tensorflow/serving also has the same issue.

If possible, provide a minimal example to reproduce the problem:

$ git clone git@github.com:grpc/grpc
$ cd grpc
$ bazel build //:grpc

Environment info

• Operating System: MacOS Sierra 10.12.6
• Bazel version (output of bazel info release): release 0.8.1-homebrew

Have you found anything relevant by searching the web?

Related issues are tensorflow/serving#1 and grpc/grpc#13856

Anything else, information or logs or outputs that would be helpful?

$ bazel build --verbose_failures //:grpc
INFO: Analysed target //:grpc (0 packages loaded).
INFO: Found 1 target...
ERROR: /Users/vpai/Git/grpc/BUILD:224:1: Linking of rule '//:grpc' failed (Exit 1): cc_wrapper.sh failed: error executing command 
  (cd /private/var/tmp/_bazel_vpai/5285458b308b3aadd65cb54a5ac76b0c/execroot/com_github_grpc_grpc && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=10.13 \
    PATH=/Users/vpai/google-cloud-sdk/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin \
    TMPDIR=/var/folders/xd/2k15ssh10lz6088_k_2lddtw007xtq/T/ \
    XCODE_VERSION_OVERRIDE=9.1.0 \
  external/local_config_cc/cc_wrapper.sh -fobjc-link-runtime -Wl,-S -shared -o bazel-out/darwin-fastbuild/bin/libgrpc.so bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/lib/surface/init.o bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/plugin_registry/grpc_plugin_registry.o -pthread -headerpad_max_install_names -lc++ -no-canonical-prefixes -undefined dynamic_lookup)

Use --sandbox_debug to see verbose messages from the sandbox
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: illegal thread local variable reference to regular symbol __ZN9grpc_core7ExecCtx9exec_ctx_E for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //:grpc failed to build
INFO: Elapsed time: 0.504s, Critical Path: 0.31s
FAILED: Build did NOT complete successfully

[hlopko]

Can you isolate the difference between your make and bazel link invocation? You can execute bazel with --subcommands to emit all command lines. You can then compare which options are different between your make build and bazel build.

[vjpai]

Hi there! Thanks for the response. Here's an example command from make that actually builds the relevant .o; note that generating the .a is just a matter of libtool in that case:

c++ -Ithird_party/protobuf/src -Ithird_party/googletest/googletest/include -Ithird_party/googletest/googlemock/include -Ithird_party/boringssl/include -Ithird_party/cares -Ithird_party/cares/cares -g -Wall -Wextra -Werror -Wno-long-long -Wno-unused-parameter -DOSATOMIC_USE_INLINED=1 -O2 -fPIC -I. -Iinclude -I/Users/vpai/Git/grpc/gens -I/usr/local/include -DNDEBUG -DINSTALL_PREFIX=\"/usr/local\"    -std=c++11 -stdlib=libc++ -fno-exceptions  -fno-rtti -fno-exceptions -MMD -MF /Users/vpai/Git/grpc/objs/opt/src/core/lib/iomgr/exec_ctx.dep -c -o /Users/vpai/Git/grpc/objs/opt/src/core/lib/iomgr/exec_ctx.o src/core/lib/iomgr/exec_ctx.cc

Here's the result of finding the version from the c++ command:

$ c++ -v
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

And here's what I get from bazel in building the object file (which is fine):

SUBCOMMAND: # //:grpc_base_c [action 'Compiling src/core/lib/iomgr/exec_ctx.cc']
(cd /private/var/tmp/_bazel_vpai/c6cc7641f7bf8c8335597635bcca2d28/execroot/com_github_grpc_grpc && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=10.13 \
    PATH=/Users/vpai/google-cloud-sdk/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin \
    TMPDIR=/var/folders/xd/2k15ssh10lz6088_k_2lddtw007xtq/T/ \
    XCODE_VERSION_OVERRIDE=9.1.0 \
  external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -O0 -DDEBUG '-std=c++11' -iquote . -iquote bazel-out/darwin-fastbuild/genfiles -iquote external/bazel_tools -iquote bazel-out/darwin-fastbuild/genfiles/external/bazel_tools -iquote external/com_google_absl -iquote bazel-out/darwin-fastbuild/genfiles/external/com_google_absl -iquote external/com_github_madler_zlib -iquote bazel-out/darwin-fastbuild/genfiles/external/com_github_madler_zlib -isystem include -isystem bazel-out/darwin-fastbuild/genfiles/include -isystem external/bazel_tools/tools/cpp/gcc3 -isystem external/com_github_madler_zlib -isystem bazel-out/darwin-fastbuild/genfiles/external/com_github_madler_zlib -MD -MF bazel-out/darwin-fastbuild/bin/_objs/grpc_base_c/src/core/lib/iomgr/exec_ctx.d '-frandom-seed=bazel-out/darwin-fastbuild/bin/_objs/grpc_base_c/src/core/lib/iomgr/exec_ctx.o' -D__CLANG_SUPPORT_DYN_ANNOTATION__ '-isysroot __BAZEL_XCODE_SDKROOT__' -no-canonical-prefixes -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -c src/core/lib/iomgr/exec_ctx.cc -o bazel-out/darwin-fastbuild/bin/_objs/grpc_base_c/src/core/lib/iomgr/exec_ctx.o)

And then what I get from bazel in actually doing the library build:

SUBCOMMAND: # //:grpc [action 'Linking libgrpc.so']
(cd /private/var/tmp/_bazel_vpai/c6cc7641f7bf8c8335597635bcca2d28/execroot/com_github_grpc_grpc && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=10.13 \
    PATH=/Users/vpai/google-cloud-sdk/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin \
    TMPDIR=/var/folders/xd/2k15ssh10lz6088_k_2lddtw007xtq/T/ \
    XCODE_VERSION_OVERRIDE=9.1.0 \
  external/local_config_cc/cc_wrapper.sh -fobjc-link-runtime -Wl,-S -shared -o bazel-out/darwin-fastbuild/bin/libgrpc.so bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/lib/surface/init.o bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/plugin_registry/grpc_plugin_registry.o -pthread -headerpad_max_install_names -lc++ -no-canonical-prefixes -undefined dynamic_lookup)
ERROR: /Users/vpai/Git/grpc-bazel-ready/BUILD:224:1: Linking of rule '//:grpc' failed (Exit 1): cc_wrapper.sh failed: error executing command 
  (cd /private/var/tmp/_bazel_vpai/c6cc7641f7bf8c8335597635bcca2d28/execroot/com_github_grpc_grpc && \
  exec env - \
    APPLE_SDK_PLATFORM=MacOSX \
    APPLE_SDK_VERSION_OVERRIDE=10.13 \
    PATH=/Users/vpai/google-cloud-sdk/bin:/usr/local/git/current/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin:/usr/local/go/bin \
    TMPDIR=/var/folders/xd/2k15ssh10lz6088_k_2lddtw007xtq/T/ \
    XCODE_VERSION_OVERRIDE=9.1.0 \
  external/local_config_cc/cc_wrapper.sh -fobjc-link-runtime -Wl,-S -shared -o bazel-out/darwin-fastbuild/bin/libgrpc.so bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/lib/surface/init.o bazel-out/darwin-fastbuild/bin/_objs/grpc/src/core/plugin_registry/grpc_plugin_registry.o -pthread -headerpad_max_install_names -lc++ -no-canonical-prefixes -undefined dynamic_lookup)

Use --sandbox_debug to see verbose messages from the sandbox
clang: warning: argument unused during compilation: '-pthread' [-Wunused-command-line-argument]
ld: illegal thread local variable reference to regular symbol __ZN9grpc_core7ExecCtx9exec_ctx_E for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
Target //:grpc failed to build

There are certainly differences but nothing seems to speak to thread-local variables as far as I can tell.

[vjpai]

We've had some further discussion about this in grpc/grpc#13856 (cc @yashykt) . I can make this work by using thread-local variables based on pthread_key functions rather than using the __thread declaration. This seems like what we would expect from a versioning issue. Is there a way to tell what version of clang is being used with bazel or any other versioning info that would help here?

[hlopko]

For osx-with-xcode toolchain it's a little bit complicated, you have to take a look into generated external/local_config_cc/cc_wrapper.sh file, you'll see the compiler invocation there.

[vjpai]

Hmm, found that and it's just an invocation of /usr/bin/clang which is the same as my standard c++

$ /usr/bin/clang -v
Apple LLVM version 9.0.0 (clang-900.0.38)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[vjpai]

I have a workaround, if that helps. We have been using thread-local for quite a while in gRPC (using our own portable macros that effectively expand to __thread on Mac) but this was our first use of that in a header file. grpc/grpc#13916 removes the use of that in a header file and keeps it exclusively in the matching source file, and bazel builds with that just fine on Mac.

We can use this workaround for now, but I hope that we get a longer-term solution that allows us to use thread-local in header files at some point. I know that we're not the only ones to see this issue when building with bazel on mac.