Cross compiling with CMake for Android #289

keith · 2019-07-08T20:59:35Z

Currently when building a project for Android, the rules_foreign_cc build seems to still think the build is happening for macOS. This results in downstream CMake config not being correctly set if they are changing behavior based on the fact that they are cross compiling, or if some CMake variables like ANDROID are set.

The easiest way to reproduce this is to add this:

if (NOT ANDROID)
  message(FATAL_ERROR "Expected ANDROID to be set, system name is: ${CMAKE_SYSTEM_NAME}")
endif()

at the bottom of cmake_hello_world_lib/static/src/CMakeLists.txt and then run bazel build //cmake_android:app. At this point if you cd into the BUILD_TMPDIR you can also see that CMake run check commands like this one:

/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/toolchains/llvm/prebuilt/darwin-x86_64/bin/clang   \
 -D__ANDROID_API__=28 \
 -isystem/private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/sysroot/usr/include/arm-linux-androideabi \
 -target armv7-none-linux-androideabi \
 -march=armv7-a \
 -mfloat-abi=softfp \
 -mfpu=vfpv3-d16 \
 -gcc-toolchain /private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/toolchains/arm-linux-androideabi-4.9/prebuilt/darwin-x86_64 \
 -fpic \
 -no-canonical-prefixes \
 -Wno-invalid-command-line-argument \
 -Wno-unused-command-line-argument \
 -funwind-tables \
 -fstack-protector-strong \
 -fno-addrsig \
 -Werror=return-type \
 -Werror=int-to-pointer-cast \
 -Werror=pointer-to-int-cast \
 -Werror=implicit-function-declaration  \
 --sysroot=/private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/platforms/android-28/arch-arm \
 -isystem /private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/sources/cxx-stl/llvm-libc++/include \
 -isystem /private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/sources/cxx-stl/llvm-libc++abi/include \
 -isystem /private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/sources/android/support/include \
 -isystem/private/var/tmp/_bazel_ksmiley/623733d138991bf6c2de613f55fdd0a9/sandbox/darwin-sandbox/179/execroot/rules_foreign_cc_tests/external/androidndk/ndk/sysroot/usr/include  \
 -isysroot /Applications/Xcode-11.0.0b3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk \
 -mmacosx-version-min=10.14   \
 -o CMakeFiles/cmTC_8f5fe.dir/testCCompiler.c.o   \
 -c /var/folders/q5/7_wh20z93c74nj3tllwjl5nr0000gn/T/tmp.UDH7XLoL/CMakeFiles/CMakeTmp/testCCompiler.c

Where both --sysroot and -isysroot are passed, and the -isysroot value is the macOS SDK, which makes me think that CMake definitely doesn't understand that it should be compiling for Android.

It seems like the way rules_foreign_cc sets everything up doesn't handle cross compilation, is that something we want to handle in CMake specific ways?

The text was updated successfully, but these errors were encountered:

github-actions · 2021-04-19T22:53:36Z

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days. Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_foreign_cc!

github-actions · 2021-05-20T22:57:13Z

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"

cpsauer · 2022-07-01T02:08:21Z

@keith, could I ask you how this one ended up? I assume cross compiling for android is still broken like this?
(If so, and we know a collaborator, perhaps we should ask to reopen? Looks like there are a fair number of issues around cross compiling.)

cpsauer · 2022-07-01T03:48:38Z

Thanks for reopening, @UebelAndre! I should take that to mean that this issue is still outstanding, yeah?

UebelAndre · 2022-07-01T04:08:42Z

I currently understand it to be. In general anything closed by the bot just means the issues has become stale and it's closed until someone shows interest in it again.

keith · 2022-07-05T16:44:29Z

I don't recall at this point. We definitely use rules_foreign_cc in envoy targeting android, so it works at least in some cases

cpsauer · 2022-07-06T08:19:38Z

Thanks for your reply, Keith. Hope you had a great 4th weekend!

cpsauer · 2022-07-06T08:29:01Z

Can confirm, unfortunately, that this issue still reproduces, both with ANDROID and isysroot'ing the macOS SDK.

cpsauer · 2022-07-06T08:38:53Z

Though you can get around the former with, for example

cache_entries = select({
        "@platforms//os:android" : {"ANDROID": "1"}
    }),

(If you have platform_mappings set up)

cpsauer · 2022-07-06T08:49:29Z

But sadly, that macOS SDK isysroot still torpedos the build, causing redefinition errors.

cpsauer · 2022-07-09T01:41:54Z

Aha! Okay, so you can get considerably further by setting the following instead.

            "CMAKE_SYSTEM_NAME": "Android",
            "CMAKE_CXX_COMPILER_WORKS": "1",
            "BUILD_SHARED_LIBS": "off",

But there are still NEON and STL issues further down the line in the example I was working with.

Seems like a considerable number of additional settings would have to be propagated from Bazel -> Cmake to make this work well out of the box.

TimoPtr · 2022-08-01T14:56:38Z

I had the same issue in my project. I found out that the foreign rule is not setting the CMAKE_SYSTEM_NAME anywhere so it means that it takes the host value even when you crossbuild.

My build was working on Linux but not on MacOS with the same error. I printed the value of CMAKE_SYSTEM_NAME in Linux and it was set by default to Linux, but on MacOS it was darwin.

Setting the CMAKE_SYSTEM_NAME to Android leads to other issue about the STL like @cpsauer was mentioning that's why CMAKE_CXX_COMPILER_WORKS was needed. But some libraries use this feature to detect flags so it cannot be disable.

Since the foreign rule is properly setting the crosstool toolchain the values and it works on Linux, I did set the CMAKE_SYSTEM_NAME to Linux even on MacOS and it works since the crosstool path are correct.

    "ANDROID": "ON", 
    "CMAKE_SYSTEM_NAME": "Linux",

keith · 2022-09-22T21:52:17Z

Interestingly I have a test over here with the new NDK infra bazelbuild/rules_android_ndk#24 that fails with this same issue, but the same test works fine with the bazel core NDK infra 🤔

keith · 2023-01-05T23:58:12Z

Ok I at least discovered 1 interesting thing about this while debugging my issue above. When cmake checks if the compiler is valid today, it defaults to macOS but that silently succeeds anyways depending on what features are being used. For me this didn't come up in some cases today because -search_paths_first is what causes the failure (at least in some cases), and that flag is silently ignored (or maybe interpreted as something else?!?) by ld.gold, but not ignored by lld, which is why the new NDKs fail immediately.

keith · 2023-01-06T00:02:25Z

I wonder if it would be reasonable to make rules_foreign_cc set the 2 variables mentioned above:

    "ANDROID": "ON", 
    "CMAKE_SYSTEM_NAME": "Linux",

whenever we're building for android? Are there any known downsides to that as a workaround? It's a bit annoying to set as a user if you target more than just android

jsharpe · 2023-01-06T09:41:35Z

I wonder if it would be reasonable to make rules_foreign_cc set the 2 variables mentioned above:
    "ANDROID": "ON", 
    "CMAKE_SYSTEM_NAME": "Linux",
whenever we're building for android? Are there any known downsides to that as a workaround? It's a bit annoying to set as a user if you target more than just android

Seems reasonable but I'm not up to speed enough on the whole platform eco-system code, especially with the special cases of Android, to know how to do detect this within the cmake framework. More generally the CMake rules need some logic for cross compilation support in general; I suspect that to do this generically the user may need to provide platform -> CMAKE_SYSTEM_NAME mappings manually as I don't think there is a way to know a-priori for an arbritary platform what the underlying system actually is if an OS based constraint isn't provided (which AFAIK isn't mandatory?)

keith · 2023-01-06T18:46:25Z

For discussion: #997

keith · 2023-01-15T17:44:42Z

hrm actually with my tests with envoy-mobile https://github.com/envoyproxy/envoy/tree/main/mobile + rules_android_ndk + my patch, my patch isn't enough. maybe there are some other variables we need to set for this?

keith · 2023-01-16T18:16:23Z

hrm i might have just been unlucky with zlib and something about its cmake config, because some others do appear to work fine 🤔

cpsauer · 2023-01-16T23:00:58Z

(Perhaps they condition less on platform or something?)

Keith, if zlib for mobile is the primary issue, might it make sense to just link against the SDK-provided zlibs? That is -lz to linkopts. That's what we do for rules boost and for curl over in https://github.com/hedronvision/bazel-make-cc-https-easy

This was referenced Jul 8, 2019

ci: upgrade to Bazel 0.27.0 envoyproxy/envoy-mobile#116

Closed

document which aspects of the host system leak into the build #290

Closed

github-actions bot added the Can Close? label Apr 19, 2021

github-actions bot closed this as completed May 20, 2021

UebelAndre reopened this Jul 1, 2022

UebelAndre removed the Can Close? label Jul 1, 2022

cpsauer mentioned this issue Aug 5, 2022

Support cross-compiling when using Bazel platforms #869

Open

keith mentioned this issue Sep 22, 2022

Add rules_foreign_cc example bazelbuild/rules_android_ndk#24

Open

keith mentioned this issue Jan 6, 2023

Fix cross compiling for android on macOS #997

Merged

jsharpe closed this as completed in #997 Feb 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cross compiling with CMake for Android #289

Cross compiling with CMake for Android #289

keith commented Jul 8, 2019

github-actions bot commented Apr 19, 2021

github-actions bot commented May 20, 2021

cpsauer commented Jul 1, 2022 •

edited

Loading

cpsauer commented Jul 1, 2022

UebelAndre commented Jul 1, 2022

keith commented Jul 5, 2022

cpsauer commented Jul 6, 2022

cpsauer commented Jul 6, 2022

cpsauer commented Jul 6, 2022

cpsauer commented Jul 6, 2022

cpsauer commented Jul 9, 2022

TimoPtr commented Aug 1, 2022

keith commented Sep 22, 2022

keith commented Jan 5, 2023

keith commented Jan 6, 2023

jsharpe commented Jan 6, 2023

keith commented Jan 6, 2023

keith commented Jan 15, 2023

keith commented Jan 16, 2023

cpsauer commented Jan 16, 2023

Cross compiling with CMake for Android #289

Cross compiling with CMake for Android #289

Comments

keith commented Jul 8, 2019

github-actions bot commented Apr 19, 2021

github-actions bot commented May 20, 2021

cpsauer commented Jul 1, 2022 • edited Loading

cpsauer commented Jul 1, 2022

UebelAndre commented Jul 1, 2022

keith commented Jul 5, 2022

cpsauer commented Jul 6, 2022

cpsauer commented Jul 6, 2022

cpsauer commented Jul 6, 2022

cpsauer commented Jul 6, 2022

cpsauer commented Jul 9, 2022

TimoPtr commented Aug 1, 2022

keith commented Sep 22, 2022

keith commented Jan 5, 2023

keith commented Jan 6, 2023

jsharpe commented Jan 6, 2023

keith commented Jan 6, 2023

keith commented Jan 15, 2023

keith commented Jan 16, 2023

cpsauer commented Jan 16, 2023

cpsauer commented Jul 1, 2022 •

edited

Loading