Auto merge of #132 - malbarbo:android-test, r=malbarbo

Add limited android test support

We could provide android test support by launching an android emulator and running tests in the emulator (like how it is done in libc and rust), but there is licenses issues in redistributing the sdk (see 3.4 of https://developer.android.com/studio/terms.html). Off course, I maybe be wrong about this.

So we download and build bionic libc from source, which I think allows we to distribute the binaries. Having the android libc allows running native binaries using qemu, which can be enough for projects like, nix, ripgrep, etc.

Related #82

Author: homunkulus

.travis.yml

@@ -27,11 +27,11 @@ matrix:
     - env: TARGET=x86_64-unknown-linux-musl                     STD=1 OPENSSL=0.5.5  RUN=1 DEPLOY=1
 
     # Android
-    - env: TARGET=aarch64-linux-android           CPP=1         STD=1 OPENSSL=0.5.5
-    - env: TARGET=arm-linux-androideabi           CPP=1         STD=1 OPENSSL=0.5.5
-    - env: TARGET=armv7-linux-androideabi         CPP=1         STD=1 OPENSSL=0.5.5
-    - env: TARGET=i686-linux-android              CPP=1         STD=1 OPENSSL=0.5.5
-    - env: TARGET=x86_64-linux-android            CPP=1         STD=1 OPENSSL=0.5.5
+    - env: TARGET=aarch64-linux-android           CPP=1         STD=1 OPENSSL=0.5.5  RUN=1
+    - env: TARGET=arm-linux-androideabi           CPP=1         STD=1 OPENSSL=0.5.5  RUN=1
+    - env: TARGET=armv7-linux-androideabi         CPP=1         STD=1 OPENSSL=0.5.5  RUN=1
+    - env: TARGET=i686-linux-android              CPP=1         STD=1 OPENSSL=0.5.5  RUN=1
+    - env: TARGET=x86_64-linux-android            CPP=1         STD=1 OPENSSL=0.5.5  RUN=1
 
     # OSX
     - env: TARGET=i686-apple-darwin                     DYLIB=1 STD=1                RUN=1

README.md

@@ -189,16 +189,16 @@ worst, "hang" (never terminate).
 
 | Target                               |  libc  |   GCC   | OpenSSL | C++ | QEMU  | `test` |
 |--------------------------------------|--------|---------|---------|:---:|-------|:------:|
-| `aarch64-linux-android`              | N/A    | 4.9     | 1.0.2k  | ✓   | N/A   |        |
+| `aarch64-linux-android` [5]          | N/A    | 4.9     | 1.0.2k  | ✓   | N/A   |   ✓    |
 | `aarch64-unknown-linux-gnu`          | 2.19   | 4.8.2   | 1.0.2k  | ✓   | 2.8.0 |   ✓    |
-| `arm-linux-androideabi`              | N/A    | 4.9     | 1.0.2k  | ✓   | N/A   |        |
+| `arm-linux-androideabi` [5]          | N/A    | 4.9     | 1.0.2k  | ✓   | N/A   |   ✓    |
 | `arm-unknown-linux-gnueabi`          | 2.19   | 4.8.2   | 1.0.2k  | ✓   | 2.8.0 |   ✓    |
 | `arm-unknown-linux-musleabi`         | 1.1.15 | 5.3.1   | N/A     |     | 2.8.0 |   ✓    |
-| `armv7-linux-androideabi`            | N/A    | 4.9     | 1.0.2k  | ✓   | N/A   |        |
+| `armv7-linux-androideabi` [5]        | N/A    | 4.9     | 1.0.2k  | ✓   | N/A   |   ✓    |
 | `armv7-unknown-linux-gnueabihf`      | 2.15   | 4.6.2   | 1.0.2k  | ✓   | 2.8.0 |   ✓    |
 | `armv7-unknown-linux-musleabihf`     | 1.1.15 | 5.3.1   | N/A     |     | 2.8.0 |   ✓    |
 | `asmjs-unknown-emscripten` [4]       | 1.1.15 | 1.37.13 | N/A     | ✓   | N/A   |   ✓    |
-| `i686-linux-android`                 | N/A    | 4.9     | 1.0.2k  | ✓   | N/A   |        |
+| `i686-linux-android` [5]             | N/A    | 4.9     | 1.0.2k  | ✓   | N/A   |   ✓    |
 | `i686-pc-windows-gnu`                | N/A    | 6.2.0   | N/A     | ✓   | N/A   |   ✓    |
 | `i686-unknown-freebsd` [1]           | 10.2   | 5.3.0   | 1.0.2k  |     | N/A   |        |
 | `i686-unknown-linux-gnu`             | 2.15   | 4.6.2   | 1.0.2k  | ✓   | N/A   |   ✓    |
@@ -218,7 +218,7 @@ worst, "hang" (never terminate).
 | `thumbv7em-none-eabihf` [3]          | 2.2.0  | 5.3.1   | N/A     |     | N/A   |        |
 | `thumbv7m-none-eabi` [3]             | 2.2.0  | 5.3.1   | N/A     |     | N/A   |        |
 | `wasm32-unknown-emscripten` [4]      | 1.1.15 | 1.37.13 | N/A     | ✓   | N/A   |   ✓    |
-| `x86_64-linux-android`               | N/A    | 4.9     | 1.0.2k  | ✓   | N/A   |        |
+| `x86_64-linux-android` [5]           | N/A    | 4.9     | 1.0.2k  | ✓   | N/A   |   ✓    |
 | `x86_64-pc-windows-gnu`              | N/A    | 6.2.0   | N/A     | ✓   | N/A   |   ✓    |
 | `x86_64-sun-solaris` [1]             | 2.11   | 5.3.0   | 1.0.2k  |     | N/A   |        |
 | `x86_64-unknown-dragonfly` [1] [2]   | 4.6.0  | 5.3.0   | 1.0.2k  |     | N/A   |   ✓    |
@@ -237,6 +237,12 @@ where libc was extracted.
 [4] libc = musl, gcc = emcc; Some projects that use libc may fail due to wrong
     definitions (will be fixed by https://github.com/rust-lang/libc/pull/610)
 
+[5] Only works with native tests, that is, tests that do not depends on the
+    Android Runtime. For i686 some tests may fails with the error `assertion
+    failed: signal(libc::SIGPIPE, libc::SIG_IGN) != libc::SIG_ERR`, see
+    [issue #140](https://github.com/japaric/cross/issues/140) for more
+    information.
+
 ## Debugging
 
 ### QEMU_STRACE (v0.1.9+)

docker/aarch64-linux-android/Dockerfile

@@ -12,16 +12,23 @@ RUN apt-get update && \
 COPY xargo.sh /
 RUN bash /xargo.sh
 
+COPY qemu.sh /
+RUN bash /qemu.sh aarch64 android
+
 COPY android-ndk.sh /
 RUN bash /android-ndk.sh arm64 21
 ENV PATH=$PATH:/android-ndk/bin
 
 COPY openssl.sh /
 RUN bash /openssl.sh linux-generic64 aarch64-linux-android- -mandroid -fomit-frame-pointer
 
+COPY android-system.sh /
+RUN bash /android-system.sh arm64
+
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=aarch64-linux-android-gcc \
+    CARGO_TARGET_AARCH64_LINUX_ANDROID_RUNNER=qemu-aarch64 \
     CC_aarch64_linux_android=aarch64-linux-android-gcc \
     CXX_aarch64_linux_android=aarch64-linux-android-g++ \
     DEP_Z_ROOT=/android-ndk/sysroot/usr/ \
@@ -30,4 +37,7 @@ ENV CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=aarch64-linux-android-gcc \
     OPENSSL_INCLUDE_DIR=/openssl/include \
     OPENSSL_LIB_DIR=/openssl/lib \
     RUST_TEST_THREADS=1 \
-    HOME=/tmp/
+    HOME=/tmp/ \
+    TMPDIR=/tmp/ \
+    ANDROID_ROOT=/system \
+    ANDROID_DATA=/

docker/android-system.sh

@@ -0,0 +1,102 @@
+set -ex
+
+main() {
+    local arch=$1
+    local td=$(mktemp -d)
+    pushd $td
+
+    local dependencies=(
+        ca-certificates
+        curl
+        gcc-multilib
+        git
+        g++-multilib
+        make
+        python
+    )
+
+    # fake java and javac, it is not necessary for what we build, but the build
+    # script ask for it
+    cat << EOF > /usr/bin/java
+#!/bin/bash
+echo "java version \"1.7.0\""
+echo "OpenJDK Runtime Environment (IcedTea 2.6.9)"
+echo "OpenJDK 64-Bit Server VM (build 24.131-b00, mixed mode)"
+EOF
+
+    cat << EOF > /usr/bin/javac
+#!/bin/bash
+echo "javac 1.7.0"
+EOF
+
+    chmod +x /usr/bin/java
+    chmod +x /usr/bin/javac
+
+    # more faking
+    export ANDROID_JAVA_HOME=/tmp
+    mkdir /tmp/lib/
+    touch /tmp/lib/tools.jar
+
+    apt-get update
+    local purge_list=(default-jre)
+    for dep in ${dependencies[@]}; do
+        if ! dpkg -L $dep; then
+            apt-get install --no-install-recommends -y $dep
+            purge_list+=( $dep )
+        fi
+    done
+
+    curl -O https://storage.googleapis.com/git-repo-downloads/repo
+    chmod +x repo
+
+    # this is the minimum set of modules that are need to build bionic
+    # this was created by trial and error
+    ./repo init -u https://android.googlesource.com/platform/manifest -b android-5.0.0_r1
+    ./repo sync bionic
+    ./repo sync build
+    ./repo sync external/compiler-rt
+    ./repo sync external/jemalloc
+    ./repo sync external/libcxx
+    ./repo sync external/libcxxabi
+    ./repo sync external/stlport
+    ./repo sync prebuilts/clang/linux-x86/host/3.5
+    ./repo sync system/core
+    case $arch in
+        arm)
+            ./repo sync prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8
+        ;;
+        arm64)
+            ./repo sync prebuilts/gcc/linux-x86/arm/arm-linux-androideabi-4.8
+            ./repo sync prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9
+        ;;
+        x86)
+            ./repo sync prebuilts/gcc/linux-x86/x86/x86_64-linux-android-4.8
+        ;;
+        x86_64)
+            ./repo sync prebuilts/gcc/linux-x86/x86/x86_64-linux-android-4.8
+        ;;
+    esac
+
+    # avoid build tests
+    rm bionic/linker/tests/Android.mk bionic/tests/Android.mk
+
+    source build/envsetup.sh
+    lunch aosp_$arch-user
+    mmma bionic/
+
+    if [ $arch = "arm" ]; then
+        mv out/target/product/generic/system/ /
+    else
+        mv out/target/product/generic_$arch/system/ /
+    fi
+
+    # clean up
+    apt-get purge --auto-remove -y ${purge_list[@]}
+
+    popd
+
+    rm -rf $td
+    rm $0
+}
+
+main "${@}"

docker/arm-linux-androideabi/Dockerfile

@@ -12,16 +12,23 @@ RUN apt-get update && \
 COPY xargo.sh /
 RUN bash /xargo.sh
 
+COPY qemu.sh /
+RUN bash /qemu.sh arm android
+
 COPY android-ndk.sh /
 RUN bash /android-ndk.sh arm 21
 ENV PATH=$PATH:/android-ndk/bin
 
 COPY openssl.sh /
 RUN bash /openssl.sh android arm-linux-androideabi-
 
+COPY android-system.sh /
+RUN bash /android-system.sh arm
+
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_ARM_LINUX_ANDROIDEABI_LINKER=arm-linux-androideabi-gcc \
+    CARGO_TARGET_ARM_LINUX_ANDROIDEABI_RUNNER=qemu-arm \
     CC_arm_linux_androideabi=arm-linux-androideabi-gcc \
     CXX_arm_linux_androideabi=arm-linux-androideabi-g++ \
     DEP_Z_ROOT=/android-ndk/sysroot/usr/ \
@@ -30,4 +37,7 @@ ENV CARGO_TARGET_ARM_LINUX_ANDROIDEABI_LINKER=arm-linux-androideabi-gcc \
     OPENSSL_INCLUDE_DIR=/openssl/include \
     OPENSSL_LIB_DIR=/openssl/lib \
     RUST_TEST_THREADS=1 \
-    HOME=/tmp/
+    HOME=/tmp/ \
+    TMPDIR=/tmp/ \
+    ANDROID_ROOT=/system \
+    ANDROID_DATA=/

docker/armv7-linux-androideabi/Dockerfile

@@ -12,16 +12,23 @@ RUN apt-get update && \
 COPY xargo.sh /
 RUN bash /xargo.sh
 
+COPY qemu.sh /
+RUN bash /qemu.sh arm android
+
 COPY android-ndk.sh /
 RUN bash /android-ndk.sh arm 21
 ENV PATH=$PATH:/android-ndk/bin
 
 COPY openssl.sh /
 RUN bash /openssl.sh android-armv7 arm-linux-androideabi-
 
+COPY android-system.sh /
+RUN bash /android-system.sh arm
+
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_LINKER=arm-linux-androideabi-gcc \
+    CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_RUNNER=qemu-arm \
     CC_armv7_linux_androideabi=arm-linux-androideabi-gcc \
     CXX_armv7_linux_androideabi=arm-linux-androideabi-g++ \
     DEP_Z_ROOT=/android-ndk/sysroot/usr/ \
@@ -30,4 +37,7 @@ ENV CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_LINKER=arm-linux-androideabi-gcc \
     OPENSSL_INCLUDE_DIR=/openssl/include \
     OPENSSL_LIB_DIR=/openssl/lib \
     RUST_TEST_THREADS=1 \
-    HOME=/tmp/
+    HOME=/tmp/ \
+    TMPDIR=/tmp/ \
+    ANDROID_ROOT=/system \
+    ANDROID_DATA=/

docker/i686-linux-android/Dockerfile

@@ -12,16 +12,31 @@ RUN apt-get update && \
 COPY xargo.sh /
 RUN bash /xargo.sh
 
+# We could supposedly directly run i686 binaries like we do for x86_64, but
+# doing so generates an assertion failure:
+#   ... assertion failed: signal(libc::SIGPIPE, libc::SIG_IGN) != libc::SIG_ERR
+#   ... src/libstd/sys/unix/mod.rs
+#   fatal runtime error: failed to initiate panic, error 5
+#
+# Running with qemu works as expected
+COPY qemu.sh /
+RUN bash /qemu.sh i386 android
+
 COPY android-ndk.sh /
 RUN bash /android-ndk.sh x86 21
 ENV PATH=$PATH:/android-ndk/bin
 
+# Build with no-asm to make openssl linked binaries position-independent (PIE)
 COPY openssl.sh /
-RUN bash /openssl.sh android-x86 i686-linux-android-
+RUN bash /openssl.sh android-x86 i686-linux-android- no-asm
+
+COPY android-system.sh /
+RUN bash /android-system.sh x86
 
 # Libz is distributed in the android ndk, but for some unknown reason it is not
 # found in the build process of some crates, so we explicit set the DEP_Z_ROOT
 ENV CARGO_TARGET_I686_LINUX_ANDROID_LINKER=i686-linux-android-gcc \
+    CARGO_TARGET_I686_LINUX_ANDROID_RUNNER=qemu-i386 \
     CC_i686_linux_android=i686-linux-android-gcc \
     CXX_i686_linux_android=i686-linux-android-g++ \
     DEP_Z_ROOT=/android-ndk/sysroot/usr/ \
@@ -30,4 +45,7 @@ ENV CARGO_TARGET_I686_LINUX_ANDROID_LINKER=i686-linux-android-gcc \
     OPENSSL_INCLUDE_DIR=/openssl/include \
     OPENSSL_LIB_DIR=/openssl/lib \
     RUST_TEST_THREADS=1 \
-    HOME=/tmp/
+    HOME=/tmp/ \
+    TMPDIR=/tmp/ \
+    ANDROID_ROOT=/system \
+    ANDROID_DATA=/