Skip to content

Commit

Permalink
Add support for different Android versions.
Browse files Browse the repository at this point in the history
We provide the following arguments to configure the Android version:
- `ANDROID_NDK`
- `ANDROID_SDK` (note that this is dependent on `ANDROID_VERSION)
- `ANDROID_VERSION`
- `ANDROID_SYSTEM_COMPLETE`: do a complete Android build
- `ANDROID_SYSTEM_NONE`: do not build the Android system, disables runner support

We now support NDK versions r10e-r25b, SDK versions 21-33, and Android versions 5.0, 5.1, 6.0, 7.0, 8.0, 8.1, 9.0, and 10.0. We also validate that the NDK, SDK, and Android versions are compatible.

Next, we've improved the removal of unittests during the build process, to ensure fast builds while maintaining compatibility with various newer Android versions. To do this, we've implemented a Python library and a script. The Python library contains a complete parser (correctly parses all valid input) for Soong blueprint files, using an LALR grammar, and a rudimentary parser for Makefiles.

The Soong parser removes any `gtest` dependencies, as well as any subdirectories or scope names containing `test`. For example:

```
cc_library {
    name: "lib",
    srcs: ["lib.cc",],
}
cc_test {
    name: "test",
    defaults: ["target"],
    srcs: ["test.cc"],
}
```

Will become:

```
cc_library {
    name: "lib",
    srcs: ["lib.cc",],
}
```

The Makefile parser first splits the file based on conditional directives (`ifeq`, `endif`, etc.) to ensure any subsequent processing doesn't lead to unbalanced directives. Next, we split the text within each directive based on comment sections used in the Android source tree. For example:

```
test_tags := tests

include $(call subdir,$(LOCAL_PATH))

c_flags := \
  -g \
  -Wall
```

We can therefore remove the `Benchmarks` and `Unit tests` sections without removing the `Other section`.

The Python library is reasonably performant (it adds no noticeable overhead to the build process) and is compact (in total, < 60KB). Also, it is much more resilient than a series of `sed` scripts.

Finally, extensive unittests have been added for the Python library, for both code linting (`flake8`) and unittests (via `tox`). Since we cannot assume the presence of Python on the host machine, the tests can be optionally enabled via the `--python` flag (or `PYTHON` environment variable, to hook into the git hooks), and custom overrides for the `flake8` and `tox` commands are provided (since the user may wish to specify a specific Python version, such as `python3.7 -m flake8`).
  • Loading branch information
Alexhuszagh committed Oct 25, 2022
1 parent d5c08da commit 79fb560
Show file tree
Hide file tree
Showing 39 changed files with 3,618 additions and 163 deletions.
4 changes: 4 additions & 0 deletions .changes/1023.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"description": "support different Android NDK, API, and Android versions using Docker build args.",
"type": "added"
}
9 changes: 9 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,12 @@
**/*.log
/cargo-timing*.html
CHANGELOG.md.draft

# python stuff
__pycache__/
.pytest_cache/
*.py[cod]
*$py.class
*.egg-info/
*.egg
.tox
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions docker/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# don't copy any of the python artifacts to the docker context
__pycache__/
.pytest_cache/
*.py[cod]
*$py.class
**/*.egg-info/
*.egg
.tox

# also skip our test suite
android/tests/
34 changes: 29 additions & 5 deletions docker/Dockerfile.aarch64-linux-android
Original file line number Diff line number Diff line change
Expand Up @@ -10,27 +10,51 @@ RUN /cmake.sh
COPY xargo.sh /
RUN /xargo.sh

COPY qemu.sh /
RUN /qemu.sh aarch64

ARG ANDROID_NDK=r21d
ARG ANDROID_SDK=28
ARG ANDROID_VERSION=9.0.0_r1
ARG ANDROID_SYSTEM_NONE=0
ARG ANDROID_SYSTEM_COMPLETE=0
ARG PYTHON_TMPDIR=/tmp/android

COPY validate-android-args.sh /
RUN /validate-android-args.sh arm64

COPY android-ndk.sh /
RUN /android-ndk.sh arm64 28
RUN /android-ndk.sh arm64
ENV PATH=$PATH:/android-ndk/bin

COPY android-system.sh /
RUN mkdir -p $PYTHON_TMPDIR
COPY android $PYTHON_TMPDIR
RUN /android-system.sh arm64

COPY qemu.sh /
RUN /qemu.sh aarch64

ENV CROSS_SYSROOT=/android-ndk/sysroot
RUN cp $CROSS_SYSROOT/usr/lib/aarch64-linux-android/28/libz.so /system/lib/
COPY android-symlink.sh /
RUN /android-symlink.sh aarch64 aarch64-linux-android

COPY android-runner /

# Libz is distributed in the android ndk, but for some unknown reason it is not
# found in the build process of some crates, so we explicit set the DEP_Z_ROOT
ENV CARGO_TARGET_AARCH64_LINUX_ANDROID_LINKER=aarch64-linux-android-gcc \
CARGO_TARGET_AARCH64_LINUX_ANDROID_RUNNER="/android-runner aarch64" \
AR_aarch64_linux_android=aarch64-linux-android-ar \
AS_aarch64_linux_android=aarch64-linux-android-as \
CC_aarch64_linux_android=aarch64-linux-android-gcc \
CXX_aarch64_linux_android=aarch64-linux-android-g++ \
LD_aarch64_linux_android=aarch64-linux-android-ld \
NM_aarch64_linux_android=aarch64-linux-android-nm \
OBJCOPY_aarch64_linux_android=aarch64-linux-android-objcopy \
OBJDUMP_aarch64_linux_android=aarch64-linux-android-objdump \
RANLIB_aarch64_linux_android=aarch64-linux-android-ranlib \
READELF_aarch64_linux_android=aarch64-linux-android-readelf \
SIZE_aarch64_linux_android=aarch64-linux-android-size \
STRINGS_aarch64_linux_android=aarch64-linux-android-strings \
STRIP_aarch64_linux_android=aarch64-linux-android-strip \
BINDGEN_EXTRA_CLANG_ARGS_aarch64_linux_android="--sysroot=$CROSS_SYSROOT" \
DEP_Z_INCLUDE="$CROSS_SYSROOT/usr/include"/ \
RUST_TEST_THREADS=1 \
Expand Down
34 changes: 28 additions & 6 deletions docker/Dockerfile.arm-linux-androideabi
Original file line number Diff line number Diff line change
Expand Up @@ -10,27 +10,49 @@ RUN /cmake.sh
COPY xargo.sh /
RUN /xargo.sh

COPY qemu.sh /
RUN /qemu.sh arm

ARG ANDROID_NDK=r21d
ARG ANDROID_SDK=28
ARG ANDROID_VERSION=9.0.0_r1
ARG ANDROID_SYSTEM_NONE=0
ARG ANDROID_SYSTEM_COMPLETE=0
ARG PYTHON_TMPDIR=/tmp/android

COPY validate-android-args.sh /
RUN /validate-android-args.sh arm

COPY android-ndk.sh /
RUN /android-ndk.sh arm 28
RUN /android-ndk.sh arm
ENV PATH=$PATH:/android-ndk/bin

COPY android-system.sh /
COPY android-system.sh remove_android_tests.py /
RUN /android-system.sh arm

COPY qemu.sh /
RUN /qemu.sh arm

ENV CROSS_SYSROOT=/android-ndk/sysroot
RUN cp $CROSS_SYSROOT/usr/lib/arm-linux-androideabi/28/libz.so /system/lib/
COPY android-symlink.sh /
RUN /android-symlink.sh arm arm-linux-androideabi

COPY android-runner /

# Libz is distributed in the android ndk, but for some unknown reason it is not
# found in the build process of some crates, so we explicit set the DEP_Z_ROOT
ENV CARGO_TARGET_ARM_LINUX_ANDROIDEABI_LINKER=arm-linux-androideabi-gcc \
CARGO_TARGET_ARM_LINUX_ANDROIDEABI_RUNNER="/android-runner arm" \
AR_arm_linux_androideabi=arm-linux-androideabi-ar \
AS_arm_linux_androideabi=arm-linux-androideabi-as \
CC_arm_linux_androideabi=arm-linux-androideabi-gcc \
CXX_arm_linux_androideabi=arm-linux-androideabi-g++ \
LD_arm_linux_androideabi=arm-linux-androideabi-ld \
NM_arm_linux_androideabi=arm-linux-androideabi-nm \
OBJCOPY_arm_linux_androideabi=arm-linux-androideabi-objcopy \
OBJDUMP_arm_linux_androideabi=arm-linux-androideabi-objdump \
RANLIB_arm_linux_androideabi=arm-linux-androideabi-ranlib \
READELF_arm_linux_androideabi=arm-linux-androideabi-readelf \
SIZE_arm_linux_androideabi=arm-linux-androideabi-size \
STRINGS_arm_linux_androideabi=arm-linux-androideabi-strings \
STRIP_arm_linux_androideabi=arm-linux-androideabi-strip \
BINDGEN_EXTRA_CLANG_ARGS_arm_linux_androideabi="--sysroot=$CROSS_SYSROOT" \
DEP_Z_INCLUDE="$CROSS_SYSROOT/usr/include/" \
RUST_TEST_THREADS=1 \
Expand Down
34 changes: 28 additions & 6 deletions docker/Dockerfile.armv7-linux-androideabi
Original file line number Diff line number Diff line change
Expand Up @@ -10,27 +10,49 @@ RUN /cmake.sh
COPY xargo.sh /
RUN /xargo.sh

COPY qemu.sh /
RUN /qemu.sh arm

ARG ANDROID_NDK=r21d
ARG ANDROID_SDK=28
ARG ANDROID_VERSION=9.0.0_r1
ARG ANDROID_SYSTEM_NONE=0
ARG ANDROID_SYSTEM_COMPLETE=0
ARG PYTHON_TMPDIR=/tmp/android

COPY validate-android-args.sh /
RUN /validate-android-args.sh arm

COPY android-ndk.sh /
RUN /android-ndk.sh arm 28
RUN /android-ndk.sh arm
ENV PATH=$PATH:/android-ndk/bin

COPY android-system.sh /
COPY android-system.sh remove_android_tests.py /
RUN /android-system.sh arm

COPY qemu.sh /
RUN /qemu.sh arm

ENV CROSS_SYSROOT=/android-ndk/sysroot
RUN cp $CROSS_SYSROOT/usr/lib/arm-linux-androideabi/28/libz.so /system/lib/
COPY android-symlink.sh /
RUN /android-symlink.sh arm arm-linux-androideabi

COPY android-runner /

# Libz is distributed in the android ndk, but for some unknown reason it is not
# found in the build process of some crates, so we explicit set the DEP_Z_ROOT
ENV CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_LINKER=arm-linux-androideabi-gcc \
CARGO_TARGET_ARMV7_LINUX_ANDROIDEABI_RUNNER="/android-runner arm" \
AR_armv7_linux_androideabi=arm-linux-androideabi-ar \
AS_armv7_linux_androideabi=arm-linux-androideabi-as \
CC_armv7_linux_androideabi=arm-linux-androideabi-gcc \
CXX_armv7_linux_androideabi=arm-linux-androideabi-g++ \
LD_armv7_linux_androideabi=arm-linux-androideabi-ld \
NM_armv7_linux_androideabi=arm-linux-androideabi-nm \
OBJCOPY_armv7_linux_androideabi=arm-linux-androideabi-objcopy \
OBJDUMP_armv7_linux_androideabi=arm-linux-androideabi-objdump \
RANLIB_armv7_linux_androideabi=arm-linux-androideabi-ranlib \
READELF_armv7_linux_androideabi=arm-linux-androideabi-readelf \
SIZE_armv7_linux_androideabi=arm-linux-androideabi-size \
STRINGS_armv7_linux_androideabi=arm-linux-androideabi-strings \
STRIP_armv7_linux_androideabi=arm-linux-androideabi-strip \
BINDGEN_EXTRA_CLANG_ARGS_armv7_linux_androideabi="--sysroot=$CROSS_SYSROOT" \
DEP_Z_INCLUDE="$CROSS_SYSROOT/usr/include/" \
RUST_TEST_THREADS=1 \
Expand Down
41 changes: 32 additions & 9 deletions docker/Dockerfile.i686-linux-android
Original file line number Diff line number Diff line change
Expand Up @@ -10,34 +10,57 @@ RUN /cmake.sh
COPY xargo.sh /
RUN /xargo.sh

COPY android-ndk.sh /
RUN /android-ndk.sh x86 28
ENV PATH=$PATH:/android-ndk/bin

COPY android-system.sh /
RUN /android-system.sh x86

# We could supposedly directly run i686 binaries like we do for x86_64, but
# doing so generates an assertion failure:
# ... assertion failed: signal(libc::SIGPIPE, libc::SIG_IGN) != libc::SIG_ERR
# ... src/libstd/sys/unix/mod.rs
# fatal runtime error: failed to initiate panic, error 5
#
# Running with qemu works as expected
# Running with qemu works as expected. it also ensures that're we're
# running on a CPU common 32-bit x86 systems.
COPY qemu.sh /
RUN /qemu.sh i386

ARG ANDROID_NDK=r21d
ARG ANDROID_SDK=28
ARG ANDROID_VERSION=9.0.0_r1
ARG ANDROID_SYSTEM_NONE=0
ARG ANDROID_SYSTEM_COMPLETE=0
ARG PYTHON_TMPDIR=/tmp/android

COPY validate-android-args.sh /
RUN /validate-android-args.sh x86

COPY android-ndk.sh /
RUN /android-ndk.sh x86
ENV PATH=$PATH:/android-ndk/bin

COPY android-system.sh remove_android_tests.py /
RUN /android-system.sh x86

ENV CROSS_SYSROOT=/android-ndk/sysroot
RUN cp $CROSS_SYSROOT/usr/lib/i686-linux-android/28/libz.so /system/lib/
COPY android-symlink.sh /
RUN /android-symlink.sh i386 i686-linux-android

COPY android-runner /

# Libz is distributed in the android ndk, but for some unknown reason it is not
# found in the build process of some crates, so we explicit set the DEP_Z_ROOT
ENV CARGO_TARGET_I686_LINUX_ANDROID_LINKER=i686-linux-android-gcc \
CARGO_TARGET_I686_LINUX_ANDROID_RUNNER="/android-runner i686" \
AR_i686_linux_android=i686-linux-android-ar \
AS_i686_linux_android=i686-linux-android-as \
CC_i686_linux_android=i686-linux-android-gcc \
CXX_i686_linux_android=i686-linux-android-g++ \
LD_i686_linux_android=i686-linux-android-ld \
NM_i686_linux_android=i686-linux-android-nm \
OBJCOPY_i686_linux_android=i686-linux-android-objcopy \
OBJDUMP_i686_linux_android=i686-linux-android-objdump \
RANLIB_i686_linux_android=i686-linux-android-ranlib \
READELF_i686_linux_android=i686-linux-android-readelf \
SIZE_i686_linux_android=i686-linux-android-size \
STRINGS_i686_linux_android=i686-linux-android-strings \
STRIP_i686_linux_android=i686-linux-android-strip \
BINDGEN_EXTRA_CLANG_ARGS_i686_linux_android="--sysroot=$CROSS_SYSROOT" \
DEP_Z_INCLUDE="$CROSS_SYSROOT/usr/include/" \
LIBZ_SYS_STATIC=1 \
Expand Down
23 changes: 17 additions & 6 deletions docker/Dockerfile.thumbv7neon-linux-androideabi
Original file line number Diff line number Diff line change
Expand Up @@ -10,18 +10,29 @@ RUN /cmake.sh
COPY xargo.sh /
RUN /xargo.sh

COPY qemu.sh /
RUN /qemu.sh arm

ARG ANDROID_NDK=r21d
ARG ANDROID_SDK=28
ARG ANDROID_VERSION=9.0.0_r1
ARG ANDROID_SYSTEM_NONE=0
ARG ANDROID_SYSTEM_COMPLETE=0
ARG PYTHON_TMPDIR=/tmp/android

COPY validate-android-args.sh /
RUN /validate-android-args.sh arm

COPY android-ndk.sh /
RUN /android-ndk.sh arm 28
RUN /android-ndk.sh arm
ENV PATH=$PATH:/android-ndk/bin

COPY android-system.sh /
COPY android-system.sh remove_android_tests.py /
RUN /android-system.sh arm

COPY qemu.sh /
RUN /qemu.sh arm

ENV CROSS_SYSROOT=/android-ndk/sysroot
RUN cp $CROSS_SYSROOT/usr/lib/arm-linux-androideabi/28/libz.so /system/lib/
COPY android-symlink.sh /
RUN /android-symlink.sh arm arm-linux-androideabi

COPY android-runner /

Expand Down
36 changes: 29 additions & 7 deletions docker/Dockerfile.x86_64-linux-android
Original file line number Diff line number Diff line change
Expand Up @@ -10,28 +10,50 @@ RUN /cmake.sh
COPY xargo.sh /
RUN /xargo.sh

# Using qemu allows older host cpus (without sse4) to execute the target binaries
COPY qemu.sh /
RUN /qemu.sh x86_64

ARG ANDROID_NDK=r21d
ARG ANDROID_SDK=28
ARG ANDROID_VERSION=9.0.0_r1
ARG ANDROID_SYSTEM_NONE=0
ARG ANDROID_SYSTEM_COMPLETE=0
ARG PYTHON_TMPDIR=/tmp/android

COPY validate-android-args.sh /
RUN /validate-android-args.sh x86_64

COPY android-ndk.sh /
RUN /android-ndk.sh x86_64 28
RUN /android-ndk.sh x86_64
ENV PATH=$PATH:/android-ndk/bin

COPY android-system.sh /
COPY android-system.sh remove_android_tests.py /
RUN /android-system.sh x86_64

# Using qemu allows older host cpus (without sse4) to execute the target binaries
COPY qemu.sh /
RUN /qemu.sh x86_64

ENV CROSS_SYSROOT=/android-ndk/sysroot
RUN cp $CROSS_SYSROOT/usr/lib/x86_64-linux-android/28/libz.so /system/lib/
COPY android-symlink.sh /
RUN /android-symlink.sh x86_64 x86_64-linux-android

COPY android-runner /

# Libz is distributed in the android ndk, but for some unknown reason it is not
# found in the build process of some crates, so we explicit set the DEP_Z_ROOT
ENV CARGO_TARGET_X86_64_LINUX_ANDROID_LINKER=x86_64-linux-android-gcc \
CARGO_TARGET_X86_64_LINUX_ANDROID_RUNNER="/android-runner x86_64" \
AR_x86_64_linux_android=x86_64-linux-android-ar \
AS_x86_64_linux_android=x86_64-linux-android-as \
CC_x86_64_linux_android=x86_64-linux-android-gcc \
CXX_x86_64_linux_android=x86_64-linux-android-g++ \
LD_x86_64_linux_android=x86_64-linux-android-ld \
NM_x86_64_linux_android=x86_64-linux-android-nm \
OBJCOPY_x86_64_linux_android=x86_64-linux-android-objcopy \
OBJDUMP_x86_64_linux_android=x86_64-linux-android-objdump \
RANLIB_x86_64_linux_android=x86_64-linux-android-ranlib \
READELF_x86_64_linux_android=x86_64-linux-android-readelf \
SIZE_x86_64_linux_android=x86_64-linux-android-size \
STRINGS_x86_64_linux_android=x86_64-linux-android-strings \
STRIP_x86_64_linux_android=x86_64-linux-android-strip \
BINDGEN_EXTRA_CLANG_ARGS_x86_64_linux_android="--sysroot=$CROSS_SYSROOT" \
DEP_Z_INCLUDE="$CROSS_SYSROOT/usr/include/" \
RUST_TEST_THREADS=1 \
Expand Down
Loading

0 comments on commit 79fb560

Please sign in to comment.