From 0b76790c1c4daaf0a8031ef7bff04b39abe1efbb Mon Sep 17 00:00:00 2001 From: Dan Albert Date: Wed, 20 Apr 2016 11:21:25 -0700 Subject: [PATCH] Fix #84. Add supporting docs. --- CHANGELOG.md | 81 +++++++++++++++++++ OWNERS | 10 +++ README.md | 137 +++++++++++++++++++++++++++++++ docs/GeneratingSysroots.md | 87 ++++++++++++++++++++ docs/Testing.md | 161 +++++++++++++++++++++++++++++++++++++ docs/Toolchains.md | 130 ++++++++++++++++++++++++++++++ 6 files changed, 606 insertions(+) create mode 100644 CHANGELOG.md create mode 100644 OWNERS create mode 100644 README.md create mode 100644 docs/GeneratingSysroots.md create mode 100644 docs/Testing.md create mode 100644 docs/Toolchains.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 000000000..b481146c9 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,81 @@ +Changelog +========= + +Report issues to [GitHub]. + +[GitHub]: https://github.com/android-ndk/ndk/issues + +NDK +--- + * Removed all sysroots for pre-GB platform levels. We dropped support for them + in r11, but neglected to actually remove them. + * Exception handling when using `c++_shared` on ARM32 now mostly works (see + [Known Issues](#known-issues)). The unwinder will now be linked into each + linked object rather than into libc++ itself. + * Default compiler flags have been pruned: + https://github.com/android-ndk/ndk/issues/27. + * Full changes here: https://android-review.googlesource.com/#/c/207721/5. + * `-fno-limit-debug-info` has been enabled by default for Clang debug builds. + This should improve debugability with LLDB. + * `--build-id` is now enabled by default. + * This will be shown in native crash reports so you can easily identify + which version of your code was running. + * `NDK_USE_CYGPATH` should no longer cause problems with libgcc: + http://b.android.com/195486. + * `-Wl,--warn-shared-textrel` and`-Wl,--fatal-warnings` are now enabled by + default. If you have shared text relocations, your app will not load on + Marshmallow or later (and have never been allowed for 64-bit apps). + * Precompiled headers should work better: + https://github.com/android-ndk/ndk/issues/14 and + https://github.com/android-ndk/ndk/issues/16. + +Clang +----- + + * Clang has been updated to 3.8svn (r256229, build 2690385). + * Note that Clang packaged in the Windows 64 NDK is actually 32-bit. + * `__thread` should work for real this time. + +GCC +--- + + * Synchronized with the ChromeOS GCC @ `google/gcc-4_9` r227810. + * Backported coverage sanitizer patch from ToT (r231296). + * Fixed libatomic to not use ifuncs: + https://github.com/android-ndk/ndk/issues/31. + +Binutils +-------- + + * "Erratum 843419 found and fixed" info messages are silenced. + * Introduced option '--long-plt' to fix internal linker error when linking huge + arm32 binaries. + * Fixed wrong run time stubs for AArch64. This was causing jump addresses to be + calculated incorrectly for very large DSOs. + * Introduced default option '--no-apply-dynamic' to work around a dynamic + linker bug for earlier Android releases. + * NDK r11 KI for `dynamic_cast` not working with Clang, x86, `stlport_static` + and optimization has been fixed. + +GDB +--- + + * Updated to GDB 7.11: https://www.gnu.org/software/gdb/news/. + +Known Issues +------------ + + * This is not intended to be a comprehensive list of all outstanding bugs. + * x86 ASAN still does work. See discussion on + https://android-review.googlesource.com/#/c/186276/ + * Exception unwinding with `c++_shared` still does not work for ARM on + Gingerbread or Ice Cream Sandwich. + * Bionic headers and libraries for Marshmallow and N are not yet exposed + despite the presence of android-24. Those platforms are still the Lollipop + headers and libraries (not a regression from r11). + * RenderScript tools are not present (not a regression from r11): + https://github.com/android-ndk/ndk/issues/7. + * r12 will most likely not work with Android Studio/Gradle right away. The + Gradle plugin will need to be updated to match the build changes we made in + `ndk-build`. + * We've regressed on http://b.android.com/41770. diff --git a/OWNERS b/OWNERS new file mode 100644 index 000000000..6013a550f --- /dev/null +++ b/OWNERS @@ -0,0 +1,10 @@ +danalbert +enh +jmgao + +# For questions rather than code review, send email to: +# +# android-ndk@google.com (Googlers only) +# android-ndk@googlegroups.com (public) +# +# Ask on Stack Overflow: http://stackoverflow.com/questions/tagged/android-ndk. diff --git a/README.md b/README.md new file mode 100644 index 000000000..a8c168446 --- /dev/null +++ b/README.md @@ -0,0 +1,137 @@ +Android Native Development Kit (NDK) +==================================== + +The latest version of this document is available at +https://android.googlesource.com/platform/ndk/+/master/README.md. + +**Note:** This document is for developers _of_ the NDK, not developers +that use the NDK. + +The NDK allows Android application developers to include +native code in their Android application packages, compiled as JNI shared +libraries. + +Other Resources +--------------- + +This doc gives a high level overview of the NDK's build, packaging, and test +process. For other use cases, or more in depth documentation, refer to the +following sources: + + * User documentation is available on the [Android Developer website]. + * Adding a new NDK API or platform version? Check [Generating Sysroots]. + * Working on Clang or GCC? See [Toolchains.md]. + * Discussions related to the Android NDK happen on the [android-ndk Google + Group]. + * File bugs against the NDK at https://github.com/android-ndk/ndk/issues. + +[Android Developer website]: https://developer.android.com/ndk/index.html +[android-ndk Google Group]: http://groups.google.com/group/android-ndk +[Generating Sysroots]: docs/GeneratingSysroots.md +[Toolchains.md]: docs/Toolchains.md + +Building the NDK +================ + +Both Linux and Windows host binaries are built on Linux machines. Windows host +binaries are built via MinGW cross compiler. Systems without a working MinGW +compiler can use `build/tools/build-mingw64-toolchain.sh` to generate their own +and be added to the `PATH` for build scripts to discover. + +Building binaries for Mac OS X requires at least 10.8. + +Target headers and binaries are built on Linux. + +Components +---------- + +The NDK consists of three parts: host binaries, target prebuilts, and others +(build system, docs, samples, tests). + +### Host Binaries + +* `toolchains/` contains GCC and Clang toolchains. + * `$TOOLCHAIN/config.mk` contains ARCH and ABIS this toolchain can handle. + * `$TOOLCHAIN/setup.mk` contains toolchain-specific default CFLAGS/LDFLAGS + when this toolchain is used. +* `prebuilt/$HOST_TAG` contains build dependencies and additional tools. + * make, awk, python, yasm, and for Windows: cmp.exe and echo.exe + * `ndk-depends`, `ndk-stack` and `ndk-gdb` can also be found here. + +### Target Headers and Binaries + +* `platforms/android-$VERSION/arch-$ARCH_NAME/` contains headers and libraries + for each API level. + * The build system sets `--sysroot` to one of these directories based on + user-specified `APP_ABI` and `APP_PLATFORM`. +* `sources/cxx-stl/$STL` contains the headers and libraries for the various C++ + STLs. +* `prebuilt/android-$ARCH/gdbserver` contains gdbserver. + +### Others + +* `build/` contains the ndk-build system and scripts to rebuild NDK. +* `sources/android` and `sources/third_party` contain modules that can be used + in apps (cpufeatures, native\_app\_glue, etc) via `$(call import-module, + $MODULE)` +* `tests/` + +Prerequisites +------------- + +* [AOSP NDK Repository](http://source.android.com/source/downloading.html) + * Check out the branch `master-ndk` + + ```bash + repo init -u https://android.googlesource.com/platform/manifest \ + -b master-ndk + + # Googlers, use + repo init -u \ + persistent-https://android.git.corp.google.com/platform/manifest \ + -b master-ndk + ``` + +* Additional Linux Dependencies (available from apt): + * bison + * flex + * libtool + * mingw-w64 + * pbzip2 (optional, improves packaging times) + * texinfo + * python3 (used for Vulkan validation layer generation) + * python-lxml (used for Vulkan validation layer generation) +* Mac OS X also requires Xcode. + +Host/Target prebuilts +--------------------- + +### For Linux or Darwin: + +```bash +$ python checkbuild.py +``` + +### For Windows, from Linux: + +```bash +$ python checkbuild.py --system windows # Or windows64. +``` + +`checkbuild.py` also accepts a variety of other options to speed up local +builds, namely `--arch` and `--module`. + +Packaging +--------- + +By default, `checkbuild.py` will also package the NDK and run basic tests. To +skip the packaging step, use the `--no-package` flag. Note that running the +tests does require the packaging step. + +If you need to re-run just the packaging step without going through a build, +packaging is handled by `build/tools/package.py`. + +Testing +------- + +Testing is discussed in [Testing.md](docs/Testing.md). diff --git a/docs/GeneratingSysroots.md b/docs/GeneratingSysroots.md new file mode 100644 index 000000000..2bb72246c --- /dev/null +++ b/docs/GeneratingSysroots.md @@ -0,0 +1,87 @@ +Generating NDK Sysroots +======================= + +The latest version of this document is available at +https://android.googlesource.com/platform/ndk/+/master/docs/GeneratingSysroots.md. + +The `platforms/` subtree of an NDK package contains sysroots for each +architecture and API level. The contents of these sysroots are generated as part +of the NDK build. + +The Sysroot Defintions +---------------------- + +The contents of the sysroots are defined by `development/ndk/platforms`. +Note that this is from the top level of the Android tree, not within the `ndk/` +subdirectory. + +The layout of this tree is as follows: + + * `android-$VERSION` + * `include` + * `arch-$ARCH` + * `include` + * Architecture specific headers. Typically the `asm` and `machine` + directories. + * `lib` + * Prebuilt static libraries. + * Note that for the time being these are only libc, libm, libz, and + libstdc++, and that they are rarely updated (we have versions for + android-3, android-9, and android-21, but not any other + releases). + * `src` + * Source for the CRT objects (crtbegin, crtend, etc.). + * `symbols` + * Contains interface definitions for the shared platform libraries. + * Discussed in detail in [Shared Library Definitions]. + +These trees are a very reduced form of a sysroot. Not every header or library +will be in every platform version. Instead, [gen-platforms.sh] will build the +full sysroot from what is defined here. This is done to ease maintenance of the +headers and libraries. If, for example, `string.h` remained the same between +Gingerbread and KitKat, but was updated in Lollipop, there need only be a copy +of `string.h` in android-9 and android-21, not in every intermediate release. + +Shared Library Defintions +------------------------- + +When an NDK binary links against a platform library, rather than shipping a copy +of the library with the binary, it loads the library from the system itself (in +the case of an app, many of the libraries will already be loaded into the +address space by the zygote). As such, the shared libraries in the NDK do not +need to be valid aside from the symbols they expose. This simplifies things, +because all we need to generate the shared libraries for a given architecture +and platform version is a list of functions and variables. + +The `android-$VERSION/arch-$ARCH/symbols/*.txt` files contain these library +definitions. Each file is named either `$LIBNAME.functions.txt` or +`$LIBNAME.variables.txt`. For each such file, [gen-platforms.sh] will create a +stub shared library for that library with the functions and variables defined in +those lists. + +For some platform versions (currently only android-23), there is also a +`$LIBNAME.versions.txt`. This is a linker version script that will be used to +apply symbol versions when generating the stub shared library. + +Examples of each of these three file types can be found in +[development/ndk/platforms/android-23/arch-arm64/symbols]. + +Adding a New Platform API +------------------------- + +These instructions are valid both for adding new native frameworks APIs and +updating bionic. + + 1. If the platform version you are adding your API to does not yet exist, there + are no special steps; just make the new directory. + 2. Add your (possibly new, possibly updated) headers to the include directory. + 3. Add [Shared Library Definitions] files for each architecture to the + `arch-$ARCH/symbols` directory. + 4. Add tests to [tests/device](../tests/device). + 5. Build and test an NDK (see [README.md](../README.md)). + 6. Upload your change(s) and add the current [NDK owners](../OWNERS) as + reviewers. + +[Shared Library Definitions]: #shared-library-defintions +[gen-platforms.sh]: ../build/tools/gen-platforms.sh +[development/ndk/platforms/android-23/arch-arm64/symbols]: https://android.googlesource.com/platform/development/+/master/ndk/platforms/android-23/arch-arm64/symbols diff --git a/docs/Testing.md b/docs/Testing.md new file mode 100644 index 000000000..3d165cb85 --- /dev/null +++ b/docs/Testing.md @@ -0,0 +1,161 @@ +Testing the NDK +=============== + +The latest version of this document is available at +https://android.googlesource.com/platform/ndk/+/master/docs/Testing.md. + +Testing Tools +------------- + +There are currently three tools used in testing: + + 1. `run-all.py`, for testing a specific configuration. + 2. `validate.py`, for testing many configurations. + 3. `ndk-test.sh`, for testing libc++. + +A test configuration is a tuple of (ABI, target platform, toolchain, device). + +At some point the three of these will most likely merge into one script. + +### Testing a Single Configuration: [run-all.py] + +For targeted testing during development, `run-all.py` can be used to verify a +single test configuration, as well as run specific tests. + +Running the NDK tests requires a complete NDK package (see [README.md] for +building instructions). + +[README.md]: ../README.md + +From the NDK source directory (not the extracted package): + +```bash +$ python tests/run-all.py --abi $ABI_TO_TEST path/to/built/ndk +``` + +In the case of a local build, the path to the built NDK will be +`out/android-ndk-$RELEASE` unless you've overridden `OUT_DIR`. + +The default toolchain for testing is Clang. To run the tests with GCC, use the +option `--toolchain 4.9`. + +The full test suite includes tests which run on a device or emulator, so you'll +need to have `adb` in your path and `ANDROID_SERIAL` set if more than one +device/emulator is connected. If you do not have a device capable of running the +tests, you can run just the `build` or `awk` test suites with the `--suite` +flag. + +### Testing Multiple Configurations: [validate.py] + +When testing multiple configurations, `validate.py` will automatically detect +connected devices/emulators and choose a subset of them to fit our QA +configuration. This script will run all of the tests across all available ABIs +on several devices, and thus will take a long time (takes ~75 minutes on my +machine, even with a few configurations unavailable). As such, this isn't +suitable for active development, but should be run for QA and for any changes +that might have a wide impact (compiler updates, `ndk-build` changes, sysroot +changes, etc). + +To use this script, connect any devices and launch any emulators you need for +testing (make sure ADB has authorization to control them), then run: + +```bash +$ python tests/validate.py path/to/built/ndk +``` + +In the case of a local build, the path to the built NDK will be +`out/android-ndk-$RELEASE` unless you've overridden `OUT_DIR`. + +By default, test logs will be placed in $PWD/test-logs. This can be controlled +with the `--log-dir` flag. + +### Testing libc++: [ndk-test.sh] + +The libc++ tests are not currently integrated into the main NDK tests. To run +the libc++ tests: + +```bash +$ NDK=/path/to/extracted/ndk sources/cxx-stl/llvm-libc++/libcxx/ndk-test.sh $ABI +``` + +Note that these tests are far from failure free. In general, most of these tests +are locale related and fail because we don't support anything beyond the C +locale. + +Setting Up a Test Environment +----------------------------- + +To run the NDK tests, you will need: + + * An NDK. The NDK doesn't necessarily need to contain support for every + architecture. + * `adb` in your path. + * This is only needed if you're running device tests. + * Always use the latest available version of `adb`. Note that the version + of `adb` in the SDK manager might be surprisingly old. It's best to use a + version built fresh from AOSP. + * A device or emulator attached. + * Again, only needed for device tests. + +### Devices and Emulators + +For testing a release, make sure you're testing against the released builds of +Android. + +For Nexus devices, factory images are available here: +https://developers.google.com/android/nexus/images. Googlers, you may want to +use the flash station to get a userdebug image since that is needed for ASAN +testing. You should still make sure you also test on user builds because that is +what all of our users have. + +For emulators, use emulator images from the SDK rather than from a platform +build. Again, these are what our users will be using. + +After installing the emulator images from the SDK manager, they can be +configured and launched for testing with (assuming the SDK tools directory is in +your path): + +```bash +$ android create avd --name $NAME --target android-$LEVEL --abi $ABI +$ emulator -avd $NAME -no-window -no-audio -no-skin +``` + +This will create a new virtual device and launch it in a headless state. + +QA Configuration +---------------- + +The current configuration we use to test NDK releases is as written in +[validate.py]: + +```python +{ + 10: { + 'armeabi': None, + 'armeabi-v7a': None, + 'armeabi-v7a-hard': None, + }, + 16: { + 'armeabi': None, + 'armeabi-v7a': None, + 'armeabi-v7a-hard': None, + }, + 23: { + 'armeabi': None, + 'armeabi-v7a': None, + 'armeabi-v7a-hard': None, + 'arm64-v8a': None, + 'x86': None, + 'x86_64': None, + }, +} +``` + +Each API level/ABI pair will be checked with both Clang and GCC. + +Note that there are no ARM64 emulators whatsoever in the SDK manager. Testing +ARM64 will require a physical device. + +[run-all.py]: ../tests/run-all.py +[validate.py]: ../tests/validate.py +[ndk-test.sh]: ../sources/cxx-stl/llvm-libc++/libcxx/ndk-test.sh diff --git a/docs/Toolchains.md b/docs/Toolchains.md new file mode 100644 index 000000000..3be9c9b94 --- /dev/null +++ b/docs/Toolchains.md @@ -0,0 +1,130 @@ +Working with the NDK Toolchains +=============================== + +The latest version of this document is available at +https://android.googlesource.com/platform/ndk/+/master/docs/Testing.md. + +The toolchains shipped in the NDK are not built as a part of the NDK build +process. Instead they are built separately and checked into git as prebuilts +that are repackaged when shipped in the NDK. This applies to both Clang and GCC. + +Both toolchains are built separately. An artifact of the build is a tarball of +the compiler for distribution. That artifact is unpacked into a location in the +Android tree and checked in. The NDK build step for each toolchain simply packs +up the contents of the directory. + +Note: Any changes to either toolchain need to be tested in the platform *and* +the NDK. The platform and the NDK both get their toolchains from the same build. + +This process is far too manual. `checkbuild.py` should be updated (or additional +scripts added) to ease this process. + +Clang +----- + +Clang's build process is described in the platform's [ReadmeAndroid.md]. Note +that Clang cannot be built from the NDK tree. The output tarball is extracted to +`prebuilts/clang/host/$HOST/clang-$BUILD_NUMBER`. `checkbuild.py --module +clang` repackages this as-is as `$OUT_DIR/dist/llvm-$HOST_TAG.zip`. + +[ReadmeAndroid.md]: https://android.googlesource.com/platform/external/clang/+/dev/ReadmeAndroid.md + +### Testing Local Changes + +To test a Clang you just built: + +```bash +$ export CLANG_PREBUILTS=`realpath ../prebuilts/clang/host/linux-x86` +$ rm -r $CLANG_PREBUILTS/clang-dev +$ tar xf path/to/clang-dev-linux-x86_64.tar.bz2 -C $CLANG_PREBUILTS +# Edit ndk/build/tools/build-llvm.py and change the version to 'clang-dev'. +$ ../prebuilts/ndk/symlink-clang.py dev +$ ./checkbuild.py +# Run tests. +``` + +For details about running tests, see [Testing.md]. + +[Testing.md]: Testing.md + +This installs the new Clang into the prebuilts directory so it can be packaged +by the NDK build. The `symlink-clang.py` line updates the symlinks in prebuilts +NDK to point at the new Clang. The Clang in `prebuilts/ndk` is used for building +libc++ in the NDK. The difference between it and `prebuilts/clang` is the +directory layout, which differs so that `ndk-build` can use it. + +If you need to make changes to Clang after running the above steps, future +updates can be done more quickly with: + +```bash +$ rm -r $CLANG_PREBUILTS/clang-dev +$ tar xf path/to/clang-dev-linux-x86_64.bz2 -C $CLANG_PREBUILTS +$ ./checkbuild.py --module clang --force-package +# Run tests. +``` + +We don't need to rebuild the whole NDK since we've already built most of it. +Note that any NDK modules that were built with Clang won't be rebuilt (e.g. +libc++), so this is only going to be useful if you're debugging a test. + +### Updating to a New Clang + +These steps need to be run after installing the new prebuilt from the build +server to `prebuilts/clang` (see the Clang docs). + +```bash +# Edit ndk/build/tools/build-llvm.py and update the build number. +$ ../prebuilts/ndk/symlink/clang.py # Latest version autodetected. +$ ./checkbuid.py +# Run tests. +``` + +GCC (and binutils) +------------------ + +GCC's build process is described in [GCC's README.md][GCCReadme]. Unlike Clang, +GCC can be built from the NDK tree. The output tarball is extracted to +`prebuilts/ndk/current/toolchains/$TOOLCHAIN/prebuilt/$HOST_TAG`. `checkbuild.py +--module gcc` repackages this as-is as `$OUT_DIR/dist/gcc-$ARCH-$HOST_TAG.zip`. + +Note that GCC is a bit more complicated to update than Clang. `prebuilts/ndk` +contains only a prebuilt GCC, not prebuilt target gnustl (GNU's libstdc++, +referred to as gnustl in Android). As such, changes submitted to `toolchain/gcc` +will affect the gnustl build (`checkbuild.py --module gnustl`) will take effect +immediately. + +[GCCReadme]: https://android.googlesource.com/toolchain/gcc/+/master/README.md + +### Testing Local Changes + +To test a GCC you just built: + +```bash +$ export INSTALL_DIR=`realpath ../prebuilts/ndk/current` +$ export TOOLCHAIN_DIR=$INSTALL_DIR/toolchains/$HOST_TAG +$ rm -r $TOOLCHAIN_PATH/$TOOLCHAIN +$ unzip ../out/dist/gcc-$ARCH-$HOST.zip -d $TOOLCHAIN_DIR +$ ./checkbuild.py +# Run tests. +``` + +We're building an NDK and then replacing it's GCC with our own. + +For details about running tests, see [Testing.md]. + +Additional changes will not require a full `checkbuild.py`. Instead: + +```bash +$ ./checkbuild.py --module gcc +$ ./checkbuild.py --module gcclibs +$ ./checkbuild.py --module gnustl --force-package +# Run tests. +``` + +### Updating to a New GCC + +```bash +$ ../prebuilts/ndk/update-gcc.py $BUILD_NUMBER +$ ./checkbuild.py +# Run tests. +```