Bazel CI: Android test failing with Bazel 4.1.0

[meteorcloudy]

https://buildkite.com/bazel/android-testing/builds/1309#

(04:08:01) ERROR: /workdir/ui/espresso/BasicSample/BUILD.bazel:17:15: Zipaligning apk failed: (Exit 127): zipalign_binary failed: error executing command
  (cd /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/ec321eb2cc2d0f8f91b676b6d4c66c29/execroot/__main__ && \
  exec env - \
  bazel-out/k8-py2-opt-exec-EEBBF29D/bin/external/androidsdk/zipalign_binary -p 4 bazel-out/k8-fastbuild/bin/ui/espresso/BasicSample/BasicSample_unsigned.apk bazel-out/k8-fastbuild/bin/ui/espresso/BasicSample/_dx/BasicSample/zipaligned_BasicSample.apk)
Execution platform: //:android_platform
bazel-out/k8-py2-opt-exec-EEBBF29D/bin/external/androidsdk/zipalign_binary.runfiles/androidsdk/build-tools/30.0.1/zipalign: error while loading shared libraries: /usr/local/lib/libc++.so: file too short
(04:08:01) INFO: Elapsed time: 39.761s, Critical Path: 5.02s
(04:08:01) INFO: 1693 processes: 1313 remote cache hit, 380 internal.
//ui/espresso/BasicSample:BasicSampleInstrumentationTest_19_x86       NO STATUS
//ui/espresso/BasicSample:BasicSampleInstrumentationTest_21_x86       NO STATUS
//ui/espresso/BasicSample:BasicSampleInstrumentationTest_22_x86       NO STATUS
//ui/espresso/BasicSample:BasicSampleInstrumentationTest_23_x86       NO STATUS
//ui/espresso/CustomMatcherSample:CustomMatcherSampleInstrumentationTest_19_x86 NO STATUS
//ui/espresso/CustomMatcherSample:CustomMatcherSampleInstrumentationTest_21_x86 NO STATUS
//ui/espresso/CustomMatcherSample:CustomMatcherSampleInstrumentationTest_22_x86 NO STATUS
//ui/espresso/CustomMatcherSample:CustomMatcherSampleInstrumentationTest_23_x86 NO STATUS
//ui/espresso/DataAdapterSample:DataAdapterSampleInstrumentationTest_19_x86 NO STATUS
//ui/espresso/DataAdapterSample:DataAdapterSampleInstrumentationTest_21_x86 NO STATUS
//ui/espresso/DataAdapterSample:DataAdapterSampleInstrumentationTest_22_x86 NO STATUS
//ui/espresso/DataAdapterSample:DataAdapterSampleInstrumentationTest_23_x86 NO STATUS
//ui/espresso/IdlingResourceSample:IdlingResourceSampleInstrumentationTest_19_x86 NO STATUS
//ui/espresso/IdlingResourceSample:IdlingResourceSampleInstrumentationTest_21_x86 NO STATUS
//ui/espresso/IdlingResourceSample:IdlingResourceSampleInstrumentationTest_22_x86 NO STATUS
//ui/espresso/IdlingResourceSample:IdlingResourceSampleInstrumentationTest_23_x86 NO STATUS
//ui/espresso/IntentsAdvancedSample:IntentsAdvancedSampleInstrumentationTest_19_x86 NO STATUS
//ui/espresso/IntentsAdvancedSample:IntentsAdvancedSampleInstrumentationTest_21_x86 NO STATUS
//ui/espresso/IntentsAdvancedSample:IntentsAdvancedSampleInstrumentationTest_22_x86 NO STATUS
//ui/espresso/IntentsAdvancedSample:IntentsAdvancedSampleInstrumentationTest_23_x86 NO STATUS
//ui/espresso/IntentsBasicSample:IntentsBasicSampleInstrumentationTest_19_x86 NO STATUS
//ui/espresso/IntentsBasicSample:IntentsBasicSampleInstrumentationTest_21_x86 NO STATUS
//ui/espresso/IntentsBasicSample:IntentsBasicSampleInstrumentationTest_22_x86 NO STATUS
//ui/espresso/IntentsBasicSample:IntentsBasicSampleInstrumentationTest_23_x86 NO STATUS
//ui/espresso/MultiWindowSample:MultiWindowSampleInstrumentationTest_19_x86 NO STATUS
//ui/espresso/MultiWindowSample:MultiWindowSampleInstrumentationTest_21_x86 NO STATUS
//ui/espresso/MultiWindowSample:MultiWindowSampleInstrumentationTest_22_x86 NO STATUS
//ui/espresso/MultiWindowSample:MultiWindowSampleInstrumentationTest_23_x86 NO STATUS
//ui/espresso/RecyclerViewSample:RecyclerViewSampleInstrumentationTest_19_x86 NO STATUS
//ui/espresso/RecyclerViewSample:RecyclerViewSampleInstrumentationTest_21_x86 NO STATUS
//ui/espresso/RecyclerViewSample:RecyclerViewSampleInstrumentationTest_22_x86 NO STATUS
//ui/espresso/RecyclerViewSample:RecyclerViewSampleInstrumentationTest_23_x86 NO STATUS
//ui/uiautomator/BasicSample:BasicSampleInstrumentationTest_19_x86    NO STATUS
//ui/uiautomator/BasicSample:BasicSampleInstrumentationTest_22_x86    NO STATUS
//ui/uiautomator/BasicSample:BasicSampleInstrumentationTest_23_x86    NO STATUS
//ui/uiautomator/BasicSample:BasicSampleInstrumentationTest_21_x86 FAILED TO BUILD
 
Executed 0 out of 36 tests: 1 fails to build and 35 were skipped.

Culprit should be bazel-contrib/rules_jvm_external#562

[meteorcloudy]

bazel-out/k8-py2-opt-exec-EEBBF29D/bin/external/androidsdk/zipalign_binary.runfiles/androidsdk/build-tools/30.0.1/zipalign: error while loading shared libraries: /usr/local/lib/libc++.so: file too short

@philwo Does this looks like something wrong with the Android SKD installation on the VM?

[philwo]

@meteorcloudy I think this is the same issue as this one: #384 (comment)

IIRC the wrapper script somehow didn't symlink the required library correctly. :/

[meteorcloudy]

I hacked the wrapper script a bit to print some information:

-rwxr-xr-x 3 nobody nogroup 1081224 Jun 23 18:15 bazel-out/k8-py2-opt-exec-EEBBF29D/bin/external/androidsdk/zipalign_binary.runfiles/androidsdk/build-tools/30.0.1/lib64/libc++.so
--------------
--------------
-rw-r--r-- 1 root root 28 Jan  1  1970 /usr/local/lib/libc++.so

It shows the desired libc++.so is indeed in the runfiles directory, but somehow it still loads the one under /usr/local/lib/libc++.so, which looks like an empty file. Is /usr/local/lib always preferred when dynamically loading a shared library?

[meteorcloudy]

/cc @ahumesky

[meteorcloudy]

The dynamic linker search path order is explained at https://unix.stackexchange.com/questions/367600/what-is-the-order-that-linuxs-dynamic-linker-searches-paths-in

I suspect there is a change on the RBE docker image that somehow introduced /usr/local/lib/libc++.so, and /usr/local/lib is presented in /etc/ld.so.conf.

I'm able to fix the problem by adding "export LD_LIBRARY_PATH=$${SDK}/build-tools/%s/lib64" % build_tools_directory, to the wrapper script.

So I think the solutions are:

• Fix the docker image to remove the empty /usr/local/lib/libc++.so. We don't have this file on our CI docker image.
• Set LD_LIBRARY_PATH in the wrapper script to make sure the libc++ in the runfiles tree are always found first, but this require changes in Bazel.

[meteorcloudy]

FYI @comius

[coeuvre]

Another way to fix this is using Bazel CI image on RBE. I am working with @philwo to find an easy way for downstream projects to use the pre-generated configurations by rbe_configs_gen.

[comius]

This seems to be fixed, but underlying issue (bazelbuild/bazel#13409) popped back up.

[coeuvre]

Created #403 to use Bazel CI image on RBE. But build fails with other errors:

ui/espresso/IntentsAdvancedSample/app/src/androidTest/java/com/example/android/testing/espresso/intents/AdvancedSample/ImageViewHasDrawableMatcher.java:19: error: cannot find symbol
import androidx.test.espresso.matcher.BoundedDiagnosingMatcher;
                                     ^
  symbol:   class BoundedDiagnosingMatcher
  location: package androidx.test.espresso.matcher

Any ideas?

[mrk-han]

Can close and track this in #417