6.0.0 have a memory regression triggering OOM in JVM while doing android desugaring

[fishy]

Description of the bug:

I'm at the android directory under https://github.com/fishy/notifbot/tree/v0.9.1, with some local changes:

$ git diff --cached
diff --git a/android/.bazelversion b/android/.bazelversion
new file mode 100644
index 0000000..09b254e
--- /dev/null
+++ b/android/.bazelversion
@@ -0,0 +1 @@
+6.0.0
diff --git a/android/WORKSPACE b/android/WORKSPACE
index 20cef3b..c1d7579 100644
--- a/android/WORKSPACE
+++ b/android/WORKSPACE
@@ -3,6 +3,7 @@ workspace(name = "notifbot")
 android_sdk_repository(
     name = "androidsdk",
     api_level = 31,
+    build_tools_version = "30.0.3",  # this is required by bazel 5.4.0 but not by bazel 6.0.0
 )
 
 load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive", "http_jar")

Running it with bazel 6.0.0 will hit a memory issue on one of my machines:

$ bazelisk build :debug
INFO: Invocation ID: fbb49e2b-1d44-4dbc-a367-d98e961aeed4
INFO: Analyzed target //:debug (25 packages loaded, 553 targets configured).
INFO: Found 1 target...
ERROR: /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/maven/BUILD:300:11: Desugaring external/maven/v1/https/maven.google.com/androidx/concurrent/concurrent-futures/1.0.0/processed_concurrent-futures-1.0.0.jar for Android failed: Worker process returned an unparseable WorkResponse!

Did you try to print something to stdout? Workers aren't allowed to do this, as it breaks the protocol between Bazel and the worker process.

---8<---8<--- Start of response ---8<---8<---
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/execroot/notifbot/hs_err_pid13072.log
---8<---8<--- End of response ---8<---8<---

---8<---8<--- Exception details ---8<---8<---
(omitted)
---8<---8<--- End of exception details ---8<---8<---

---8<---8<--- Start of log, file at /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/bazel-workers/worker-19-Desugar.log ---8<---8<---
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000600000000, 8589934592, 0) failed; error='Not enough space' (errno=12)
---8<---8<--- End of log ---8<---8<---
Target //:debug failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 0.953s, Critical Path: 0.22s
INFO: 13 processes: 13 internal.
FAILED: Build did NOT complete successfully

Switching to 5.4.0 (by changing .bazelversion file) it will work correctly, so this is definitely an issue between 6.0.0 and 5.4.0.

The memory issue with 6.0.0 happens on a crostini environment (with 8GB total system memory, but according to free it seems crostini only got 6GB of them), but not on a linux machine with 32GB of memory. There are also some kernel diff which might contribute to it, but either way it should not require >6GB memory to build an Android app.

Crostini (with the issue):

$ free
               total        used        free      shared  buff/cache   available
Mem:         6654620     1587168     3764596        2324     1302856     5067452
Swap:              0           0           0
$ uname -a
Linux penguin 5.15.89-16172-g8db7d2810659 #1 SMP PREEMPT Mon Feb 27 18:39:10 PST 2023 x86_64 GNU/Linux

Linux machine (without the issue):

$ free
               total        used        free      shared  buff/cache   available
Mem:        32764736     1708304    10316676       51384    21259568    31056432
Swap:        8273916        1024     8272892
$ uname -a
Linux muto 6.1.0-5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.12-1 (2023-02-15) x86_64 GNU/Linux

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

See above.

Which operating system are you running Bazel on?

No response

What is the output of bazel info release?

release 6.0.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

$ git remote get-url origin; git rev-parse master; git rev-parse HEAD
git@github.com:fishy/notifbot.git
0f134d597e3bae4bc9ff5b65122ac16f57927adb
0f134d597e3bae4bc9ff5b65122ac16f57927adb

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

Full log file of `/home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/execroot/notifbot/hs_err_pid13072.log`

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.
# Possible reasons:
#   The system is out of physical RAM or swap space
#   The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (os_linux.cpp:3016), pid=13072, tid=13079
#
# JRE version:  (11.0.15+10) (build )
# Java VM: OpenJDK 64-Bit Server VM (11.0.15+10-LTS, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/sbin/crash_reporter --user=%P:%s:%u:%g:%f" (or dumping to /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/execroot/notifbot/core.13072)
#

---------------  S U M M A R Y ------------

Command Line: -XX:ErrorFile=/dev/stderr -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -Xms8g -Xmx8g -Dcom.android.tools.r8.sortMethodsOnCfWriting -Dcom.android.tools.r8.allowAllDesugaredInput -Dcom.android.tools.r8.noCfMarkerForDesugaredCode -Dcom.android.tools.r8.lambdaClassFieldsNotFinal -Dcom.android.tools.r8.createSingletonsForStatelessLambdas com.google.devtools.build.android.r8.Desugar --persistent_worker --desugared_lib_config /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/execroot/notifbot/bazel-out/k8-opt-exec-2B5CBBC6/bin/external/bazel_tools/tools/android/desugar_java8.runfiles/bazel_tools/tools/android/chm_only_desugar_jdk_libs.json

Host: 12th Gen Intel(R) Core(TM) i5-1235U, 12 cores, 6G, Debian GNU/Linux bookworm/sid
Time: Sat Mar  4 20:12:46 2023 PST elapsed time: 0.006343 seconds (0d 0h 0m 0s)

---------------  T H R E A D  ---------------

Current thread (0x00007b3078028000):  JavaThread "Unknown thread" [_thread_in_vm, id=13079, stack(0x00007b307d5df000,0x00007b307d6df000)]

Stack: [0x00007b307d5df000,0x00007b307d6df000],  sp=0x00007b307d6dd160,  free space=1016k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xe78546]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1c6
V  [libjvm.so+0xe794bb]  VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x2b
V  [libjvm.so+0x699c15]  report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xc5
V  [libjvm.so+0xbd1768]  os::pd_commit_memory_or_exit(char*, unsigned long, unsigned long, bool, char const*)+0xd8
V  [libjvm.so+0xbc79ed]  os::commit_memory_or_exit(char*, unsigned long, unsigned long, bool, char const*)+0x1d
V  [libjvm.so+0x7b4e6c]  G1PageBasedVirtualSpace::commit_preferred_pages(unsigned long, unsigned long)+0x6c
V  [libjvm.so+0x7b524e]  G1PageBasedVirtualSpace::commit(unsigned long, unsigned long)+0x19e
V  [libjvm.so+0x7be946]  G1RegionsLargerThanCommitSizeMapper::commit_regions(unsigned int, unsigned long, WorkGang*)+0x46
V  [libjvm.so+0x83a904]  HeapRegionManager::commit_regions(unsigned int, unsigned long, WorkGang*)+0x54
V  [libjvm.so+0x83b510]  HeapRegionManager::make_regions_available(unsigned int, unsigned int, WorkGang*)+0x30
V  [libjvm.so+0x83b937]  HeapRegionManager::expand_at(unsigned int, unsigned int, WorkGang*) [clone .part.25]+0xf7
V  [libjvm.so+0x772790]  G1CollectedHeap::expand(unsigned long, WorkGang*, double*)+0xe0
V  [libjvm.so+0x772e91]  G1CollectedHeap::initialize()+0x681
V  [libjvm.so+0xe3c211]  Universe::initialize_heap()+0x41
V  [libjvm.so+0xe3c4cc]  universe_init()+0x4c
V  [libjvm.so+0x85aa07]  init_globals()+0x57
V  [libjvm.so+0xe1614b]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x30b
V  [libjvm.so+0x9039f2]  JNI_CreateJavaVM+0x52
C  [libjli.so+0x3a68]  JavaMain+0x88
C  [libjli.so+0x8709]  ThreadJavaMain+0x9


---------------  P R O C E S S  ---------------

Threads class SMR info:
_java_thread_list=0x0000571fb16b9db0, length=0, elements={
}

Java Threads: ( => current thread )

Other Threads:
  0x00007b3078041800 GCTaskThread "GC Thread#0" [stack: 0x00007b307d294000,0x00007b307d394000] [id=13083]
  0x00007b3078063800 ConcurrentGCThread "G1 Main Marker" [stack: 0x00007b307c6c2000,0x00007b307c7c2000] [id=13084]
  0x00007b3078065800 ConcurrentGCThread "G1 Conc#0" [stack: 0x00007b307c5c0000,0x00007b307c6c0000] [id=13085]

[error occurred during error reporting (printing all threads), id 0xb, SIGSEGV (0xb) at pc=0x00007b307df4ae60]

VM state:not at safepoint (not fully initialized)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007b30780268f0] Heap_lock - owner thread: 0x00007b3078028000

Heap address: 0x0000000600000000, size: 8192 MB, Compressed Oops mode: Non-zero based: 0x0000000600000000
Narrow klass base: 0x0000000000000000, Narrow klass shift: 0

GC Heap History (0 events):
No events

Deoptimization events (0 events):
No events

Classes redefined (0 events):
No events

Internal exceptions (0 events):
No events

Events (2 events):
Event: 0.002 Protecting memory [0x00007b307d5df000,0x00007b307d5e3000] with protection modes 0
Event: 0.002 Loaded shared library /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libzip.so


Dynamic libraries:
571fafff4000-571fafff5000 r-xp 00000000 00:2c 8115709                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/bin/java
571fafff6000-571fafff7000 r--p 00001000 00:2c 8115709                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/bin/java
571fafff7000-571fafff8000 rw-p 00002000 00:2c 8115709                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/bin/java
571fb16b5000-571fb16d6000 rw-p 00000000 00:00 0                          [heap]
7b303c000000-7b303c021000 rw-p 00000000 00:00 0 
7b303c021000-7b3040000000 ---p 00000000 00:00 0 
7b3042000000-7b3044000000 rw-p 00000000 00:00 0 
7b3044000000-7b3044021000 rw-p 00000000 00:00 0 
7b3044021000-7b3048000000 ---p 00000000 00:00 0 
7b3048d8b000-7b3049000000 rw-p 00000000 00:00 0 
7b3049000000-7b305c000000 ---p 00000000 00:00 0 
7b305c000000-7b305c021000 rw-p 00000000 00:00 0 
7b305c021000-7b3060000000 ---p 00000000 00:00 0 
7b306015e000-7b30607b4000 rw-p 00000000 00:00 0 
7b30607b4000-7b3060a24000 rwxp 00000000 00:00 0 
7b3060a24000-7b3060ee8000 ---p 00000000 00:00 0 
7b3060ee8000-7b3061158000 rwxp 00000000 00:00 0 
7b3061158000-7b306f7b4000 ---p 00000000 00:00 0 
7b306f7b4000-7b3078000000 r--s 00000000 00:2c 8115677                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/modules
7b3078000000-7b307809c000 rw-p 00000000 00:00 0 
7b307809c000-7b307c000000 ---p 00000000 00:00 0 
7b307c0b9000-7b307c5be000 rw-p 00000000 00:00 0 
7b307c5be000-7b307c5bf000 ---p 00000000 00:00 0 
7b307c5bf000-7b307c6c0000 rw-p 00000000 00:00 0 
7b307c6c0000-7b307c6c1000 ---p 00000000 00:00 0 
7b307c6c1000-7b307d292000 rw-p 00000000 00:00 0 
7b307d292000-7b307d293000 ---p 00000000 00:00 0 
7b307d293000-7b307d399000 rw-p 00000000 00:00 0 
7b307d399000-7b307d566000 ---p 00000000 00:00 0 
7b307d566000-7b307d56b000 rw-p 00000000 00:00 0 
7b307d56b000-7b307d575000 ---p 00000000 00:00 0 
7b307d575000-7b307d57c000 r-xp 00000000 00:2c 8115682                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libzip.so
7b307d57c000-7b307d57d000 ---p 00007000 00:2c 8115682                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libzip.so
7b307d57d000-7b307d57e000 r--p 00007000 00:2c 8115682                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libzip.so
7b307d57e000-7b307d57f000 rw-p 00008000 00:2c 8115682                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libzip.so
7b307d57f000-7b307d59a000 r-xp 00000000 00:2c 8115629                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libjimage.so
7b307d59a000-7b307d59b000 ---p 0001b000 00:2c 8115629                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libjimage.so
7b307d59b000-7b307d59d000 r--p 0001b000 00:2c 8115629                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libjimage.so
7b307d59d000-7b307d59e000 rw-p 0001d000 00:2c 8115629                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libjimage.so
7b307d59e000-7b307d5c7000 r-xp 00000000 00:2c 8115634                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libjava.so
7b307d5c7000-7b307d5c8000 r--p 00028000 00:2c 8115634                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libjava.so
7b307d5c8000-7b307d5c9000 rw-p 00029000 00:2c 8115634                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libjava.so
7b307d5c9000-7b307d5ca000 rw-p 00000000 00:00 0 
7b307d5ca000-7b307d5d7000 r-xp 00000000 00:2c 8115670                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libverify.so
7b307d5d7000-7b307d5d9000 r--p 0000c000 00:2c 8115670                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libverify.so
7b307d5d9000-7b307d5da000 rw-p 0000e000 00:2c 8115670                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/libverify.so
7b307d5da000-7b307d5db000 r--p 00000000 00:2c 4250314                    /usr/lib/x86_64-linux-gnu/librt.so.1
7b307d5db000-7b307d5dc000 r-xp 00001000 00:2c 4250314                    /usr/lib/x86_64-linux-gnu/librt.so.1
7b307d5dc000-7b307d5dd000 r--p 00002000 00:2c 4250314                    /usr/lib/x86_64-linux-gnu/librt.so.1
7b307d5dd000-7b307d5de000 r--p 00002000 00:2c 4250314                    /usr/lib/x86_64-linux-gnu/librt.so.1
7b307d5de000-7b307d5df000 rw-p 00003000 00:2c 4250314                    /usr/lib/x86_64-linux-gnu/librt.so.1
7b307d5df000-7b307d5e3000 ---p 00000000 00:00 0 
7b307d5e3000-7b307d6df000 rw-p 00000000 00:00 0 
7b307d6df000-7b307d6ef000 r--p 00000000 00:2c 4250303                    /usr/lib/x86_64-linux-gnu/libm.so.6
7b307d6ef000-7b307d762000 r-xp 00010000 00:2c 4250303                    /usr/lib/x86_64-linux-gnu/libm.so.6
7b307d762000-7b307d7bc000 r--p 00083000 00:2c 4250303                    /usr/lib/x86_64-linux-gnu/libm.so.6
7b307d7bc000-7b307d7bd000 r--p 000dc000 00:2c 4250303                    /usr/lib/x86_64-linux-gnu/libm.so.6
7b307d7bd000-7b307d7be000 rw-p 000dd000 00:2c 4250303                    /usr/lib/x86_64-linux-gnu/libm.so.6
7b307d7be000-7b307e93f000 r-xp 00000000 00:2c 8115660                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/server/libjvm.so
7b307e93f000-7b307e940000 ---p 01181000 00:2c 8115660                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/server/libjvm.so
7b307e940000-7b307ea05000 r--p 01181000 00:2c 8115660                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/server/libjvm.so
7b307ea05000-7b307ea40000 rw-p 01246000 00:2c 8115660                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/server/libjvm.so
7b307ea40000-7b307ea9c000 rw-p 00000000 00:00 0 
7b307ea9c000-7b307eac2000 r--p 00000000 00:2c 4250300                    /usr/lib/x86_64-linux-gnu/libc.so.6
7b307eac2000-7b307ec17000 r-xp 00026000 00:2c 4250300                    /usr/lib/x86_64-linux-gnu/libc.so.6
7b307ec17000-7b307ec6a000 r--p 0017b000 00:2c 4250300                    /usr/lib/x86_64-linux-gnu/libc.so.6
7b307ec6a000-7b307ec6e000 r--p 001ce000 00:2c 4250300                    /usr/lib/x86_64-linux-gnu/libc.so.6
7b307ec6e000-7b307ec70000 rw-p 001d2000 00:2c 4250300                    /usr/lib/x86_64-linux-gnu/libc.so.6
7b307ec70000-7b307ec7d000 rw-p 00000000 00:00 0 
7b307ec7d000-7b307ec7e000 r--p 00000000 00:2c 4250302                    /usr/lib/x86_64-linux-gnu/libdl.so.2
7b307ec7e000-7b307ec7f000 r-xp 00001000 00:2c 4250302                    /usr/lib/x86_64-linux-gnu/libdl.so.2
7b307ec7f000-7b307ec80000 r--p 00002000 00:2c 4250302                    /usr/lib/x86_64-linux-gnu/libdl.so.2
7b307ec80000-7b307ec81000 r--p 00002000 00:2c 4250302                    /usr/lib/x86_64-linux-gnu/libdl.so.2
7b307ec81000-7b307ec82000 rw-p 00003000 00:2c 4250302                    /usr/lib/x86_64-linux-gnu/libdl.so.2
7b307ec82000-7b307ec91000 r-xp 00000000 00:2c 8115679                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/jli/libjli.so
7b307ec91000-7b307ec92000 r--p 0000e000 00:2c 8115679                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/jli/libjli.so
7b307ec92000-7b307ec93000 rw-p 0000f000 00:2c 8115679                    /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/remotejdk11_linux/lib/jli/libjli.so
7b307ec93000-7b307ec94000 r--p 00000000 00:2c 4250312                    /usr/lib/x86_64-linux-gnu/libpthread.so.0
7b307ec94000-7b307ec95000 r-xp 00001000 00:2c 4250312                    /usr/lib/x86_64-linux-gnu/libpthread.so.0
7b307ec95000-7b307ec96000 r--p 00002000 00:2c 4250312                    /usr/lib/x86_64-linux-gnu/libpthread.so.0
7b307ec96000-7b307ec97000 r--p 00002000 00:2c 4250312                    /usr/lib/x86_64-linux-gnu/libpthread.so.0
7b307ec97000-7b307ec98000 rw-p 00003000 00:2c 4250312                    /usr/lib/x86_64-linux-gnu/libpthread.so.0
7b307ec98000-7b307ec9b000 r--p 00000000 00:2c 2320856                    /usr/lib/x86_64-linux-gnu/libz.so.1.2.13
7b307ec9b000-7b307ecae000 r-xp 00003000 00:2c 2320856                    /usr/lib/x86_64-linux-gnu/libz.so.1.2.13
7b307ecae000-7b307ecb5000 r--p 00016000 00:2c 2320856                    /usr/lib/x86_64-linux-gnu/libz.so.1.2.13
7b307ecb5000-7b307ecb6000 r--p 0001c000 00:2c 2320856                    /usr/lib/x86_64-linux-gnu/libz.so.1.2.13
7b307ecb6000-7b307ecb7000 rw-p 0001d000 00:2c 2320856                    /usr/lib/x86_64-linux-gnu/libz.so.1.2.13
7b307ecb9000-7b307ecc1000 rw-s 00000000 00:2c 8229764                    /tmp/hsperfdata_fishy/13072
7b307ecc1000-7b307ecc2000 ---p 00000000 00:00 0 
7b307ecc2000-7b307ecc3000 r--p 00000000 00:00 0 
7b307ecc3000-7b307ecc5000 rw-p 00000000 00:00 0 
7b307ecc5000-7b307ecc6000 r--p 00000000 00:2c 4250297                    /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7b307ecc6000-7b307eceb000 r-xp 00001000 00:2c 4250297                    /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7b307eceb000-7b307ecf5000 r--p 00026000 00:2c 4250297                    /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7b307ecf5000-7b307ecf7000 r--p 00030000 00:2c 4250297                    /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7b307ecf7000-7b307ecf9000 rw-p 00032000 00:2c 4250297                    /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
7ffc7f03d000-7ffc7f060000 rw-p 00000000 00:00 0                          [stack]
7ffc7f191000-7ffc7f195000 r--p 00000000 00:00 0                          [vvar]
7ffc7f195000-7ffc7f197000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]


VM Arguments:
jvm_args: -XX:ErrorFile=/dev/stderr -XX:+TieredCompilation -XX:TieredStopAtLevel=1 -Xms8g -Xmx8g -Dcom.android.tools.r8.sortMethodsOnCfWriting -Dcom.android.tools.r8.allowAllDesugaredInput -Dcom.android.tools.r8.noCfMarkerForDesugaredCode -Dcom.android.tools.r8.lambdaClassFieldsNotFinal -Dcom.android.tools.r8.createSingletonsForStatelessLambdas 
java_command: com.google.devtools.build.android.r8.Desugar --persistent_worker --desugared_lib_config /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/execroot/notifbot/bazel-out/k8-opt-exec-2B5CBBC6/bin/external/bazel_tools/tools/android/desugar_java8.runfiles/bazel_tools/tools/android/chm_only_desugar_jdk_libs.json
java_class_path (initial): bazel-out/k8-opt-exec-2B5CBBC6/bin/external/bazel_tools/tools/android/desugar_java8.runfiles/notifbot/../bazel_tools/src/tools/android/java/com/google/devtools/build/android/r8/libr8.jar:bazel-out/k8-opt-exec-2B5CBBC6/bin/external/bazel_tools/tools/android/desugar_java8.runfiles/notifbot/../android_gmaven_r8/jar/downloaded.jar:bazel-out/k8-opt-exec-2B5CBBC6/bin/external/bazel_tools/tools/android/desugar_java8.runfiles/notifbot/../android_tools/all_android_tools_deploy.jar
Launcher Type: SUN_STANDARD

[Global flags]
     intx CICompilerCount                          = 4                                         {product} {ergonomic}
     uint ConcGCThreads                            = 3                                         {product} {ergonomic}
    ccstr ErrorFile                                = /dev/stderr                               {product} {command line}
     uint G1ConcRefinementThreads                  = 10                                        {product} {ergonomic}
   size_t G1HeapRegionSize                         = 4194304                                   {product} {ergonomic}
    uintx GCDrainStackTargetSize                   = 64                                        {product} {ergonomic}
   size_t InitialHeapSize                          = 8589934592                                {product} {command line}
   size_t MarkStackSize                            = 4194304                                   {product} {ergonomic}
   size_t MaxHeapSize                              = 8589934592                                {product} {command line}
   size_t MinHeapDeltaBytes                        = 4194304                                   {product} {ergonomic}
    uintx NonNMethodCodeHeapSize                   = 7549744                                {pd product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 244108496                              {pd product} {ergonomic}
    uintx ProfiledCodeHeapSize                     = 0                                      {pd product} {ergonomic}
    uintx ReservedCodeCacheSize                    = 251658240                              {pd product} {ergonomic}
     bool SegmentedCodeCache                       = true                                      {product} {ergonomic}
     bool TieredCompilation                        = true                                   {pd product} {command line}
     intx TieredStopAtLevel                        = 1                                         {product} {command line}
     bool UseCompressedClassPointers               = true                                 {lp64_product} {ergonomic}
     bool UseCompressedOops                        = true                                 {lp64_product} {ergonomic}
     bool UseG1GC                                  = true                                      {product} {ergonomic}

Logging:
Log output configuration:
 #0: stdout all=warning uptime,level,tags
 #1: stderr all=off uptime,level,tags

Environment Variables:
PATH=/home/fishy/.cache/bazelisk/downloads/bazelbuild/bazel-6.0.0-linux-x86_64/bin:/home/fishy/.gopath/bin:/home/fishy/.nix-profile/bin:/nix/var/nix/profiles/default/bin:/home/fishy/.local/bin:/home/fishy/bin:/home/fishy/work/android-sdk/tools:/home/fishy/work/android-sdk/platform-tools:/home/fishy/work/android-sdk/build-tools/33.0.2:/home/fishy/work/android-sdk/cmdline-tools/latest/bin:/usr/lib/go-1.19/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
LC_CTYPE=C.UTF-8

Signal Handlers:
SIGSEGV: [libjvm.so+0xe79520], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xe79520], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0xe79520], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0xbcc2a0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0xbcc2a0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0xe79520], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.so+0xbcc130], sa_mask[0]=00100000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGINT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGQUIT: SIG_DFL, sa_mask[0]=00000000000000000000000000000000, sa_flags=none


---------------  S Y S T E M  ---------------

OS:PRETTY_NAME="Debian GNU/Linux bookworm/sid"
NAME="Debian GNU/Linux"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
uname:Linux 5.15.89-16172-g8db7d2810659 #1 SMP PREEMPT Mon Feb 27 18:39:10 PST 2023 x86_64
OS uptime: 0 days 9:39 hours
libc:glibc 2.36 NPTL 2.36 
rlimit (soft/hard): STACK 8192k/infinity , CORE 0k/infinity , NPROC 25975/25975 , NOFILE 1048576/1048576 , AS infinity/infinity , CPU infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity , MEMLOCK infinity/infinity
load average:0.01 0.54 1.90

/proc/meminfo:
MemTotal:        6654620 kB
MemFree:         3703380 kB
MemAvailable:    5023944 kB
Buffers:               0 kB
Cached:          1320564 kB
SwapCached:            0 kB
Active:           926808 kB
Inactive:        1953000 kB
Active(anon):        316 kB
Inactive(anon):  1561256 kB
Active(file):     926492 kB
Inactive(file):   391744 kB
Unevictable:        2096 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:              1712 kB
Writeback:             0 kB
AnonPages:       1559248 kB
Mapped:            99560 kB
Shmem:              2324 kB
KReclaimable:      72400 kB
Slab:                  0 kB
SReclaimable:          0 kB
SUnreclaim:            0 kB
KernelStack:        9660 kB
PageTables:        10636 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     3327308 kB
Committed_AS:    3440844 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       10884 kB
VmallocChunk:          0 kB
Percpu:             6656 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      141312 kB
DirectMap2M:     6139904 kB
DirectMap1G:     2097152 kB

/sys/kernel/mm/transparent_hugepage/enabled:
always [madvise] never
/sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):
always defer defer+madvise [madvise] never

Process Memory:
Virtual Size: 1046152K (peak: 9453384K)
Resident Set Size: 18436K (peak: 18436K) (anon: 6072K, file: 12364K, shmem: 0K)
Swapped out: 0K
C-Heap outstanding allocations: 726K

/proc/sys/kernel/threads-max (system-wide limit on the number of threads):
51950
/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):
262144
/proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):
32768

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-11
cpu_memory_nodes: 0
active_processor_count: 12
cpu_quota: -1
cpu_period: 100000
cpu_shares: -1
memory_limit_in_bytes: -1
memory_and_swap_limit_in_bytes: -2
memory_soft_limit_in_bytes: -1
memory_usage_in_bytes: 2920148992
memory_max_usage_in_bytes: 5220859904

KVM virtualization detected
Steal ticks since vm start: 0
Steal ticks percentage since vm start:  0.000

CPU:total 12 (initial active 12) (12 cores per cpu, 1 threads per core) family 6 model 154 stepping 4 microcode 0x1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref, lzcnt, tsc, tscinvbit, bmi1, bmi2, adx, sha, fma
CPU Model and flags from /proc/cpuinfo:
model name      : 12th Gen Intel(R) Core(TM) i5-1235U
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq vmx ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves arat umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize arch_capabilities
Online cpus:
0-11
Offline cpus:

BIOS frequency limitation:
<Not Available>
Frequency switch latency (ns):
<Not Available>
Available cpu frequencies:
<Not Available>
Current governor:
<Not Available>
Core performance/turbo boost:
<Not Available>

Memory: 4k page, physical 6654620k(443424k free), swap 0k(0k free)

vm_info: OpenJDK 64-Bit Server VM (11.0.15+10-LTS) for linux-amd64 JRE (Zulu11.56+19-CA) (11.0.15+10-LTS), built on Apr 17 2022 05:03:56 by "zulu_re" with gcc 8.3.0

END.

[Wyverald]

This happened with the Andrdoid desugarer but it might be more relevant for @larsrc-google since it's about workers.

[larsrc-google]

I don't know the details of all the worker implementations. Does it happen without the build_tools_version change on 6.0.0? You could check with bazel aquery if the two Bazel versions have the same number of inputs? 6GB is not very much for compiling with workers, I will say. You might want to reduce --worker_max_instances if you haven't already. You could also look at the profile output to see if 6.0.0 uses significantly more memory.

[fishy]

@larsrc-google yes it hits the same OOM issue if I remove the build_tools_version line, and yes it still hits the same OOM issue if I run bazelisk build --worker_max_instances=2 :debug with 6.0.0.

[larsrc-google]

Ok, so I'm trying to reproduce this on my Linux box, using cgroups to limit the amount of memory available for the Bazel process and its subprocesses. At 1GB, it's marginal, sometimes working, sometimes crashing, always slow (because the JVM notices cgroups memory limits and reduces heap accordingly).

$ XDG_RUNTIME_DIR=/run/user/$( id -u ) systemd-run --user --scope bash
$ cat /proc/self/cgroup 
0::/user.slice/user-121084.slice/user@121084.service/app.slice/run-r0c139c46d1ba419a8b1bdce51faf2118.scope
$ echo 1200000000 > /sys/fs/cgroup/user.slice/user-121084.slice/user@121084.service/app.slice/run-r0c139c46d1ba419a8b1bdce51faf2118.scope/memory.max
$ bazel-6.0.0 --bazelrc=/dev/null build :debug --noexperimental_worker_multiplex
[...]
INFO: Build completed successfully, 693 total actions
$ echo 1000000000 > /sys/fs/cgroup/user.slice/user-121084.slice/user@121084.service/app.slice/run-r0c139c46d1ba419a8b1bdce51faf2118.scope/memory.max
$ bazel-6.0.0 --bazelrc=/dev/null build :debug --noexperimental_worker_multiplex
[...]
ERROR: /usr/local/google/home/larsrc/.cache/bazel/_bazel_larsrc/41f74d7762b46d7434366991fb177523/external/maven/BUILD:70:11: Merging compiled Android resources for @maven//:androidx_annotation_annotation_experimental failed: (Aborted): ResourceProcessorBusyBox failed: error executing command (from target @maven//:androidx_annotation_annotation_experimental) bazel-out/k8-opt-exec-2B5CBBC6/bin/external/bazel_tools/src/tools/android/java/com/google/devtools/build/android/ResourceProcessorBusyBox --tool MERGE_COMPILED -- --classJarOutput ... (remaining 11 arguments skipped)

Use --sandbox_debug to see verbose messages from the sandbox and retain the sandbox build root for debugging
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007fa85011d0c0, pid=2, tid=6
[...]

That's not the same error as yours. The line Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory is most curious - why would it try to allocate 8GB? That's a lot!

You also said you have some kernel diffs, does it still happen without those? I'm having trouble recreating it, even with --local_ram_resources=HOST_RAM*2 and --worker_max_instances=8. Could you show what these workers' command lines are? If for some reason they try to set a huge heap that there's no room for in the Crostini, that's obviously bad. I don't see any indication that we're doing that in the rule def, though (

bazel/src/main/java/com/google/devtools/build/lib/rules/android/DexArchiveAspect.java

Line 537 in 56a9efb

.setMnemonic("Desugar")

)

[fishy]

I tried to add --verbose_failures but it doesn't really give me the full raw command it ran, so I still don't know what it actually tried to do.

But the line it pointed out in the generated build file (from the output of ERROR: /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/maven/BUILD:144:11: Desugaring external/maven/_aar/androidx_appcompat_appcompat/classes_and_libs_merged.jar for Android failed: Worker process returned an unparseable WorkResponse!) is just an aar_import rule:

$ cat --number /home/fishy/.cache/bazel/_bazel_fishy/321239f8e86392297cce34a42c99650d/external/maven/BUILD | egrep -A 21 -w "144"
   144  aar_import(
   145          name = "androidx_appcompat_appcompat",
   146          aar = "v1/https/maven.google.com/androidx/appcompat/appcompat/1.3.1/appcompat-1.3.1.aar",
   147          deps = [
   148                  ":androidx_activity_activity",
   149                  ":androidx_annotation_annotation",
   150                  ":androidx_appcompat_appcompat_resources",
   151                  ":androidx_collection_collection",
   152                  ":androidx_core_core",
   153                  ":androidx_cursoradapter_cursoradapter",
   154                  ":androidx_drawerlayout_drawerlayout",
   155                  ":androidx_fragment_fragment",
   156                  ":androidx_lifecycle_lifecycle_runtime",
   157                  ":androidx_lifecycle_lifecycle_viewmodel",
   158                  ":androidx_savedstate_savedstate",
   159          ],
   160          tags = [
   161                  "maven_coordinates=androidx.appcompat:appcompat:1.3.1",
   162                  "maven_url=https://maven.google.com/androidx/appcompat/appcompat/1.3.1/appcompat-1.3.1.aar",
   163          ],
   164          visibility = ["//visibility:public"],
   165  )

so maybe something changed in aar_import in 6.0.0 that triggered the bug?

if you have some other args that could produce more debugging info I'm happy to run them.

[larsrc-google]

I would expect this would be a rules change rather than a Bazel change. @timpeut, could you take a look?

[AnDevi]

I have the same issue as @fishy . I'm using bazel 7.1.0

ERROR: project/myapp/src/BUILD.bazel:68:15: Desugaring projext/myapp/app-sdk_resources.jar for Android failed: Worker process returned an unparseable WorkResponse!
Did you try to print something to stdout? Workers aren't allowed to do this, as it breaks the protocol between Bazel and the worker process.
---8<---8<--- Start of response ---8<---8<---
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (mmap) failed to map 8589934592 bytes for committing reserved memory.
# An error report file with more information is saved as:
# /home/gitlab-runner/.cache/bazel/_bazel_gitlab-runner/c3520769e8c575c6ddec9182e3131931/execroot/_main/hs_err_pid1135.log
---8<---8<--- End of response ---8<---8<---
---8<---8<--- Exception details ---8<---8<---
---8<---8<--- End of exception details ---8<---8<---
---8<---8<--- Start of log, file at /home/gitlab-runner/.cache/bazel/_bazel_gitlab-runner/c3520769e8c575c6ddec9182e3131931/bazel-workers/worker-3-Desugar.log ---8<---8<---
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000600000000, 8589934592, 0) failed; error='Not enough space' (errno=12)
---8<---8<--- End of log ---8<---8<---

I can build it on my Mac with 16gb of RAM, I can build it inside docker in which I limit RAM even to 2GBs, but when I building it on our CI on AWS runner I got a problem with memory. This runner has 8GB of RAM, when I switch it 16GB compilation is successful, but I would rather want to stay with this 8GB's one.

I tried compilation with flags --host_jvm_args=-Xmx4g, --strategy=Desugar=sandboxed, --javacopt=-J-Xmx4g, --worker_max_instances=2 non of it helps in here.

What I think it's also interesting I my case it tries to allocate the same amount of memory (8589934592 ) as in @fishy 's case ..

[AnDevi]

I made additional test and I limited my app to empty android_binary no srcs and no deps at all:

android_binary(
    name = "test_app",
    manifest = "main/AndroidManifest.xml",
)

I'm building just it and I still facing the same problem.

INFO: Analyzed target //apps/test (108 packages loaded, 7167 targets configured).

This log is also little disturbing, why such a large number of targets has been configured for such a small project, may it be connected to the problem that @fishy and I described ?