IntentService -- OutOfMemoryError

[jia-tse-zocdoc]

Hi,

I'm seeing this error, happening only on Android 8 (Oreo) devices.

Fatal Exception: java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Try again
       at java.lang.Thread.nativeCreate(Thread.java)
       at java.lang.Thread.start(Thread.java:733)
       at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:970)
       at java.util.concurrent.ThreadPoolExecutor.processWorkerExit(ThreadPoolExecutor.java:1038)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1180)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
       at java.lang.Thread.run(Thread.java:764)

The crash console is showing there are 1177 threads created (doubt this is intentional!). And most of the threads look like this:

IntentService[EventHandlerService]
       at android.os.MessageQueue.nativePollOnce(MessageQueue.java)
       at android.os.MessageQueue.next(MessageQueue.java:325)
       at android.os.Looper.loop(Looper.java:142)
       at android.os.HandlerThread.run(HandlerThread.java:65)

I'm not entirely sure this is an error from the optimizely SDK, but I saw you have this file, so it might be related.

android-sdk/event-handler/src/main/java/com/optimizely/ab/android/event_handler/EventIntentService.java

Line 53 in 992acca

super("EventHandlerService");

[matt-harris-mc]

We are also seeing similar crashes with a slightly different stack trace.

Fatal Exception: java.lang.OutOfMemoryError: pthread_create (1040KB stack) failed: Out of memory
       at java.lang.Thread.nativeCreate(Thread.java)
       at java.lang.Thread.start(Thread.java:733)
       at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:970)
       at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1377)
       at android.os.AsyncTask.executeOnExecutor(AsyncTask.java:651)
       at com.optimizely.ab.android.shared.ScheduledJobService.onStartJob(ScheduledJobService.java:145)
       at android.app.job.JobService$1.onStartJob(JobService.java:71)
       at android.app.job.JobServiceEngine$JobHandler.handleMessage(JobServiceEngine.java:108)
       at android.os.Handler.dispatchMessage(Handler.java:106)
       at android.os.Looper.loop(Looper.java:164)
       at android.app.ActivityThread.main(ActivityThread.java:6494)
       at java.lang.reflect.Method.invoke(Method.java)
       at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:438)
       at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:807)

[thomaszurkan-optimizely]

Thanks for the info. We hope to have a fix out this week.

[thomaszurkan-optimizely]

@jia-tse-zocdoc when you say Oreo 8, you mean Oreo 8 and above? correct?

[jia-tse-zocdoc]

@thomaszurkan-optimizely -- the particular out of memory error I was seeing is only on 8. All 120 users are on Android 8.

However, there is another issue affecting 8 & above: #269. I'm not quite sure if it's the same as the issues reported 2 years ago

[thomaszurkan-optimizely]

I have done a couple of things to make sure we don't start multiple event handlers for the same event. Do you know if you are generating a lot of events? Thousands? This should be fixed in 3.0.1. Patch 2.1.5 coming soon.

[jia-tse-zocdoc]

we have about 15-20 experiments, 15-20 metrics/events all sprinkled throughout the app. I can imagine 100s, but not 1000s. Are you able to look into your server backend and confirm that you're receiving thousands of events from us per user?

[thomaszurkan-optimizely]

@jia-tse-zocdoc I will look into it. But, that was the feedback I needed. This should be fixed in 3.0.1. We are going to patch 2.1.4 as well. Thank you for the help!

[iain-merrick-fanduel]

We're hitting the same issue. We're seeing both ANRs and OOM crashes, where the logs show hundreds of threads called IntentService[EventIntentService]. I'm trying 2.1.4 right now but it looks like the issue is not fixed.

The JobWorkService calls EventIntentService.onCreate but never seems to call onDestroy. Could this be where threads are being leaked?

android-sdk/shared/src/main/java/com/optimizely/ab/android/shared/JobWorkService.java

Line 106 in 2575789

intentService.onCreate();

Calling onCreate directly seems like a bad idea, and there is some other reflection-based code in JobWorkService that looks risky.

[iain-merrick-fanduel]

Seeing this mostly on Android 8 and 9.

[thomaszurkan-optimizely]

@iain-merrick-fanduel thanks for the feedback. Can you use 3.0.1? We didn't patch 2.1.4.

[jia-tse-zocdoc]

I have stopped seeing these errors in the 3.x versions...., 🤞

[thomaszurkan-optimizely]

I believe that this issue is fixed with the latest sdk being 3.2.1