CPU Hovers at 20-30% after OpenCensus has been added to a GCP AppEngine Flex service

[gkohen]

Please answer these questions before submitting a bug report.

What version of OpenCensus are you using?

0.12.2

What JVM are you using (java -version)?

8

What did you do?

If possible, provide a recipe for reproducing the error.
Include OpenCensus library in our build.gradle:

    compile "io.opencensus:opencensus-api:0.12.2"
    compile "io.opencensus:opencensus-exporter-trace-stackdriver:0.12.2"
    runtime "io.opencensus:opencensus-impl:0.12.2"

In Spring Boot using GCP AppEngine Flexible, we've defined a class to initialize the stackdriver exporter.

package com.jda.starter.utils.tracing;

import com.jda.starter.Application;

import io.opencensus.exporter.trace.stackdriver.StackdriverTraceConfiguration;
import io.opencensus.exporter.trace.stackdriver.StackdriverTraceExporter;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.annotation.Profile;
import org.springframework.context.event.ContextRefreshedEvent;
import org.springframework.context.event.EventListener;
import org.springframework.stereotype.Component;

import java.io.IOException;

@Component
@Profile(Application.Profiles.PRODUCTION)
public class OpenCensusExporterInitializer {

  private static final Logger LOGGER = LoggerFactory.getLogger(OpenCensusExporterInitializer.class);

  @EventListener(ContextRefreshedEvent.class)
  public void contextRefreshedEvent(ContextRefreshedEvent event) {
    initialize();
  }

  private void initialize() {
    try {
      StackdriverTraceExporter.createAndRegister(StackdriverTraceConfiguration.builder().build());
    } catch (IOException exception) {
      LOGGER.warn("Could not initialize OpenCensus export to Stackdriver");
    }
  }
}

What did you expect to see?

CPU to be close to 0 utilization when no traffic takes place.

What did you see instead?

CPU hovering between 20-30% once OpenCesus has been added to a service

[DazWilkin]

cc @mtwo @shahprit

[dinooliva]

Yang, please take a look into this.

[dinooliva]

We believe the issue may be with the disruptor framework - can you retry using opencensus-impl-lite and let us know whether that solves the problem?

[songy23]

As @dinooliva said, could you please change
runtime "io.opencensus:opencensus-impl:0.12.2"
to
runtime "io.opencensus:opencensus-impl-lite:0.12.2"
and see how the CPU usage goes?

[gkohen]

Great news. After updating to the *-lite version the issue has been resolved.

[songy23]

Great!

[sebright]

@dinooliva @songy23 Is this a known issue with Disruptor?

[songy23]

I think so, @bogdandrutu said he saw similar issue before on GAE. Disruptor wakes up every 100ns or so and could introduce some overhead.

[sebright]

Do you know if this has been reported in the Disruptor issue tracker?

[songy23]

I'm not sure but I haven't seen any related issues yet (https://github.com/LMAX-Exchange/disruptor/issues).

[sebright]

Reopening, since there is still an issue with opencensus-impl.

[bogdandrutu]

It is a combination of Disruptor + AppEngine that causes this problem.

[bogdandrutu]

I see that we do something special on AppEngine iff the java version is 1.7
https://github.com/census-instrumentation/opencensus-java/blob/ea590b847c87655d4aa583059daf707fc84e01b8/impl_core/src/main/java/io/opencensus/implcore/internal/DaemonThreadFactory.java

[askingcat]

I saw the increase for a project i'm currently working on as well. This project runs on GKE and uses Java 8, so it seems not only Java 7 is affected.

After switching to opencensus-impl-lite as suggested above the CPU load was reduced to a neglible amount on an idle system.

[gkohen]

OpenCensus folks, do you think the suggested configuration to move to 'lite' is a workaround or the solution to the problem?
Thanks.

[bogdandrutu]

@gkohen I've just looked more into the issue and found that the problem is with the sleeping strategy that we use for the Disruptor thread:

• Currently we use: https://github.com/LMAX-Exchange/disruptor/blob/master/src/main/java/com/lmax/disruptor/SleepingWaitStrategy.java with the default parameters. The comment says that this is a compromise for performance vs CPU usage, but with the default parameters is more focused on performance.
• Alternative for low qps and CPU sensitive applications we should use: https://github.com/LMAX-Exchange/disruptor/blob/master/src/main/java/com/lmax/disruptor/BlockingWaitStrategy.java

I think what we need is to allow users to change the sleeping strategy as well as allow them to configure arguments for the SleepingWaitStrategy

[solatis]

I just ran into this issue as well, but not using AppEngine. Using VisualVM I saw that the crulpit was the OpenCensus.Disruptor-0 thread, so that confirms the investigation into Disruptor in this issue report.

Switching to -lite addresses the issue. Using 0.14 with the stackdriver exporter.

[karlthepagan]

I have implemented opencensus-impl-lite (compared to opencensus-impl 0.13.2) in order to work around this on a CPU-sensitive application.

Because of the slight response time penalty I am in favor of adding the ability for an application to set a configurer. I appreciate the complexity in adding runtime configuration. I would be happy with a SPI implementation and will pursue a pull request in that style.

EDIT: To clarify, I found CPU use impacted my cloud-hosted application despite the upgraded LMAX package (release 0.12.2).

[langecode]

I have observed the same behaviour, although not profiled it yet. Having instrumented a simple service the idle CPU usage rises from around 3 millicores to 300 millicores - with opencensus-impl 0.15.0. The undelying OS being RHEL 7 running Kubernetes and Docker.

[steveniemitz]

to chime in, I've noticed this as well running in GKE, about 25% of our CPU time is spent in SleepingWaitStrategy.

[danielnorberg]

When running our application using opencensus-impl 0.14.0 on Java 11 we've noticed that the OpenCensus.Disruptor-0 thread is spinning and takes a whole cpu.

[bogdandrutu]

@danielnorberg see #1599 for more discussions. @songy23 is making a snapshot release for you to try it out and let us know if the fix helped.

[bogdandrutu]

@steveniemitz see discussion in #1599 as well.