Spring Reactive App backed by netty4 requires more direct memory

[hmble2]

Hi,
I have a PCF reactive app built on spring-boot-start-webflux 2.0.2 which depends on netty version 4. Deploying this app with 1 GB container size (and using java-build-pack memory calculator) crashes it due to lack of direct memory:

   io.netty.util.internal.OutOfDirectMemoryError: failed to allocate 64 byte(s) of direct memory (used: 10485751, max: 10485760)
   	at io.netty.util.internal.PlatformDependent.incrementMemoryCounter(PlatformDependent.java:640)
   	at io.netty.util.internal.PlatformDependent.allocateDirectNoCleaner(PlatformDependent.java:594)
   	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.allocateDirect(UnpooledUnsafeNoCleanerDirectByteBuf.java:30)
   	at io.netty.buffer.UnpooledUnsafeDirectByteBuf.<init>(UnpooledUnsafeDirectByteBuf.java:68)
   	at io.netty.buffer.UnpooledUnsafeNoCleanerDirectByteBuf.<init>(UnpooledUnsafeNoCleanerDirectByteBuf.java:25)
   	at io.netty.buffer.UnsafeByteBufUtil.newUnsafeDirectByteBuf(UnsafeByteBufUtil.java:625)
   	at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:327)
   	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185)
   	at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176)
   	at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:137)
   	at io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:114)
   	at io.netty.channel.epoll.EpollRecvByteAllocatorHandle.allocate(EpollRecvByteAllocatorHandle.java:71)
   	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:793)
   	at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe$1.run(AbstractEpollChannel.java:387)
   	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
   	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
   	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309)
   	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
   	at java.lang.Thread.run(Thread.java:748)

I tried setting Direct memory using JAVA_OPTS: '-XX:+PrintFlagsFinal -XX:MaxDirectMemorySize=0m' to let JVM control allocation, however, that settings makes app fail to start.

Questions:

1. How much direct memory should I set for the reactive apps? any guideline or pointers will be helpful. App seems to survive 400 concurrent sessions with 200 MB direct memory with 200 ms latency.
2. Should I set Direct memory via JAVA_OPTS or JBP_CONFIG
3. is there a way to set max direct memory to 0 so that JVM controls allocation as per https://docs.oracle.com/javase/8/docs/technotes/tools/unix/java.html

Thanks

[nebhale]

1. There's no correct answer to this question which is why we don't attempt to make any calculation about it. It could be 64M or 64G depending on the application you've written.
2. As the documentation (and blog) state, tuning memory should be done via standard JVM flags. A cf set-env <APP> JAVA_OPTS '-XX:MaxDirectMemorySize=...' is the appropriate way to configure direct memory.
3. There is no way to set max direct memory to 0 because it's inherently unsafe to do within a memory bounded container. Setting the value to zero means that it is unbounded and can grow to whatever size is required. Given that the container you are running in does not have that flexibility (your process will be SIGKILL'd if you exceed the container memory limit) unbounding that memory space will inherently cause a container OOM. Rather than allow a behavior that will cause regular restarts of application instances, we've bounded the value and require applications that desire a different bounded size to set a bounded size explicitly.

[nebhale]

@smaldini and @rstoyanchev something seems amiss about this issue. We've established, through extensive testing on the CF Java Client, that when properly configured Netty will automatically use heap instead of direct memory when there is not enough direct memory available. This happens transparently to users; it takes the best available memory space it can and falls back silently when it cannot. Is it possible that there's something in WebFlux that is requiring direct memory (explicit use of a direct memory subclass of something)?

[hmble2]

Thanks for answering. I believe if we can make spring/netty reactive based PCF app use heap when direct memory overflows that would be great, Or, approximate direct memory consumption amount so as I can accordingly specify it based on anticipated traffic.

Regarding, -XX:MaxDirectMemorySize=0m, which is JVM default, I am under assumption that if the user doesn't specify the MaxDirectMemory or sets it to 0, then it does not mean it is unbounded/unlimited direct memory rather leaves it to JVM to determine memory limit/allocation (using sun's proprietary method sun.misc.VM.maxDirectMemory()). So for a PCF container size of 1 GB with max direct memory set to 0, when queried using sun.misc.VM.maxDirectMemory() should provide amount less than 1 GB (e.g. around 200 MB if I override calculated heap from 600 MB to 400 MB)? Unfortunately I cannot assert this in PCF because java-build-pack doesn't let app start when setting direct memory to 0, even when fixed in this cloudfoundry/java-buildpack#591 (FYI, I am using java-buildpack version 4.12). Please correct me if you feel otherwise and verify sun.misc.VM.maxDirectMemory() if possible. Also, should I raise -XX:MaxDirectMemorySize=0m in cloudfoundry/java-buildpack#591 Thanks Again

[nebhale]

/cc @glyn

[rstoyanchev]

The only thing I can think of is Netty ByteBuf's created through the ByteBufAllocator of the channel, i.e. obtained with NettyOutbound#alloc().

[glyn]

@hmble2

Regarding, -XX:MaxDirectMemorySize=0m, which is JVM default, I am under assumption that if the user doesn't specify the MaxDirectMemory or sets it to 0, then it does not mean it is unbounded/unlimited direct memory rather leaves it to JVM to determine memory limit/allocation (using sun's proprietary method sun.misc.VM.maxDirectMemory()).

Unfortunately, even when the JVM allocates a bounded amount of direct memory, the memory calculator still needs to know that value before the JVM has started in order to allocate the remaining memory to the heap.

So, for -XX:MaxDirectMemorySize=0m to work in practice, the memory calculator would have to second guess the JVM calculation of direct memory size. Since the JVM calculation is not documented, that would tie the memory calculator to the JVM implementation and inevitably the two would get out of sync.

[nebhale]

@rstoyanchev You never instantiate a ByteBufAllocator directly, only using the one from NettyOutbound?

[hmble2]

Thanks all for useful insights. I guess with the growing number of reactive apps that almost always leverage direct memory for efficiency purpose, we will run into a new problem: How much direct memory would a reactive app consume? This becomes an specific problem in container based applications where each app runs in very limited memory, and hence wise memory partitioning becomes necessity or the memory is wasted (in heap or direct bucket). Also, I believe CF Memory calculator needs to either provide some sort of approximation for direct memory or should allow a new option to specify direct memory as a percentage of container memory. In absence of such capability, PCF based apps would not scale well on memory. For example, if my app needs 200 MB direct memory in 1 GB container, then when CF auto-scaler scales it to 2 GB, I would expect direct memory also scaled up to be 400 MB (or else run into problem mentioned in first post). With the new option based approach (that specifies direct memory as percentage of container memory), memory calculator could calculate new value of direct memory based on sclaled-up container memory and feed to JVM using -XX:MaxDirectMemorySize. Thoughts?

[rstoyanchev]

@nebhale, correct, our use of ByteBufAllocator is hidden behind the DataBufferFactory abstraction, so it's constrained to very few places, and is always obtained from the NettyOutbound.

[vpavic]

Certain workloads involving Redis pub-sub and Lettuce (the default Redis driver provided by Spring Boot's Redis starters that is also based on Netty) make it very easy to hit the 10 MB direct memory set by the calculator. This is unrelated to application's runtime - it can happen both with Servlet and reactive runtimes.

At that point the options are to either make Netty use heap buffers instead (by setting -Dio.netty.noPreferDirect=true) or increasing direct memory. See redis/lettuce#705.

Not sure if anything reasonable can be done about this from the calculator perspective, but at least hopefully this will help someone running into the same issue.