Resource leak in netty tests

[carl-mastrangelo]

I haven't tracked down the responsible, but I ran with leak detection and found this:

Aug 16, 2017 10:58:18 AM io.netty.util.ResourceLeakDetector reportTracedLeak
SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
WARNING: 3 leak records were discarded because the leak record count is limited to 4. Use system property io.netty.leakDetection.maxRecords to increase the limit.
Recent access records: 4
#4:
	io.netty.buffer.AdvancedLeakAwareByteBuf.release(AdvancedLeakAwareByteBuf.java:936)
	io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:84)
	io.netty.util.ReferenceCountUtil.safeRelease(ReferenceCountUtil.java:109)
	io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:256)
	io.netty.channel.embedded.EmbeddedChannel.doWrite(EmbeddedChannel.java:723)
	io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:872)
	io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:839)
	io.netty.channel.embedded.EmbeddedChannel$EmbeddedUnsafe$1.flush(EmbeddedChannel.java:819)
	io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1321)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
	io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
	io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:191)
	io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:509)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:373)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1364)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380)
	io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:941)
	io.netty.channel.embedded.EmbeddedChannel.flushInbound(EmbeddedChannel.java:359)
	io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:320)
	io.grpc.netty.NettyHandlerTestBase.channelRead(NettyHandlerTestBase.java:140)
	io.grpc.netty.NettyServerHandlerTest.manualSetUp(NettyServerHandlerTest.java:163)
	io.grpc.netty.NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail(NettyServerHandlerTest.java:356)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:49)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:73)
	org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:273)
	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
	org.junit.runners.ParentRunner$3.run(ParentRunner.java:240)
	org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:65)
	org.junit.runners.ParentRunner.runChildren(ParentRunner.java:238)
	org.junit.runners.ParentRunner.access$000(ParentRunner.java:55)
	org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:231)
	org.junit.runners.ParentRunner.run(ParentRunner.java:316)
#3:
	io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:924)
	io.netty.buffer.AdvancedLeakAwareByteBuf.retain(AdvancedLeakAwareByteBuf.java:35)
	io.netty.util.ReferenceCountUtil.retain(ReferenceCountUtil.java:36)
	io.netty.channel.embedded.EmbeddedChannel.doWrite(EmbeddedChannel.java:721)
	io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:872)
	io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:839)
	io.netty.channel.embedded.EmbeddedChannel$EmbeddedUnsafe$1.flush(EmbeddedChannel.java:819)
	io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1321)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:776)
	io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:768)
	io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:749)
	io.netty.handler.codec.http2.Http2ConnectionHandler.flush(Http2ConnectionHandler.java:191)
	io.netty.handler.codec.http2.Http2ConnectionHandler.channelReadComplete(Http2ConnectionHandler.java:509)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelReadComplete(AbstractChannelHandlerContext.java:373)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelReadComplete(DefaultChannelPipeline.java:1364)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:398)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelReadComplete(AbstractChannelHandlerContext.java:380)
	io.netty.channel.DefaultChannelPipeline.fireChannelReadComplete(DefaultChannelPipeline.java:941)
	io.netty.channel.embedded.EmbeddedChannel.flushInbound(EmbeddedChannel.java:359)
	io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:320)
	io.grpc.netty.NettyHandlerTestBase.channelRead(NettyHandlerTestBase.java:140)
	io.grpc.netty.NettyServerHandlerTest.manualSetUp(NettyServerHandlerTest.java:163)
	io.grpc.netty.NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail(NettyServerHandlerTest.java:356)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:49)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:73)
	org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:273)
#2:
	Hint: 'DefaultChannelPipeline$HeadContext#0' will handle the message from this point.
	io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:810)
	io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:723)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeSettingsAck(DefaultHttp2FrameWriter.java:260)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.CGLIB$writeSettingsAck$6(<generated>)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d$$FastClassByMockitoWithCGLIB$$af30f311.invoke(<generated>)
	org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216)
	org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10)
	org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22)
	org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27)
	org.mockito.internal.invocation.InvocationImpl.callRealMethod(InvocationImpl.java:108)
	org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36)
	org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:93)
	org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
	org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.writeSettingsAck(<generated>)
	io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeSettingsAck(DecoratingHttp2FrameWriter.java:75)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeSettingsAck(DefaultHttp2ConnectionEncoder.java:264)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:419)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:635)
	io.netty.handler.codec.http2.DefaultHttp2FrameReader.readSettingsFrame(DefaultHttp2FrameReader.java:542)
	io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:263)
	io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
	io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:373)
	io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:244)
	io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:433)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:492)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:431)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:317)
	io.grpc.netty.NettyHandlerTestBase.channelRead(NettyHandlerTestBase.java:140)
	io.grpc.netty.NettyServerHandlerTest.manualSetUp(NettyServerHandlerTest.java:163)
	io.grpc.netty.NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail(NettyServerHandlerTest.java:356)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:49)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:73)
	org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:273)
	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)

#1:
	io.netty.buffer.AdvancedLeakAwareByteBuf.writeInt(AdvancedLeakAwareByteBuf.java:558)
	io.netty.handler.codec.http2.Http2CodecUtil.writeFrameHeaderInternal(Http2CodecUtil.java:260)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeSettingsAck(DefaultHttp2FrameWriter.java:259)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.CGLIB$writeSettingsAck$6(<generated>)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d$$FastClassByMockitoWithCGLIB$$af30f311.invoke(<generated>)
	org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216)
	org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10)
	org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22)
	org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27)
	org.mockito.internal.invocation.InvocationImpl.callRealMethod(InvocationImpl.java:108)
	org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36)
	org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:93)
	org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
	org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.writeSettingsAck(<generated>)
	io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeSettingsAck(DecoratingHttp2FrameWriter.java:75)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeSettingsAck(DefaultHttp2ConnectionEncoder.java:264)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:419)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:635)
	io.netty.handler.codec.http2.DefaultHttp2FrameReader.readSettingsFrame(DefaultHttp2FrameReader.java:542)
	io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:263)
	io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
	io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:373)
	io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:244)
	io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:433)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:492)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:431)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:317)
	io.grpc.netty.NettyHandlerTestBase.channelRead(NettyHandlerTestBase.java:140)
	io.grpc.netty.NettyServerHandlerTest.manualSetUp(NettyServerHandlerTest.java:163)
	io.grpc.netty.NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail(NettyServerHandlerTest.java:356)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:49)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:73)
	org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:273)
	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)

Created at:
	io.netty.util.ResourceLeakDetector.track(ResourceLeakDetector.java:237)
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:172)
	io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:109)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writeSettingsAck(DefaultHttp2FrameWriter.java:258)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.CGLIB$writeSettingsAck$6(<generated>)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d$$FastClassByMockitoWithCGLIB$$af30f311.invoke(<generated>)
	org.mockito.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:216)
	org.mockito.internal.creation.AbstractMockitoMethodProxy.invokeSuper(AbstractMockitoMethodProxy.java:10)
	org.mockito.internal.invocation.realmethod.CGLIBProxyRealMethod.invoke(CGLIBProxyRealMethod.java:22)
	org.mockito.internal.invocation.realmethod.FilteredCGLIBProxyRealMethod.invoke(FilteredCGLIBProxyRealMethod.java:27)
	org.mockito.internal.invocation.InvocationImpl.callRealMethod(InvocationImpl.java:108)
	org.mockito.internal.stubbing.answers.CallsRealMethods.answer(CallsRealMethods.java:36)
	org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:93)
	org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:38)
	org.mockito.internal.creation.MethodInterceptorFilter.intercept(MethodInterceptorFilter.java:51)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter$$EnhancerByMockitoWithCGLIB$$a575c56d.writeSettingsAck(<generated>)
	io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writeSettingsAck(DecoratingHttp2FrameWriter.java:75)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writeSettingsAck(DefaultHttp2ConnectionEncoder.java:264)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:419)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onSettingsRead(DefaultHttp2ConnectionDecoder.java:635)
	io.netty.handler.codec.http2.DefaultHttp2FrameReader.readSettingsFrame(DefaultHttp2FrameReader.java:542)
	io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:263)
	io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:160)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:118)
	io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:373)
	io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:244)
	io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:433)
	io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:492)
	io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:431)
	io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)
	io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
	io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)
	io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)
	io.netty.channel.embedded.EmbeddedChannel.writeInbound(EmbeddedChannel.java:317)
	io.grpc.netty.NettyHandlerTestBase.channelRead(NettyHandlerTestBase.java:140)
	io.grpc.netty.NettyServerHandlerTest.manualSetUp(NettyServerHandlerTest.java:163)
	io.grpc.netty.NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail(NettyServerHandlerTest.java:356)
	sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	java.lang.reflect.Method.invoke(Method.java:498)
	org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
	org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:49)
	org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:73)
	org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:273)
	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
	org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)

[alexvas]

BTW, I see slightly different stacktrace at our production grpc-based backend:

[07:12:15,418] [grpc-default-worker-ELG-3-11] ERROR io.netty.util.ResourceLeakDetector ~ LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://n
etty.io/wiki/reference-counted-objects.html for more information.
WARNING: 107 leak records were discarded because the leak record count is limited to 4. Use system property io.netty.leakDetection.maxRecords to increase the limit.
Recent access records: 4
#4:
        io.netty.buffer.AdvancedLeakAwareByteBuf.readBytes(AdvancedLeakAwareByteBuf.java:498)
        io.grpc.netty.NettyReadableBuffer.readBytes(NettyReadableBuffer.java:60)
        io.grpc.internal.CompositeReadableBuffer$3.readInternal(CompositeReadableBuffer.java:95)
        io.grpc.internal.CompositeReadableBuffer$ReadOperation.read(CompositeReadableBuffer.java:216)
        io.grpc.internal.CompositeReadableBuffer.execute(CompositeReadableBuffer.java:174)
        io.grpc.internal.CompositeReadableBuffer.readBytes(CompositeReadableBuffer.java:91)
        io.grpc.internal.ReadableBuffers$BufferInputStream.read(ReadableBuffers.java:327)
        io.grpc.protobuf.lite.ProtoLiteUtils$2.parse(ProtoLiteUtils.java:133)
        io.grpc.protobuf.lite.ProtoLiteUtils$2.parse(ProtoLiteUtils.java:79)
        io.grpc.MethodDescriptor.parseRequest(MethodDescriptor.java:280)
        io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messageRead(ServerCallImpl.java:246)
        io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1.runInContext(ServerImpl.java:571)
        io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:102)
        java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        java.lang.Thread.run(Thread.java:748)
#3:
        io.netty.buffer.AdvancedLeakAwareByteBuf.readUnsignedByte(AdvancedLeakAwareByteBuf.java:402)
        io.grpc.netty.NettyReadableBuffer.readUnsignedByte(NettyReadableBuffer.java:55)
        io.grpc.internal.CompositeReadableBuffer$1.readInternal(CompositeReadableBuffer.java:71)
        io.grpc.internal.CompositeReadableBuffer$ReadOperation.read(CompositeReadableBuffer.java:216)
        io.grpc.internal.CompositeReadableBuffer.execute(CompositeReadableBuffer.java:174)
        io.grpc.internal.CompositeReadableBuffer.readUnsignedByte(CompositeReadableBuffer.java:74)
        io.grpc.internal.AbstractReadableBuffer.readInt(AbstractReadableBuffer.java:48)
        io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:330)
        io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:232)
        io.grpc.internal.MessageDeframer.request(MessageDeframer.java:143)
        io.grpc.internal.AbstractStream$TransportState.requestMessagesFromDeframer(AbstractStream.java:195)
        io.grpc.netty.NettyServerStream$Sink$1.run(NettyServerStream.java:90)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        java.lang.Thread.run(Thread.java:748)
#2:
        io.netty.buffer.AdvancedLeakAwareByteBuf.readUnsignedByte(AdvancedLeakAwareByteBuf.java:402)
        io.grpc.netty.NettyReadableBuffer.readUnsignedByte(NettyReadableBuffer.java:55)
        io.grpc.internal.CompositeReadableBuffer$1.readInternal(CompositeReadableBuffer.java:71)
        io.grpc.internal.CompositeReadableBuffer$ReadOperation.read(CompositeReadableBuffer.java:216)
        io.grpc.internal.CompositeReadableBuffer.execute(CompositeReadableBuffer.java:174)
        io.grpc.internal.CompositeReadableBuffer.readUnsignedByte(CompositeReadableBuffer.java:74)
        io.grpc.internal.AbstractReadableBuffer.readInt(AbstractReadableBuffer.java:47)
        io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:330)
        io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:232)
        io.grpc.internal.MessageDeframer.request(MessageDeframer.java:143)
        io.grpc.internal.AbstractStream$TransportState.requestMessagesFromDeframer(AbstractStream.java:195)
        io.grpc.netty.NettyServerStream$Sink$1.run(NettyServerStream.java:90)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        java.lang.Thread.run(Thread.java:748)
#1:
        io.netty.buffer.AdvancedLeakAwareByteBuf.readUnsignedByte(AdvancedLeakAwareByteBuf.java:402)
        io.grpc.netty.NettyReadableBuffer.readUnsignedByte(NettyReadableBuffer.java:55)
        io.grpc.internal.CompositeReadableBuffer$1.readInternal(CompositeReadableBuffer.java:71)
        io.grpc.internal.CompositeReadableBuffer$ReadOperation.read(CompositeReadableBuffer.java:216)
        io.grpc.internal.CompositeReadableBuffer.execute(CompositeReadableBuffer.java:174)
        io.grpc.internal.CompositeReadableBuffer.readUnsignedByte(CompositeReadableBuffer.java:74)
        io.grpc.internal.AbstractReadableBuffer.readInt(AbstractReadableBuffer.java:46)
        io.grpc.internal.MessageDeframer.processHeader(MessageDeframer.java:330)
        io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:232)
        io.grpc.internal.MessageDeframer.request(MessageDeframer.java:143)
        io.grpc.internal.AbstractStream$TransportState.requestMessagesFromDeframer(AbstractStream.java:195)
        io.grpc.netty.NettyServerStream$Sink$1.run(NettyServerStream.java:90)
        io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        java.lang.Thread.run(Thread.java:748)
Created at:
        io.netty.util.ResourceLeakDetector.track(ResourceLeakDetector.java:237)
        io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:331)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:181)
        io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:172)
        io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:133)
        io.netty.channel.DefaultMaxMessagesRecvByteBufAllocator$MaxMessageHandle.allocate(DefaultMaxMessagesRecvByteBufAllocator.java:80)
        io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:122)
        io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644)
        io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
        io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
        io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
        io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        java.lang.Thread.run(Thread.java:748)

[carl-mastrangelo]

@alexvas What you are seeing is a different bug. Would you mind filing a separate issue? This one is specifically for tests. (and thus not as urgent as yours)

@ericgribkoff Message deframing change possibly?

[alexvas]

@carl-mastrangelo done in #3422

[vinodhabib]

@kannanjgithub As per the defect details I can run the whole test class as well as single unit test (NettyServerHandlerTest.headersWithInvalidContentTypeShouldFail()) which is mentioned in error logs of the defect successfully.

PFA the Successful Test class which ran in my local machine.

Even I tried some of the approaches/details which is mentioned in the below documentation for finding the Resource leaks in unit tests but unable to see any.
https://netty.io/wiki/reference-counted-objects.html

Also has you mentioned earlier some classes are mocked which are going to used in the setup() method the get required streamListner/streamTraces object instance from Factory which is required by all the tests otherwise we will be getting NPE saying streamTracerFactory is null.

Need to have a connect and have more discussion on the same with you to make sure I am not missing anything to see this issue

[vinodhabib]

As per the suggestion from @kannanjgithub working in the direction to enable to info/debug logs through logback.xml for the UT case to re-produce issue and the same in progress.

[vinodhabib]

@kannanjgithub As per your suggestion I am able to configure logback.xml and enable the loggers for UT cases but still it’s not showing required stack trace with any resource leakage issues/errors as mentioned in the defect.
PFA the log info which is set for DEBUG level. Also tried with TRACE level but not much info printed in the logs

We need to have connect and discuss more on this to make sure we are not missing anything to Reproduce this issue. Please suggest.

[kannanjgithub]

Can we try using the older Netty dependencies from before the time this issue got reported and see if we are able to repro it? We can try with netty 4.0.50 (release history for netty) netty-tcnative 2.0.5 (release history for netty-tcnative) and run the test. You need to modify the netty and nettytcnative versions in gradle/libs.versions.toml.

[vinodhabib]

@kannanjgithub tried with above mentioned old versions for netty and netty-tcnnative along multiple combination versions but most of them are giving issues with project build phase as mentioned in below snippet also followed some of the working combinations for above libraries mentioned in the "https://github.com/vinodhabib/grpc-java/blob/master/SECURITY.md" but here also initial combinations are failing at build level. not sure the old libraries available in the maven antifactory as i am not able to access the same to check.

Please suggest further? are you able to build the grpc-java project with Old versions of netty as mentioned above @kannanjgithub?

[ejona86]

For this we need to enable the leak detector at PARANOID level. https://netty.io/wiki/reference-counted-objects.html#leak-detection-levels

[vinodhabib]

@ejona86 Thanks for your reply as per your above suggestion I tried with using the JVM option ( -Dio.netty.leakDetection.level=paranoid') for NettyServerHandlerTest and able to see the below error with leakage issue.

14:03:18.643 [Time-limited test] ERROR io.netty.util.ResourceLeakDetector - LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
Recent access records: 
Created at:
	io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:402)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:188)
	io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:179)
	io.netty.buffer.AbstractByteBufAllocator.buffer(AbstractByteBufAllocator.java:116)
	io.netty.handler.codec.http2.DefaultHttp2FrameWriter.writePing(DefaultHttp2FrameWriter.java:339)
	java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	java.base/java.lang.reflect.Method.invoke(Method.java:578)
	org.mockito.internal.util.reflection.ReflectionMemberAccessor.invoke(ReflectionMemberAccessor.java:48)
	org.mockito.internal.stubbing.defaultanswers.ForwardsInvocations.answer(ForwardsInvocations.java:49)
	org.mockito.internal.handler.MockHandlerImpl.handle(MockHandlerImpl.java:110)
	org.mockito.internal.handler.NullResultGuardian.handle(NullResultGuardian.java:29)
	org.mockito.internal.handler.InvocationNotifierHandler.handle(InvocationNotifierHandler.java:34)
	org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:82)
	org.mockito.internal.creation.bytebuddy.MockMethodInterceptor.doIntercept(MockMethodInterceptor.java:56)
	org.mockito.internal.creation.bytebuddy.MockMethodInterceptor$DispatcherDefaultingToRealMethod.interceptAbstract(MockMethodInterceptor.java:161)
	io.netty.handler.codec.http2.Http2FrameWriter$MockitoMock$2041439246.writePing(Unknown Source)
	io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writePing(DecoratingHttp2FrameWriter.java:80)
	io.netty.handler.codec.http2.DefaultHttp2ConnectionEncoder.writePing(DefaultHttp2ConnectionEncoder.java:337)
	io.netty.handler.codec.http2.DecoratingHttp2FrameWriter.writePing(DecoratingHttp2FrameWriter.java:80)
	io.grpc.netty.Http2ControlFrameLimitEncoder.writePing(Http2ControlFrameLimitEncoder.java:80)
	io.grpc.netty.NettyServerHandler$GracefulShutdown.start(NettyServerHandler.java:1074)
	io.grpc.netty.NettyServerHandler$3.run(NettyServerHandler.java:393)
	io.grpc.internal.MaxConnectionIdleManager$2.run(MaxConnectionIdleManager.java:79)
	io.grpc.internal.LogExceptionRunnable.run(LogExceptionRunnable.java:42)
	io.grpc.netty.NettyHandlerTestBase$FakeClockScheduledNettyFuture$1.run(NettyHandlerTestBase.java:177)
	io.grpc.internal.FakeClock$ScheduledTask.run(FakeClock.java:98)
	io.grpc.internal.FakeClock.runDueTasks(FakeClock.java:310)
	io.grpc.internal.FakeClock.forwardTime(FakeClock.java:369)
	io.grpc.internal.FakeClock.forwardNanos(FakeClock.java:378)
	io.grpc.netty.NettyServerHandlerTest.maxConnectionIdle_goAwaySent_pingAck(NettyServerHandlerTest.java:949)
	java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	java.base/java.lang.reflect.Method.invoke(Method.java:578)

But the problem is I am unable to reproduce the same error with individual UT case running (i,e UT case where I can see the same issue while running the complete test class which I can see in the trace logs) as that will be helpful for me to Debug the issue. Any Suggestions here?

@ejona86 @kannanjgithub Expected fix for this issue is we should not see this above Leakage error while running the NettyServerHandlerTest class, right? Please confirm.

Further Analysis/Debug in progress for the same.

[ejona86]

These sorts of failures won't tend to show up if you run a single test by itself, because it requires the GC running to notice the object was leaked. You might be able to add an @After with GcFinalization.awaitFullGc() to trigger it for a single test.

[vinodhabib]

Thanks for the suggestions @ejona86. I will look into this.