[ci] SimpleNetty4TransportTests.testChannelCloseWhileConnecting

[andyb-elastic]

Does not reproduce

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+release-tests/888/console
build-888.txt

REPRODUCE WITH: ./gradlew :modules:transport-netty4:test \
  -Dtests.seed=493C139EBD4D6E7E \
  -Dtests.class=org.elasticsearch.transport.netty4.SimpleNetty4TransportTests \
  -Dtests.method="testChannelCloseWhileConnecting" \
  -Dtests.security.manager=true \
  -Dbuild.snapshot=false \
  -Dtests.jvm.argline="-Dbuild.snapshot=false" \
  -Dtests.locale=es-PR \
  -Dtests.timezone=Europe/Madrid

  1> [2018-07-23T16:08:37,101][ERROR][i.n.u.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. See http://netty.io/wiki/reference-counted-objects.html for more information.
  1> WARNING: 3 leak records were discarded because the leak record count is limited to 4. Use system property io.netty.leakDetection.maxRecords to increase the limit.
  1> Recent access records: 4
  1> #4:
  1>    Hint: 'pipelining' will handle the message from this point.
  1>    io.netty.buffer.AdvancedLeakAwareCompositeByteBuf.touch(AdvancedLeakAwareCompositeByteBuf.java:36)
  1>    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpMessage.touch(HttpObjectAggregator.java:374)
  1>    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:454)
  1>    io.netty.handler.codec.http.HttpObjectAggregator$AggregatedFullHttpRequest.touch(HttpObjectAggregator.java:404)
  1>    io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:116)

[elasticmachine]

Pinging @elastic/es-core-infra

[original-brownbear]

This can be reproduced by running org.elasticsearch.http.netty4.Netty4HttpServerPipeliningTests in a loop.
This test just fails because the buffer lived on from the test before it and now is GCed a little later.

[andyb-elastic]

Different test but looks related - same resource leak stack trace from netty

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+default-distro/176/console
build-176.txt

[original-brownbear]

Fixed in #32296 in a pretty straightforward way. Hope this is the only source (second one looks like it could be from the same source, so I have some optimism :))

[nik9000]

@original-brownbear this failed again for me:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=ubuntu&&virtual/2623/console

I can get it to fail in a loop:

 while ./gradlew :modules:transport-netty4:test -Dtests.seed=D6D51485DA928AEE -Dtests.class=org.elasticsearch.transport.netty4.SimpleNetty4TransportTests -Dtests.method="testChannelCloseWhileConnecting" -Dtests.security.manager=true -Dtests.locale=fi -Dtests.timezone=Europe/Istanbul -Dcompiler.java=10 -Druntime.java=8; do echo ok; done

[original-brownbear]

@nik9000 right, seems this is a different failure though, not a buffer leak. I'll take a look at it regardless in a bit though :)