[CI] TransportServiceHandshakeTests testAcceptsMismatchedBuildHashFromDifferentVersion failing

[tvernum]

Build scan:
https://gradle-enterprise.elastic.co/s/bo7lc6muq4ywc/tests/:server:test/org.elasticsearch.transport.TransportServiceHandshakeTests/testAcceptsMismatchedBuildHashFromDifferentVersion

Reproduction line:
./gradlew ':server:test' --tests "org.elasticsearch.transport.TransportServiceHandshakeTests.testAcceptsMismatchedBuildHashFromDifferentVersion" -Dtests.seed=DAC8299A5DC8F541 -Dtests.locale=fr-FR -Dtests.timezone=America/Antigua -Druntime.java=17

Applicable branches:
8.2

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.transport.TransportServiceHandshakeTests&tests.test=testAcceptsMismatchedBuildHashFromDifferentVersion

Failure excerpt:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2615, name=Thread-176, state=RUNNABLE, group=TGRP-TransportServiceHandshakeTests]

  at __randomizedtesting.SeedInfo.seed([DAC8299A5DC8F541:46E8E2D8AA38D7D9]:0)

  Caused by: java.lang.AssertionError: cluster:monitor/state

    at __randomizedtesting.SeedInfo.seed([DAC8299A5DC8F541]:0)
    at org.elasticsearch.transport.InboundAggregator.lambda$new$0(InboundAggregator.java:46)
    at org.elasticsearch.transport.InboundAggregator.initializeRequestState(InboundAggregator.java:197)
    at org.elasticsearch.transport.InboundAggregator.headerReceived(InboundAggregator.java:66)
    at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:138)
    at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
    at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
    at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

This looks to be a test isolation problem: this test sets up a fake transport service listening on 127.0.0.1:9300 with no registered actions, and then something else on the same machine (not this test) connects and asks for its cluster state.

@elastic/es-delivery is there any way to execute these tests in their own network namespace (e.g. a container) so that we can avoid this kind of interference?

[mark-vieira]

https://github.com/orgs/elastic/teams/es-delivery is there any way to execute these tests in their own network namespace (e.g. a container) so that we can avoid this kind of interference?

I discussed this with @henningandersen and the answer is basically "no", not really. We can't run individual test workers in isolation, even if we could this would introduce a huge amount of complexity in CI and be basically a non-starter for local testing.

We address this problem elsewhere by being smarter about the port ranges we use for test nodes. For example in ESTestCase we ensure non-overlapping port ranges by using calculating an offset based on the Gradle test worker id. For external test clusters we just use port 0 and have ES dynamically allocate a port itself which is reported back through the node ports file. I think the solution here is to make MockNode smarter and not assume nothing else is going to be concurrently running on that port. Either by a) doing something similar to ESTestCase and using a non-overlappig port range or b) dynamically allocating a port.

[DaveCTurner]

This looks to be caused by different test workers using the same port ranges. #85777 will mostly mitigate this, and #85786 tracks work towards a more robust fix. I'm closing this as the mitigation is now in place, and there's nothing particularly special about the tests involved in this failure which needs further attention.