[CI] ReadinessClusterIT testReadinessDuringRestarts failing

[martijnvg]

Build scan:
https://gradle-enterprise.elastic.co/s/k26mmnltxc7yw/tests/:server:internalClusterTest/org.elasticsearch.readiness.ReadinessClusterIT/testReadinessDuringRestarts

Reproduction line:
./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.readiness.ReadinessClusterIT.testReadinessDuringRestarts" -Dtests.seed=66FCB40B017BCF62 -Dtests.locale=id-ID -Dtests.timezone=Asia/Barnaul -Druntime.java=17

Applicable branches:
master, 8.2

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.readiness.ReadinessClusterIT&tests.test=testReadinessDuringRestarts

Failure excerpt:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1731, name=Thread-10, state=RUNNABLE, group=TGRP-ReadinessClusterIT]

  at __randomizedtesting.SeedInfo.seed([66FCB40B017BCF62:7374F15AB6D0E16D]:0)

  Caused by: java.lang.AssertionError: java.lang.IllegalArgumentException: remote node [{node_t2}{BAmYUvGFRxqIhcJ3aQ82aw}{4NMnoo4vSgGEBKSvT_RfnQ}{127.0.0.1}{127.0.0.1:45408}{cdfhilrstw}] is build [1ed69c17444b90d524fe6a85e758f75e0fa148a6] of version [8.2.0] but this node is build [unknown] of version [8.2.0] which has an incompatible wire format

    at __randomizedtesting.SeedInfo.seed([66FCB40B017BCF62]:0)
    at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:346)
    at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:142)
    at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:94)
    at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:795)
    at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:149)
    at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
    at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
    at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run(Thread.java:833)

    Caused by: java.lang.IllegalArgumentException: remote node [{node_t2}{BAmYUvGFRxqIhcJ3aQ82aw}{4NMnoo4vSgGEBKSvT_RfnQ}{127.0.0.1}{127.0.0.1:45408}{cdfhilrstw}] is build [1ed69c17444b90d524fe6a85e758f75e0fa148a6] of version [8.2.0] but this node is build [unknown] of version [8.2.0] which has an incompatible wire format

      at org.elasticsearch.transport.TransportService$HandshakeResponse.throwOnIncompatibleBuild(TransportService.java:596)
      at org.elasticsearch.transport.TransportService$HandshakeResponse.maybeThrowOnIncompatibleBuild(TransportService.java:583)
      at org.elasticsearch.transport.TransportService$HandshakeResponse.<init>(TransportService.java:577)
      at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:58)
      at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:25)
      at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1331)
      at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1318)
      at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:338)
      at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:142)
      at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:94)
      at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:795)
      at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:149)
      at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
      at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
      at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
      at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[idegtiarenko]

Might have a same cause as #83034

[DaveCTurner]

This looks to be caused by different test workers using the same port ranges. #85777 will mostly mitigate this, and #85786 tracks work towards a more robust fix. I'm closing this as the mitigation is now in place, and there's nothing particularly special about the tests involved in this failure which needs further attention.