[CI] StableMasterDisruptionIT testFollowerCheckerDetectsUnresponsiveNodeAfterMasterReelection failing

[DaveCTurner]

Doesn't seem to reproduce, but seen a few times including on a local test run.

Build scan:
https://gradle-enterprise.elastic.co/s/rlfzaojrmr5fs/tests/:server:internalClusterTest/org.elasticsearch.discovery.StableMasterDisruptionIT/testFollowerCheckerDetectsUnresponsiveNodeAfterMasterReelection

Reproduction line:
./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.discovery.StableMasterDisruptionIT.testFollowerCheckerDetectsUnresponsiveNodeAfterMasterReelection" -Dtests.seed=D3646D45AD35342 -Dtests.locale=sq -Dtests.timezone=Africa/Blantyre -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
No Yes after a few iterations on an otherwise-stressed machine.

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.discovery.StableMasterDisruptionIT&tests.test=testFollowerCheckerDetectsUnresponsiveNodeAfterMasterReelection

Failure excerpt:

com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1859, name=Thread-57, state=RUNNABLE, group=TGRP-StableMasterDisruptionIT]

  at __randomizedtesting.SeedInfo.seed([D3646D45AD35342:8B4BBC8B276FEAF4]:0)

  Caused by: java.lang.AssertionError: java.lang.IllegalArgumentException: remote node [{node_t1}{TGSMWMQ-RRaRBJF0h3JuFQ}{O_WeXHzdTTO0QySJOxau3g}{127.0.0.1}{127.0.0.1:37609}{d}] is build [22b7ed72a1bdfc47c1f01f757d4682f34859074d] of version [8.1.0] but this node is build [unknown] of version [8.1.0] which has an incompatible wire format

    at __randomizedtesting.SeedInfo.seed([D3646D45AD35342]:0)
    at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:346)
    at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:142)
    at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:94)
    at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:796)
    at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:149)
    at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
    at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
    at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run(Thread.java:833)

    Caused by: java.lang.IllegalArgumentException: remote node [{node_t1}{TGSMWMQ-RRaRBJF0h3JuFQ}{O_WeXHzdTTO0QySJOxau3g}{127.0.0.1}{127.0.0.1:37609}{d}] is build [22b7ed72a1bdfc47c1f01f757d4682f34859074d] of version [8.1.0] but this node is build [unknown] of version [8.1.0] which has an incompatible wire format

      at org.elasticsearch.transport.TransportService$HandshakeResponse.<init>(TransportService.java:582)
      at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:58)
      at org.elasticsearch.action.ActionListenerResponseHandler.read(ActionListenerResponseHandler.java:25)
      at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1330)
      at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.read(TransportService.java:1317)
      at org.elasticsearch.transport.InboundHandler.handleResponse(InboundHandler.java:338)
      at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:142)
      at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:94)
      at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:796)
      at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:149)
      at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121)
      at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86)
      at org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:74)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
      at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
      at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
      at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
      at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:620)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:583)
      at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
      at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
      at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
      at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

Also seen more recently:

  2> REPRODUCE WITH: ./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.discovery.StableMasterDisruptionIT.testFollowerCheckerDetectsUnresponsiveNodeAfterMasterReelection" -Dtests.seed=418642FA1178E98F -Dtests.jvms=4 -Dtests.locale=nl-NL
-Dtests.timezone=America/St_Johns -Druntime.java=17
  2> java.lang.AssertionError: failed to reach a stable cluster of [2] nodes. Tried via [node_t1]. last cluster state:
    cluster uuid: ihSAlOBgQYKOjErBxv_SCA [committed: true]
    version: 8
    state uuid: mmUdSwSWQp2l2lyesDnW6w
    from_diff: false
    meta data version: 6
       coordination_metadata:
          term: 2
          last_committed_config: VotingConfiguration{x5NQzu43RReCdaw6VAaKVA,lC3ePNMVTKyCnqq6t2eeAA,mBCs6dBaT4SMGxET_ZBdJg}
          last_accepted_config: VotingConfiguration{x5NQzu43RReCdaw6VAaKVA,lC3ePNMVTKyCnqq6t2eeAA,mBCs6dBaT4SMGxET_ZBdJg}
          voting tombstones: []
    metadata customs:
       index-graveyard: IndexGraveyard[[]]
    nodes:
       {node_t2}{x5NQzu43RReCdaw6VAaKVA}{2qwh8MGcSTmXP7Bhoe_vHw}{127.0.0.1}{127.0.0.1:46375}{cdfhilmrstw}
       {node_t3}{lC3ePNMVTKyCnqq6t2eeAA}{LuliJEHiQCa3nhKLGDDz6Q}{127.0.0.1}{127.0.0.1:37811}{cdfhilmrstw}
       {node_t1}{mBCs6dBaT4SMGxET_ZBdJg}{YVLoZiJLQLeDwmdqYxqA2w}{127.0.0.1}{127.0.0.1:41395}{cdfhilmrstw}, local, master
    routing_table (version 1):
    routing_nodes:
    -----node_id[lC3ePNMVTKyCnqq6t2eeAA][V]
    -----node_id[x5NQzu43RReCdaw6VAaKVA][V]
    -----node_id[mBCs6dBaT4SMGxET_ZBdJg][V]
    ---- unassigned
        at __randomizedtesting.SeedInfo.seed([418642FA1178E98F:C7FBB8A56CC45039]:0)
        at org.junit.Assert.fail(Assert.java:88)
        at org.elasticsearch.test.ESIntegTestCase.ensureStableCluster(ESIntegTestCase.java:1271)
        at org.elasticsearch.test.ESIntegTestCase.ensureStableCluster(ESIntegTestCase.java:1252)
        at org.elasticsearch.discovery.StableMasterDisruptionIT.testFollowerCheckerAfterMasterReelection(StableMasterDisruptionIT.java:162)
        at org.elasticsearch.discovery.StableMasterDisruptionIT.testFollowerCheckerDetectsUnresponsiveNodeAfterMasterReelection(StableMasterDisruptionIT.java:124)