[CI] org.elasticsearch.xpack.ccr.FollowerFailOverIT#testFailOverOnFollower

[matriv]

Failing on master

Build scan: https://gradle-enterprise.elastic.co/s/jm5pkdiktr2zq

Repro line:

./gradlew ':x-pack:plugin:ccr:internalClusterTest' --tests "org.elasticsearch.xpack.ccr.FollowerFailOverIT.testFailOverOnFollower" \
  -Dtests.seed=A1E8FAE62B8B6C9E \
  -Dtests.security.manager=true \
  -Dtests.locale=cs-CZ \
  -Dtests.timezone=America/Argentina/Rio_Gallegos \
  -Druntime.java=11

Reproduces locally?: No

Applicable branches: master

Failure history:
https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!f,value:7200000),time:(from:now-30d,mode:quick,to:now))&_a=(columns:!(_source),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'class:%20%22org.elasticsearch.xpack.ccr.FollowerFailOverIT%22%20test:%22%20testFailOverOnFollower%22%20stacktrace:%20%22timed%20out%20waiting%20for%20green%20state%22'),sort:!(time,desc))

Failure excerpt:

07:36:11 org.elasticsearch.xpack.ccr.FollowerFailOverIT > testFailOverOnFollower FAILED
07:36:11     java.lang.AssertionError: timed out waiting for green state
07:36:11         at __randomizedtesting.SeedInfo.seed([A1E8FAE62B8B6C9E:7EBB537ABFEBCFA4]:0)
07:36:11         at org.junit.Assert.fail(Assert.java:88)
07:36:11         at org.elasticsearch.xpack.CcrIntegTestCase.ensureColor(CcrIntegTestCase.java:337)
07:36:11         at org.elasticsearch.xpack.CcrIntegTestCase.ensureFollowerGreen(CcrIntegTestCase.java:311)
07:36:11         at org.elasticsearch.xpack.CcrIntegTestCase.ensureFollowerGreen(CcrIntegTestCase.java:306)
07:36:11         at org.elasticsearch.xpack.ccr.FollowerFailOverIT.testFailOverOnFollower(FollowerFailOverIT.java:101)
07:36:11 REPRODUCE WITH: ./gradlew ':x-pack:plugin:ccr:internalClusterTest' --tests "org.elasticsearch.xpack.ccr.FollowerFailOverIT.testFailOverOnFollower" -Dtests.seed=A1E8FAE62B8B6C9E -Dtests.security.manager=true -Dtests.locale=cs-CZ -Dtests.timezone=America/Argentina/Rio_Gallegos -Druntime.java=11
07:36:11 
07:36:11 org.elasticsearch.xpack.ccr.FollowerFailOverIT > classMethod FAILED
07:36:11     com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.elasticsearch.xpack.ccr.FollowerFailOverIT: 
07:36:11        1) Thread[id=712, name=Thread-23, state=TIMED_WAITING, group=TGRP-FollowerFailOverIT]
07:36:11             at java.base@11.0.6/jdk.internal.misc.Unsafe.park(Native Method)
07:36:11             at java.base@11.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
07:36:11             at java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
07:36:11             at java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
07:36:11             at java.base@11.0.6/java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:415)
07:36:11             at app//org.elasticsearch.xpack.ccr.FollowerFailOverIT.lambda$testFailOverOnFollower$0(FollowerFailOverIT.java:70)
07:36:11             at app//org.elasticsearch.xpack.ccr.FollowerFailOverIT$$Lambda$4075/0x0000000100d3dc40.run(Unknown Source)
07:36:11             at java.base@11.0.6/java.lang.Thread.run(Thread.java:834)
07:36:11        2) Thread[id=711, name=Thread-22, state=TIMED_WAITING, group=TGRP-FollowerFailOverIT]
07:36:11             at java.base@11.0.6/jdk.internal.misc.Unsafe.park(Native Method)
07:36:11             at java.base@11.0.6/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
07:36:11             at java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1079)
07:36:11             at java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1369)
07:36:11             at java.base@11.0.6/java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:415)
07:36:11             at app//org.elasticsearch.xpack.ccr.FollowerFailOverIT.lambda$testFailOverOnFollower$0(FollowerFailOverIT.java:70)
07:36:11             at app//org.elasticsearch.xpack.ccr.FollowerFailOverIT$$Lambda$4075/0x0000000100d3dc40.run(Unknown Source)
07:36:11             at java.base@11.0.6/java.lang.Thread.run(Thread.java:834)
07:36:11         at __randomizedtesting.SeedInfo.seed([A1E8FAE62B8B6C9E]:0)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/CCR)

[dimitris-athanasiou]

Another failure in: https://gradle-enterprise.elastic.co/s/t5ncm7jvhfo3k

[jrodewig]

Another failure: https://gradle-enterprise.elastic.co/s/c5mdveo3rdrfc