[CI] ShrinkIndexIT.testShrinkIndexPrimaryTerm fails

[cbuescher]

The test ShrinkIndexIT.testShrinkIndexPrimaryTerm() failed today on CI:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-darwin-compatibility/1526/console

The first this that looks like an error in the log is this:

13:29:05   1> [2018-01-09T23:27:23,148][WARN ][o.e.i.c.IndicesClusterStateService] [node_s1] [[target][0]] marking and sending shard failed due to [failed to create shard]
13:29:05   1> org.elasticsearch.index.IndexNotFoundException: no such index
13:29:05   1> 	at org.elasticsearch.index.shard.IndexShard.startRecovery(IndexShard.java:2075) ~[main/:?]
13:29:05   1> 	at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:516) ~[main/:?]
13:29:05   1> 	at org.elasticsearch.indices.IndicesService.createShard(IndicesService.java:143) ~[main/:?]
13:29:05   1> 	at org.elasticsearch.indices.cluster.IndicesClusterStateService.createShard(IndicesClusterStateService.java:552) ~[main/:?]
13:29:05   1> 	at org.elasticsearch.indices.cluster.IndicesClusterStateService.createOrUpdateShards(IndicesClusterStateService.java:529) ~[main/:?]
13:29:05   1> 	at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyClusterState(IndicesClusterStateService.java:231) ~[main/:?]
13:29:05   1> 	at org.elasticsearch.cluster.service.ClusterApplierService.lambda$callClusterStateAppliers$6(ClusterApplierService.java:498) ~[main/:?]
13:29:05   1> 	at java.lang.Iterable.forEach(Iterable.java:75) [?:1.8.0_141]
13:29:05   1> 	at org.elasticsearch.cluster.service.ClusterApplierService.callClusterStateAppliers(ClusterApplierService.java:495) [main/:?]
13:29:05   1> 	at org.elasticsearch.cluster.service.ClusterApplierService.applyChanges(ClusterApplierService.java:482) [main/:?]
13:29:05   1> 	at org.elasticsearch.cluster.service.ClusterApplierService.runTask(ClusterApplierService.java:432) [main/:?]
13:29:05   1> 	at org.elasticsearch.cluster.service.ClusterApplierService$UpdateTask.run(ClusterApplierService.java:161) [main/:?]
13:29:05   1> 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:566) [main/:?]
13:29:05   1> 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:244) [main/:?]
13:29:05   1> 	at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:207) [main/:?]
13:29:05   1> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
13:29:05   1> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
13:29:05   1> 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

The test seems to time out waiting for green state as a consequence:

13:29:07   1> [2018-01-09T23:29:04,372][INFO ][o.e.a.a.i.c.ShrinkIndexIT] [testShrinkIndexPrimaryTerm]: after test
13:29:07 FAILURE 90.4s J0 | ShrinkIndexIT.testShrinkIndexPrimaryTerm <<< FAILURES!
13:29:07    > Throwable #1: java.lang.AssertionError: timed out waiting for green state
13:29:07    > 	at __randomizedtesting.SeedInfo.seed([CD5C54E500E5EC92:E7D0617C1EBDF9DF]:0)
13:29:07    > 	at org.elasticsearch.test.ESIntegTestCase.ensureColor(ESIntegTestCase.java:947)
13:29:07    > 	at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:913)
13:29:07    > 	at org.elasticsearch.test.ESIntegTestCase.ensureGreen(ESIntegTestCase.java:902)
13:29:07    > 	at org.elasticsearch.action.admin.indices.create.ShrinkIndexIT.testShrinkIndexPrimaryTerm(ShrinkIndexIT.java:167)
13:29:07    > 	at java.lang.Thread.run(Thread.java:748)

./gradlew :core:integTest -Dtests.seed=CD5C54E500E5EC92 -Dtests.class=org.elasticsearch.action.admin.indices.create.ShrinkIndexIT -Dtests.method="testShrinkIndexPrimaryTerm" -Dtests.security.manager=true -Dtests.locale=ar-AE -Dtests.timezone=Asia/Ust-Nera

[dnhatn]

Log: testShrinkIndexPrimaryTerm.txt

[dnhatn]

Another instance on master https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=sles/1929/console

Log: testShrinkIndexPrimaryTerm.txt

[mayya-sharipova]

The same test failed on master Java 10 today.
Is not reproduced locally for me.

REPRODUCE WITH: ./gradlew :server:integTest \
  -Dtests.seed=2A7F48E21D6F8FBF \
  -Dtests.class=org.elasticsearch.action.admin.indices.create.ShrinkIndexIT \
  -Dtests.method="testShrinkIndexPrimaryTerm" \
  -Dtests.security.manager=true \
  -Dtests.locale=hy \
  -Dtests.timezone=America/Sitka

Console log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java10,nodes=virtual&&linux/158/console

Errors:

16:10:08 FAILURE 4.58s J1 | ShrinkIndexIT.testShrinkIndexPrimaryTerm <<< FAILURES!
16:10:08    > Throwable #1: java.lang.AssertionError: Accounting breaker not reset to 0 on node: node_s2
16:10:08    > Expected: <0L>
16:10:08    >      but: was <1322L>
16:10:08    > 	at __randomizedtesting.SeedInfo.seed([2A7F48E21D6F8FBF:F37D7B03379AF2]:0)
16:10:08    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
16:10:08    > 	at org.elasticsearch.test.InternalTestCluster.ensureEstimatedStats(InternalTestCluster.java:2020)
16:10:08    > 	at org.elasticsearch.test.TestCluster.assertAfterTest(TestCluster.java:93)
16:10:08    > 	at org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2051)
16:10:08    > 	at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:592)
16:10:08    > 	at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2185)
16:10:08    > 	at jdk.internal.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
16:10:08    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
16:10:08    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
16:10:08    > 	at java.base/java.lang.Thread.run(Thread.java:844)

[elasticmachine]

Pinging @elastic/es-distributed

[dnhatn]

16:10:08 > Throwable #1: java.lang.AssertionError: Accounting breaker not reset to 0 on node: node_s2
16:10:08 > Expected: <0L>
16:10:08 > but: was <1322L>

@mayya-sharipova I think this test failed because the accounting breaker was not reset correctly. This is a known issue and being tracked at #30290. I added this failure to #30290.