[CI] FullClusterRestartIT.testRollupIDSchemeAfterRestart failed

[spinscale]

This test failed in CI, but I was unable to reproduce it on osx or linux.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=amazon/1231/console

Also note that there are still running tasks according to the build output.

  1> [2018-08-10T18:48:19,692][INFO ][o.e.x.r.FullClusterRestartIT] [testRollupIDSchemeAfterRestart] There are still tasks running after this test that might break subsequent tests [indices:data/write/bulk[s], indices:data/write/bulk[s][p], indices:data/write/bulk[s][r], indices:data/write/index, xpack/rollup/job[c]].
  1> [2018-08-10T18:48:19,729][INFO ][o.e.x.r.FullClusterRestartIT] [testRollupIDSchemeAfterRestart] after test
FAILURE 12.2s | FullClusterRestartIT.testRollupIDSchemeAfterRestart <<< FAILURES!
   > Throwable #1: java.lang.AssertionError:
   > Expected: iterable over ["3310683722", "rollup-id-test$ehY4NAyVSy8xxUDZrNXXIA"] in any order
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+6.x+multijob-unix-compatibility/os/amazon/x-pack/qa/full-cluster-restart/without-system-key/build/testrun/v6.3.3-SNAPSHOT#upgradedClusterTestRunner/J0/temp/org.elasticsearch.xpack.restart.FullClusterRestartIT_A9D42E18F468D297-001
   >      but: Not matched: "621059582"
   >    at __randomizedtesting.SeedInfo.seed([A9D42E18F468D297:F1BA0C257BB524E0]:0)
   >    at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >    at org.elasticsearch.xpack.restart.FullClusterRestartIT.lambda$testRollupIDSchemeAfterRestart$5(FullClusterRestartIT.java:424)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:842)
   >    at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:816)
   >    at org.elasticsearch.xpack.restart.FullClusterRestartIT.testRollupIDSchemeAfterRestart(FullClusterRestartIT.java:412)
   >    at java.lang.Thread.run(Thread.java:748)
   >    Suppressed: java.lang.AssertionError:
   > Expected: <2>
   >      but: was <1>
   >            at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
  2> NOTE: test params are: codec=Asserting(Lucene70): {}, docValues:{}, maxPointsInLeafNode=1961, maxMBSortInHeap=7.446664250363229, sim=RandomSimilarity(queryNorm=false): {}, locale=tr-TR, timezone=Pacific/Chuuk
  2> NOTE: Linux 4.9.62-21.56.amzn1.x86_64 amd64/Oracle Corporation 1.8.0_181 (64-bit)/cpus=16,threads=1,free=418041304,total=514850816
   >            at org.elasticsearch.xpack.restart.FullClusterRestartIT.lambda$testRollupIDSchemeAfterRestart$5(FullClusterRestartIT.java:418)
   >            at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:830)
   >            ... 39 more
   >    Suppressed: java.lang.AssertionError:
   > Expected: <2>
   >      but: was <1>
  2> NOTE: All tests run in this JVM: [FullClusterRestartIT]
   >            at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >            at org.elasticsearch.xpack.restart.FullClusterRestartIT.lambda$testRollupIDSchemeAfterRestart$5(FullClusterRestartIT.java:418)
   >            at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:830)
   >            ... 39 more
   >    Suppressed: java.lang.AssertionError:
   > Expected: <2>
   >      but: was <1>
   >            at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   >            at org.elasticsearch.xpack.restart.FullClusterRestartIT.lambda$testRollupIDSchemeAfterRestart$5(FullClusterRestartIT.java:418)
   >            at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:830)
   >            ... 39 more
   >    Suppressed: java.lang.AssertionError:
   > Expected: <2>

[elasticmachine]

Pinging @elastic/es-search-aggs

[colings86]

@polyfractal could you take a look at this one?

[polyfractal]

Hm, I'm not positive what's going on here but I have a guess. Will open a PR with a test fix and get Jim's opinion.

[andyb-elastic]

I'm not sure if this is strictly related but this failure on 6.x is from one of the lines added in b32fbbe

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+matrix-java-periodic/ES_BUILD_JAVA=java10,ES_RUNTIME_JAVA=java10,nodes=virtual&&linux/236/console

Full log is too large to upload sorry. Doesn't reproduce for me

  1> [2018-08-14T13:17:05,638][INFO ][o.e.x.r.FullClusterRestartIT] [testRollupIDSchemeAfterRestart] after test                                               
ERROR   10.5s | FullClusterRestartIT.testRollupIDSchemeAfterRestart <<< FAILURES!                                                                             
   > Throwable #1: org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:45233], URI [_xpack/rollup/job/rollup-id-test/_stop], status line [HTTP/1.1 404 Not Found]          
   > {"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"Task for Rollup Job [rollup-id-test] not found"}],"type":"resource_not_found_exception","reason":"Task for Rollup Job [rollup-id-test] not found"},"status":404}                                                                              
   >    at __randomizedtesting.SeedInfo.seed([452BE0CB85C8B7F5:1D45C2F60A154182]:0)                                                                           
   >    at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:920)                                                                     >    at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)                                                                            
   >    at org.elasticsearch.xpack.restart.FullClusterRestartIT.testRollupIDSchemeAfterRestart(FullClusterRestartIT.java:416)                                 
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)                                                                     
   >    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)                                                   
   >    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                                           
   >    at java.base/java.lang.reflect.Method.invoke(Method.java:564)          
   >    at java.base/java.lang.Thread.run(Thread.java:844)                     
   > Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://[::1]:45233], URI [_xpack/rollup/job/rollup-id-test/_stop], status line [HTTP/1.1 404 Not Found]             
   > {"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"Task for Rollup Job [rollup-id-test] not found"}],"type":"resource_not_found_exception","reason":"Task for Rollup Job [rollup-id-test] not found"},"status":404}                                                                              
   >    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:540)                                                                               
   >    at org.elasticsearch.client.RestClient$1.completed(RestClient.java:529)                                                                               
   >    at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)                                                                             
   >    at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)                      
   >    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)                                           
   >    at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)                                                
   >    at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)                                          
   >    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)                                                        
   >    at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)                                                        
   >    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)                                                        
   >    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)                                                                    
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)                                                        
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)                                                       
   >    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)                                                             
   >    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)                                                                     
   >    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)                                    
   >    ... 1 more 

REPRODUCE WITH: ./gradlew :x-pack:qa:full-cluster-restart:without-system-key:v6.3.3-SNAPSHOT#upgradedClusterTestRunner -Dtests.seed=452BE0CB85C8B7F5 -Dtests.class=org.elasticsearch.xpack.restart.FullClusterRestartIT -Dtests.method="testRollupIDSchemeAfterRestart" -Dtests.security.manager=true -Dtests.locale=id-ID -Dtests.timezone=Africa/Freetown -Dcompiler.java=10 -Druntime.java=10

[polyfractal]

Grr, I think it's trying to stop the job before the task is fully initialized. I should have left the awaitBusy for the persistent task.

I'll take care of this shortly... if it keeps failing before I get there feel free to mute. Sorry for the noise!