ILM: CI Flaky testFullPolicy/testMoveToRolloverStep Tests

[talevy]

The following tests are flaky and need further invesitgation

org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.testMoveToRolloverStep
org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.testFullPolicy

reproduce with:

./gradlew :x-pack:plugin:ilm:qa:multi-node:integTestRunner -Dtests.seed=90FF073C41149E5 -Dtests.class=org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT -Dtests.method="testMoveToRolloverStep" -Dtests.security.manager=true -Dtests.locale=en -Dtests.timezone=Greenwich -Dcompiler.java=11 -Druntime.java=8

stacktrace details

 2> REPRODUCE WITH: ./gradlew :x-pack:plugin:ilm:qa:multi-node:integTestRunner -Dtests.seed=90FF073C41149E5 -Dtests.class=org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT -Dtests.method="testMoveToRolloverStep" -Dtests.security.manager=true -Dtests.locale=en -Dtests.timezone=Greenwich -Dcompiler.java=11 -Druntime.java=8
  1> [2018-10-30T17:11:13,275][WARN ][o.e.b.JNANatives         ] [[SUITE-TimeSeriesLifecycleActionsIT-seed#[90FF073C41149E5]]] Increase RLIMIT_MEMLOCK, soft limit: 65536, hard limit: 65536
  1> [2018-10-30T17:11:13,275][WARN ][o.e.b.JNANatives         ] [[SUITE-TimeSeriesLifecycleActionsIT-seed#[90FF073C41149E5]]] These can be adjusted by modifying /etc/security/limits.conf, for example: 
  2> REPRODUCE WITH: ./gradlew :x-pack:plugin:ilm:qa:multi-node:integTestRunner -Dtests.seed=90FF073C41149E5 -Dtests.class=org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT -Dtests.method="testFullPolicy" -Dtests.security.manager=true -Dtests.locale=en -Dtests.timezone=Greenwich -Dcompiler.java=11 -Druntime.java=8
  1> 	# allow user 'jenkins' mlockall
  1> 	jenkins soft memlock unlimited
  1> 	jenkins hard memlock unlimited
  1> [2018-10-30T17:11:13,276][WARN ][o.e.b.JNANatives         ] [[SUITE-TimeSeriesLifecycleActionsIT-seed#[90FF073C41149E5]]] If you are logged in interactively, you will have to re-login for the new limits to take effect.
  1> [2018-10-30T17:11:13,940][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testForceMergeAction] before test
  1> [2018-10-30T17:11:14,173][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testForceMergeAction] initializing REST clients against [http://[::1]:35091, http://[::1]:38249, http://[::1]:33747, http://[::1]:36177]
  1> [2018-10-30T17:11:21,134][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testForceMergeAction] after test
  1> [2018-10-30T17:11:21,174][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testNonexistentPolicy] before test
  1> [2018-10-30T17:11:21,591][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testNonexistentPolicy] HTTP/1.1 201 Created
  2> NOTE: leaving temporary files on disk at: /var/lib/jenkins/workspace/elastic+elasticsearch+pull-request/x-pack/plugin/ilm/qa/multi-node/build/testrun/integTestRunner/J0/temp/org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT_90FF073C41149E5-001
  1> [2018-10-30T17:11:21,598][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testNonexistentPolicy] {indices={hxxyyedkhct-000001={index=hxxyyedkhct-000001, managed=true, policy=does_not_exist, lifecycle_date=null, phase=null, phase_time=null, action=null, action_time=null, step=null, step_time=null, step_info={type=illegal_argument_exception, reason=policy [does_not_exist] does not exist}}}}
  1> [2018-10-30T17:11:21,856][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testNonexistentPolicy] after test
  1> [2018-10-30T17:11:21,869][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testAllocateOnlyAllocation] before test
  1> [2018-10-30T17:11:23,624][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testAllocateOnlyAllocation] after test
  1> [2018-10-30T17:11:23,637][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testMoveToAllocateStep] before test
  2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=704, maxMBSortInHeap=6.768407903994094, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@6c06fd13), locale=en, timezone=Greenwich
  1> [2018-10-30T17:11:26,125][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testMoveToAllocateStep] after test
  1> [2018-10-30T17:11:26,139][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testAllocateActionOnlyReplicas] before test
  2> NOTE: Linux 4.15.0-1023-gcp amd64/Oracle Corporation 1.8.0_192 (64-bit)/cpus=16,threads=1,free=413368792,total=514850816
  1> [2018-10-30T17:11:26,962][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testAllocateActionOnlyReplicas] after test
  1> [2018-10-30T17:11:26,974][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testReadOnly] before test
  1> [2018-10-30T17:11:27,608][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testReadOnly] after test
  2> NOTE: All tests run in this JVM: [TimeSeriesLifecycleActionsIT]
  1> [2018-10-30T17:11:27,622][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testMoveToRolloverStep] before test
  1> [2018-10-30T17:11:28,234][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testMoveToRolloverStep] {"phase":"new","action":"complete","name":"complete"}
  1> [2018-10-30T17:11:39,817][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testMoveToRolloverStep] after test
FAILURE 12.2s | TimeSeriesLifecycleActionsIT.testMoveToRolloverStep <<< FAILURES!
   > Throwable #1: java.lang.AssertionError
   > 	at __randomizedtesting.SeedInfo.seed([90FF073C41149E5:56B2F5E20B273372]:0)
   > 	at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:820)
   > 	at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.testMoveToRolloverStep(TimeSeriesLifecycleActionsIT.java:155)
   > 	at java.lang.Thread.run(Thread.java:748)
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testMoveToRolloverStep$7(TimeSeriesLifecycleActionsIT.java:155)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
  1> [2018-10-30T17:11:39,850][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testFullPolicy] before test
  1> [2018-10-30T17:11:52,789][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testFullPolicy] after test
FAILURE 13.0s | TimeSeriesLifecycleActionsIT.testFullPolicy <<< FAILURES!
   > Throwable #1: java.lang.AssertionError
   > 	at __randomizedtesting.SeedInfo.seed([90FF073C41149E5:ED0437484D49D27E]:0)
   > 	at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:846)
   > 	at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:820)
   > 	at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.testFullPolicy(TimeSeriesLifecycleActionsIT.java:87)
   > 	at java.lang.Thread.run(Thread.java:748)
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
   > 	Suppressed: java.lang.AssertionError
   > 		at org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.lambda$testFullPolicy$2(TimeSeriesLifecycleActionsIT.java:87)
   > 		at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:834)
   > 		... 39 more
  1> [2018-10-30T17:11:52,804][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testDelete] before test
  1> [2018-10-30T17:11:53,252][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testDelete] after test
  1> [2018-10-30T17:11:53,262][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testDeleteOnlyShouldNotMakeIndexReadonly] before test
  1> [2018-10-30T17:11:53,548][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testDeleteOnlyShouldNotMakeIndexReadonly] HTTP/1.1 201 Created
  1> [2018-10-30T17:11:53,678][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testDeleteOnlyShouldNotMakeIndexReadonly] after test
  1> [2018-10-30T17:11:53,687][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testRolloverAction] before test
  1> [2018-10-30T17:11:58,315][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testRolloverAction] after test
  1> [2018-10-30T17:11:58,325][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testShrinkAction] before test
  1> [2018-10-30T17:12:01,008][INFO ][o.e.x.i.TimeSeriesLifecycleActionsIT] [testShrinkAction] after test
Completed [1/1] in 48.63s, 12 tests, 2 failures <<< FAILURES!

[ant:junit4] JVM J0: stderr was not empty, see: /var/lib/jenkins/workspace/elastic+elasticsearch+pull-request/x-pack/plugin/ilm/qa/multi-node/build/testrun/integTestRunner/temp/junit4-J0-20181030_171111_92515324421693853845984.syserr

> Task :x-pack:plugin:ilm:qa:multi-node:integTestRunner FAILED
Tests with failures:
  - org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.testMoveToRolloverStep
  - org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.testFullPolicy

[elasticmachine]

Pinging @elastic/es-core-infra

[colings86]

Since #35113 is merged can this be closed now or is there still more work to be done here?

[talevy]

@colings86 yes. this can be closed

[bizybot]

Reopening this as I can see these intermittent failures on CI:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-darwin-compatibility/117/

org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.testMoveToRolloverStep
org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT.testFullPolicy

I could not reproduce them locally with:
./gradlew :x-pack:plugin:ilm:qa:multi-node:integTestRunner -Dtests.seed=F7A6B975A74882 -Dtests.class=org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT -Dtests.method="testFullPolicy" -Dtests.security.manager=true -Dtests.locale=el -Dtests.timezone=Africa/Lubumbashi -Dcompiler.java=11 -Druntime.java=8

./gradlew :x-pack:plugin:ilm:qa:multi-node:integTestRunner -Dtests.seed=F7A6B975A74882 -Dtests.class=org.elasticsearch.xpack.indexlifecycle.TimeSeriesLifecycleActionsIT -Dtests.method="testMoveToRolloverStep" -Dtests.security.manager=true -Dtests.locale=el -Dtests.timezone=Africa/Lubumbashi -Dcompiler.java=11 -Druntime.java=8

[talevy]

thank you @bizybot, I will take another look at this! I thought we had one the battle of
testability and flakiness

[talevy]

Haven't seen any build failures due to this since merging ^. Closing