[CI] RollupActionIT testRollupIndexAndSetNewRollupPolicy failing

[mark-vieira]

This is a new test case which has failed a few times now since being added.

Build scan:
https://gradle-enterprise.elastic.co/s/nckptifuyp4v2/tests/:x-pack:plugin:ilm:qa:multi-node:javaRestTest/org.elasticsearch.xpack.ilm.actions.RollupActionIT/testRollupIndexAndSetNewRollupPolicy#1

Repro line:
./gradlew ':x-pack:plugin:ilm:qa:multi-node:javaRestTest' --tests "org.elasticsearch.xpack.ilm.actions.RollupActionIT.testRollupIndexAndSetNewRollupPolicy" -Dtests.seed=F0A4BF22F8036B7E -Dtests.security.manager=true -Dtests.locale=sk-SK -Dtests.timezone=Pacific/Truk -Druntime.java=11

Reproduces locally?:
Nope.

Applicable branches:
master and 7.x (error on the 7.x build looked a bit different though)

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?search.relativeStartTime=P7D&search.timeZoneId=America/Los_Angeles&tests.container=org.elasticsearch.xpack.ilm.actions.RollupActionIT&tests.sortField=FAILED&tests.unstableOnly=true

Failure excerpt:

org.elasticsearch.xpack.ilm.actions.RollupActionIT > testRollupIndexAndSetNewRollupPolicy FAILED
    java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([F0A4BF22F8036B7E:7FF7996A3D79E9A]:0)

[elasticmachine]

Pinging @elastic/es-core-features (Team:Core/Features)

[mark-vieira]

Since this is a newly added test I've muted it in master and 7x..

[mark-vieira]

@andreidan since you added this test do you mind taking a look?

[dakrone]

Actually I think @talevy might be better, since he recently did the work that added the rollup stuff to ILM, @talevy can you take a look at this?

(Andrei is on the git blame because of moving the tests around)

[talevy]

yes. I'll investigate. thanks for the ping and for muting the test!

[elasticmachine]

Pinging @elastic/es-analytics-geo (Team:Analytics)

[mark-vieira]

(Andrei is on the git blame because of moving the tests around)

Ah, thanks for the clarification @dakrone. The "old" test has failed in a similar manner before as well: https://gradle-enterprise.elastic.co/s/5o6gpcxlp5uri/tests/:x-pack:plugin:ilm:qa:multi-node:javaRestTest/org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT/testRollupIndexAndSetNewRollupPolicy#1

[mark-vieira]

FYI, I've seen other builds with slightly different test failures as well but the common thing I see is this in some of the node logs:

» [2021-02-05T19:26:13,436][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [javaRestTest-0] fatal error in thread [elasticsearch[javaRestTest-0][rollup_indexing][T#1]], exiting
»  java.lang.AssertionError: null
»  	at org.elasticsearch.xpack.rollup.v2.RollupShardIndexer.checkCleanDirectory(RollupShardIndexer.java:207) ~[?:?]
»  	at org.elasticsearch.xpack.rollup.v2.RollupShardIndexer.execute(RollupShardIndexer.java:196) ~[?:?]
»  	at org.elasticsearch.xpack.rollup.v2.TransportRollupIndexerAction.shardOperation(TransportRollupIndexerAction.java:112) ~[?:?]
»  	at org.elasticsearch.xpack.rollup.v2.TransportRollupIndexerAction.shardOperation(TransportRollupIndexerAction.java:46) ~[?:?]
»  	at org.elasticsearch.action.support.broadcast.TransportBroadcastAction.lambda$asyncShardOperation$0(TransportBroadcastAction.java:291) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:728) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.0.0-SNAPSHOT.jar:8.0.0-SNAPSHOT]
»  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
»  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
»  	at java.lang.Thread.run(Thread.java:834) [?:?]

I think these might all be related.

[mark-vieira]

Also @talevy we might want to improve some of the error messages in there as per what Lee and I talked about in slack.

https://elastic.slack.com/archives/C0D8ST60Y/p1612563629493100