[CI] FollowIndexSecurityIT testAutoFollowPatterns failing

[mark-vieira]

This and testFollowIndex have failed a few times with the same error. Couldn't reproduce locally.

Build scan:
https://gradle-enterprise.elastic.co/s/pqz55bchjt7l6/tests/:x-pack:plugin:ccr:qa:security:follow-cluster/org.elasticsearch.xpack.ccr.FollowIndexSecurityIT/testAutoFollowPatterns

Reproduction line:
./gradlew ':x-pack:plugin:ccr:qa:security:follow-cluster' --tests "org.elasticsearch.xpack.ccr.FollowIndexSecurityIT.testAutoFollowPatterns" -Dtests.seed=706D9EF1E410A031 -Dtests.locale=es-PY -Dtests.timezone=Asia/Colombo -Druntime.java=17

Applicable branches:
master, 8.1

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.ccr.FollowIndexSecurityIT&tests.test=testAutoFollowPatterns

Failure excerpt:

java.lang.AssertionError: 
Expected: a value equal to or greater than <1>
     but: <0> was less than <1>

  at __randomizedtesting.SeedInfo.seed([706D9EF1E410A031:D9CA20DA2CFE31B5]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.junit.Assert.assertThat(Assert.java:923)
  at org.elasticsearch.xpack.ccr.ESCCRRestTestCase.verifyAutoFollowMonitoring(ESCCRRestTestCase.java:247)
  at org.elasticsearch.xpack.ccr.FollowIndexSecurityIT.lambda$testAutoFollowPatterns$15(FollowIndexSecurityIT.java:183)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1070)
  at org.elasticsearch.xpack.ccr.FollowIndexSecurityIT.testAutoFollowPatterns(FollowIndexSecurityIT.java:181)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:833)

[elasticmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[ywangd]

Happened again on master intake https://gradle-enterprise.elastic.co/s/o2nsdcoi5izzm/tests/:x-pack:plugin:ccr:qa:security:follow-cluster/org.elasticsearch.xpack.ccr.FollowIndexSecurityIT/testAutoFollowPatterns?top-execution=1

[tlrx]

Looks similar to #84403 so I'll look at both.

[pgomulka]

another one in 7.17 https://gradle-enterprise.elastic.co/s/ftqmusykfoj22/tests/:x-pack:plugin:ccr:qa:security:follow-cluster/org.elasticsearch.xpack.ccr.FollowIndexSecurityIT/testAutoFollowPatterns

[henningandersen]

This also failed on a 7.17 intake build

Looks like a permission issue for retention leases. I think we saw this before. I am not 100% sure it is the same underlying cause as the other failures here, but figured I'd rather report it here than separate (same test case).

[2022-03-24T10:32:36,511][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [follow-cluster-0] fatal error in thread [Thread-5], exiting
java.lang.AssertionError: ElasticsearchSecurityException[action [indices:admin/seq_no/renew_retention_lease] is unauthorized for user [test_ccr] with roles [ccruser], this action is granted by the index privileges [manage,all]]

[tlrx]

@henningandersen I think the issue you mention requires #84467 to be backported to 7.17 (I'm checking with Ievgen).

[Luegg]

Another one https://gradle-enterprise.elastic.co/s/xmqr6idbdvhii

[tlrx]

Another one https://gradle-enterprise.elastic.co/s/xmqr6idbdvhii

@Luegg this is a different test, I reopened 81302.