SearchAfterIT.testsShouldFail failure on CI

[gwbrown]

This failed on a PR build with what should be unrelated changes. According to build-stats, that's the only time this has failed in the past 90 days.

Build scan

Reproduce line (does not reproduce locally):

./gradlew ':server:integTest' --tests "org.elasticsearch.search.searchafter.SearchAfterIT.testsShouldFail" -Dtests.seed=88B485F15AB25193 -Dtests.security.manager=true -Dtests.locale=es-BO -Dtests.timezone=SystemV/AST4 -Dcompiler.java=13

Error:

Expected: a string containing "`search_after` cannot be used in a scroll context."
     but: was "shard [[X24pND3CT123n68A2FwMsA][test][0]], reason [org.elasticsearch.transport.RemoteTransportException: [node_s0][127.0.0.1:45625][indices:data/read/search[phase/dfs]]
Caused by: [test/AmxN6LAESp66Xi9X1jqAuQ] org.elasticsearch.index.query.QueryShardException: No mapping found for [field1] in order to sort on
	at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:334)
	at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:146)
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:769)
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:594)
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:553)
	at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:315)
	at org.elasticsearch.search.SearchService.lambda$executeDfsPhase$0(SearchService.java:311)
	at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:157)
	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58)
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:688)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
], cause [[test/AmxN6LAESp66Xi9X1jqAuQ] org.elasticsearch.index.query.QueryShardException: No mapping found for [field1] in order to sort on
	at org.elasticsearch.search.sort.FieldSortBuilder.build(FieldSortBuilder.java:334)
	at org.elasticsearch.search.sort.SortBuilder.buildSort(SortBuilder.java:146)
	at org.elasticsearch.search.SearchService.parseSource(SearchService.java:769)
	at org.elasticsearch.search.SearchService.createContext(SearchService.java:594)
	at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:553)
	at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:315)
	at org.elasticsearch.search.SearchService.lambda$executeDfsPhase$0(SearchService.java:311)
	at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:157)
	at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:58)
	at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:688)
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(Thread.java:834)
]"
Close stacktrace
at __randomizedtesting.SeedInfo.seed([88B485F15AB25193:7EE3C4B9EA071DA4]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
at org.junit.Assert.assertThat(Assert.java:956)
at org.junit.Assert.assertThat(Assert.java:923)
at org.elasticsearch.search.searchafter.SearchAfterIT.testsShouldFail(SearchAfterIT.java:67)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:834)

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[romseygeek]

The failure looks to be due to a delay in propagating a dynamic mapping update from the cluster state to one of the shards. I think we can fix it by explicitly creating the mapping for field1 at the top of the test, although it's difficult to tell as this is such a rare failure (twice in three years)

[DaveCTurner]

Looks like https://gradle-enterprise.elastic.co/s/gpym5qee36puo was also due to this.