SearchServiceTests.testCloseSearchContextOnRewriteException failure in CI

[romseygeek]

Failed for the first time in 6.6 today, does not reproduce:

java.lang.AssertionError: expected:<4> but was:<2>
	at __randomizedtesting.SeedInfo.seed([294506CB775D8415:354F8B75FC0B1804]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:631)
	at org.elasticsearch.search.SearchServiceTests.testCloseSearchContextOnRewriteException(SearchServiceTests.java:214)

Reproduce line:

REPRODUCE WITH: ./gradlew :server:unitTest \
  -Dtests.seed=294506CB775D8415 \
  -Dtests.class=org.elasticsearch.search.SearchServiceTests \
  -Dtests.method="testCloseSearchContextOnRewriteException" \
  -Dtests.security.manager=true \
  -Dtests.locale=ar-JO \
  -Dtests.timezone=Canada/Newfoundland \
  -Dcompiler.java=11 \
  -Druntime.java=8

Build log: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.6+multijob-unix-compatibility/os=opensuse/130/console

[elasticmachine]

Pinging @elastic/es-search

[cbuescher]

I found another similar looking error from about a year ago (18.3.2018), back then on the 6.x branch:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-java-periodic/ESJAVA=java9,ESRUNTIME=java10,nodes=linux/7/console

./gradlew :server:test \
  -Dtests.seed=52D6EB2EA1F2638F \
  -Dtests.class=org.elasticsearch.search.SearchServiceTests \
  -Dtests.method="testCloseSearchContextOnRewriteException" \
  -Dtests.security.manager=true \
  -Dtests.locale=rm-CH \
  -Dtests.timezone=America/Sitka

java.lang.AssertionError: expected:<3> but was:<2>
	at __randomizedtesting.SeedInfo.seed([52D6EB2EA1F2638F:4EDC66902AA4FF9E]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:631)
	at org.elasticsearch.search.SearchServiceTests.testCloseSearchContextOnRewriteException(SearchServiceTests.java:163)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)

[cbuescher]

The last one on 6.6. looks like this:

12:15:48    > Throwable #1: java.lang.AssertionError: expected:<4> but was:<2>
12:15:48    > 	at __randomizedtesting.SeedInfo.seed([294506CB775D8415:354F8B75FC0B1804]:0)
12:15:48    > 	at org.elasticsearch.search.SearchServiceTests.testCloseSearchContextOnRewriteException(SearchServiceTests.java:214)
12:15:48    > 	at java.lang.Thread.run(Thread.java:748)

[cbuescher]

Found another even earlier one from 2017:

using successful commit from
  20170905233910-976C2DD1
  elastic+elasticsearch+master+intake
  2017-09-06T00:42:43.827Z (3h 41m 47s ago)
branch master
commit f6a489f323cc0337a6124f22026a81d77dd09814

java.lang.AssertionError: expected:<2> but was:<3>
	at __randomizedtesting.SeedInfo.seed([51C48596F7A2FBE9:4DCE08287CF467F8]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:631)
	at org.elasticsearch.search.SearchServiceTests.testCloseSearchContextOnRewriteException(SearchServiceTests.java:145)