IndicesRequestCacheIT#testQueryRewrite fails

[dnhatn]

IndicesRequestCacheIT#testQueryRewrite fails with an unexpected cache count.

FAILURE 7.39s J1 | IndicesRequestCacheIT.testQueryRewrite <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: 
   > Expected: <3L>
   >      but: was <2L>
   > 	at __randomizedtesting.SeedInfo.seed([7B6066660D23E3B1:8A7EC1C3E5F6C580]:0)
   > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
   > 	at org.elasticsearch.indices.IndicesRequestCacheIT.testQueryRewrite(IndicesRequestCacheIT.java:131)
   > 	at java.lang.Thread.run(Thread.java:748)

This does not reproduce locally for me.

./gradlew :server:integTest \
  -Dtests.seed=7B6066660D23E3B1 \
  -Dtests.class=org.elasticsearch.indices.IndicesRequestCacheIT \
  -Dtests.method="testQueryRewrite" \
  -Dtests.security.manager=true \
  -Dtests.locale=en \
  -Dtests.timezone=Europe/Gibraltar \
  -Dcompiler.java=10 \
  -Druntime.java=8

CI: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+periodic/6845/console
Log: testQueryRewrite.txt

[elasticmachine]

Pinging @elastic/es-search-aggs

[javanna]

Although on a different test method in the same test class, I think this is the same failure: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java11,nodes=virtual&&linux/265/console . I can't repro either.

07:10:35   1> [2018-08-28T08:10:30,847][INFO ][o.e.i.IndicesRequestCacheIT] [testQueryRewriteDatesWithNow] after test
07:10:35 FAILURE 2.58s J1 | IndicesRequestCacheIT.testQueryRewriteDatesWithNow <<< FAILURES!
07:10:35    > Throwable #1: java.lang.AssertionError: 
07:10:35    > Expected: <2L>
07:10:35    >      but: was <1L>
07:10:35    > 	at __randomizedtesting.SeedInfo.seed([FACB28E405660D99:4EC92144B233E176]:0)
07:10:35    > 	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
07:10:35    > 	at org.elasticsearch.indices.IndicesRequestCacheIT.testQueryRewriteDatesWithNow(IndicesRequestCacheIT.java:349)
07:10:35    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
07:10:35    > 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
07:10:35    > 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
07:10:35    > 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
07:10:35    > 	at java.base/java.lang.Thread.run(Thread.java:834)

[colings86]

I merged #33313 last week that should hopefully fix this and if not sure provide better information if it occurs again. I'm going to close this issue for now but please re-open if the issue re-occurs

[imotov]

@colings86 after days of silence we have got two new failures in the last 10 hours. Could you take a look if they provide more information:

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+release-tests/33/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+matrix-java-periodic/ES_BUILD_JAVA=java11,ES_RUNTIME_JAVA=java8,nodes=virtual&&linux/22/console

[colings86]

I opened #34180 to add some more logging information and hopefully see whats going on better

[colings86]

#34180 is now merged. Please ping me if this fails again as the PR only adds debugging information.