GatewayIndexStateIT.testRecoverBrokenIndexMetadata fails on master

[colings86]

Build URL: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/2952/console

Reproduce Command (reproduces locally with this seed):

./gradlew :server:integTest -Dtests.seed=EECC1AE0C30C4E5F -Dtests.class=org.elasticsearch.gateway.GatewayIndexStateIT -Dtests.method="testRecoverBrokenIndexMetadata" -Dtests.security.manager=true -Dtests.locale=de -Dtests.timezone=Europe/Athens -Dcompiler.java=12 -Druntime.java=8

Stack trace:

ERROR   31.6s J5 | GatewayIndexStateIT.testRecoverBrokenIndexMetadata <<< FAILURES!
   > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=6072, name=elasticsearch[node_t0][generic][T#4], state=RUNNABLE, group=TGRP-GatewayIndexStateIT]
   > 	at __randomizedtesting.SeedInfo.seed([EECC1AE0C30C4E5F:1675F3837244CBA3]:0)
   > Caused by: java.lang.AssertionError: max seq. no. [-1] does not match [0]
   > 	at __randomizedtesting.SeedInfo.seed([EECC1AE0C30C4E5F]:0)
   > 	at org.elasticsearch.index.engine.ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:142)
   > 	at org.elasticsearch.index.engine.ReadOnlyEngine.<init>(ReadOnlyEngine.java:116)
   > 	at org.elasticsearch.index.engine.NoOpEngine.<init>(NoOpEngine.java:46)
   > 	at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1447)
   > 	at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1397)
   > 	at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433)
   > 	at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95)
   > 	at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308)
   > 	at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93)
   > 	at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1686)
   > 	at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2339)
   > 	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:677)
   > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
   > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
   > 	at java.lang.Thread.run(Thread.java:748)

Will muted this test shortly

[elasticmachine]

Pinging @elastic/es-distributed

[colings86]

Muted on master in cd8dd04

[dnhatn]

This test failure relates to #40423 where we don't build analyzers when opening the IndexService for a closed index. However, the actual cause is shards of the closed index are not flushed before closing because the verification step does not go through when primaries are unassigned.

What if the primaries are not all allocated? The verification step cannot go through or shards might not be properly flushed. Should we close the close index API? Should we allocate an empty routing table that cannot be allocated at all?

This is already tracked in the meta-issue of replicate closed indices (#33888). I linked this test failure to the bullet and leave this muted.

[bizybot]

One of my PR builds failed but for a different test: testRecoverMissingAnalyzer

Reproducible locally with command:

./gradlew :server:integTest \
-Dtests.seed=F3F656161D025A0 \
-Dtests.class=org.elasticsearch.gateway.GatewayIndexStateIT \
-Dtests.method="testRecoverMissingAnalyzer" \
-Dtests.security.manager=true \
-Dtests.locale=id-ID \
-Dtests.timezone=Pacific/Galapagos \
-Dcompiler.java=12 \
-Druntime.java=11

   > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=83, name=elasticsearch[node_t1][generic][T#2], state=RUNNABLE, group=TGRP-GatewayIndexStateIT]
   >    at __randomizedtesting.SeedInfo.seed([F3F656161D025A0:E796CB546B33FF52]:0)
   > Caused by: java.lang.AssertionError: max seq. no. [-1] does not match [0]
   >    at __randomizedtesting.SeedInfo.seed([F3F656161D025A0]:0)
   >    at org.elasticsearch.index.engine.ReadOnlyEngine.assertMaxSeqNoEqualsToGlobalCheckpoint(ReadOnlyEngine.java:143)
   >    at org.elasticsearch.index.engine.ReadOnlyEngine.<init>(ReadOnlyEngine.java:117)
   >    at org.elasticsearch.index.engine.NoOpEngine.<init>(NoOpEngine.java:46)
   >    at org.elasticsearch.index.shard.IndexShard.innerOpenEngineAndTranslog(IndexShard.java:1447)
   >    at org.elasticsearch.index.shard.IndexShard.openEngineAndRecoverFromTranslog(IndexShard.java:1397)
   >    at org.elasticsearch.index.shard.StoreRecovery.internalRecoverFromStore(StoreRecovery.java:433)
   >    at org.elasticsearch.index.shard.StoreRecovery.lambda$recoverFromStore$0(StoreRecovery.java:95)
   >    at org.elasticsearch.index.shard.StoreRecovery.executeRecovery(StoreRecovery.java:308)
   >    at org.elasticsearch.index.shard.StoreRecovery.recoverFromStore(StoreRecovery.java:93)
   >    at org.elasticsearch.index.shard.IndexShard.recoverFromStore(IndexShard.java:1686)
   >    at org.elasticsearch.index.shard.IndexShard.lambda$startRecovery$9(IndexShard.java:2339)
   >    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:677)
   >    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   >    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

[dnhatn]

I muted testRecoverMissingAnalyzer in aaec11f.

[ywelsch]

@dnhatn is this test fixed now by flushing at the end of peer recovery? If that's not sufficient, should we just explicitly flush in the test before we restart the node?