[CI] RemoveCorruptedShardDataCommandTests testCorruptedIndex fails

[vladimirdolzhenko]

logs: https://www.google.com/url?q=https%3A%2F%2Felasticsearch-ci.elastic.co%2Fjob%2Felastic%2Belasticsearch%2B6.x%2Bmultijob-unix-compatibility%2Fos%3Damazon%2F42%2Fconsole&sa=D&sntz=1&usg=AFQjCNFFG4hg9naqoVYMqXTq8OD-lszrlw

It is reproducible with:

./gradlew :server:test -Dtests.seed=F13528ED31F73FE7 -Dtests.class=org.elasticsearch.index.shard.RemoveCorruptedShardDataCommandTests -Dtests.method="testCorruptedIndex" -Dtests.security.manager=true -Dtests.locale=es-CO -Dtests.timezone=Canada/Central -Dcompiler.java=11 -Druntime.java=8

The reason:
corrupting segments file, when Lucene reads it, it throws generic IOException at https://github.com/apache/lucene-solr/blob/1d85cd783863f75cea133fb9c452302214165a4d/lucene/core/src/java/org/apache/lucene/store/DataInput.java#L141 and that's why ES would not consider it as an index corruption and does not catch it at https://github.com/elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/index/store/Store.java#L188 and therefore would not create a corruption marker (while RemoveCorruptedShardDataCommand relies on its presence)

It would be better if Lucene throws more specific exception rather generic one.

update: https://issues.apache.org/jira/browse/LUCENE-8525 to address this problem

[elasticmachine]

Pinging @elastic/es-distributed

[vladimirdolzhenko]

Another one:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+multijob-unix-compatibility/os=centos/58/console

REPRODUCE WITH: ./gradlew :server:test \
  -Dtests.seed=C8F1C7CB781F757A \
  -Dtests.class=org.elasticsearch.index.shard.RemoveCorruptedShardDataCommandTests \
  -Dtests.method="testCorruptedIndex" \
  -Dtests.security.manager=true \
  -Dtests.locale=pt-PT \
  -Dtests.timezone=Antarctica/Mawson \
  -Dcompiler.java=11 \
  -Druntime.java=8

stacktrace:

ERROR   0.46s J2 | RemoveCorruptedShardDataCommandTests.testCorruptedIndex <<< FAILURES!
   > Throwable #1: java.io.IOException: Invalid vInt detected (too many bits)
   > 	at __randomizedtesting.SeedInfo.seed([C8F1C7CB781F757A:F553B10F61CC26A7]:0)
   > 	at org.apache.lucene.store.DataInput.readVInt(DataInput.java:141)
   > 	at org.apache.lucene.store.DataInput.readSetOfStrings(DataInput.java:287)
   > 	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:379)
   > 	at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289)
   > 	at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:432)
   > 	at org.apache.lucene.index.SegmentInfos$1.doBody(SegmentInfos.java:429)
   > 	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:680)
   > 	at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:632)
   > 	at org.apache.lucene.index.SegmentInfos.readLatestCommit(SegmentInfos.java:434)
   > 	at org.apache.lucene.index.DirectoryReader.listCommits(DirectoryReader.java:260)
   > 	at org.elasticsearch.index.translog.TruncateTranslogAction.getCleanStatus(TruncateTranslogAction.java:71)
   > 	at org.elasticsearch.index.shard.RemoveCorruptedShardDataCommand.lambda$execute$1(RemoveCorruptedShardDataCommand.java:328)