[Meta] Lucene 9 Upgrade Punch List

[nknize]

The following outstanding (and currently muted) test failures still need to be addressed as part of the Lucene 9 upgrade:

• PluginsServiceTests failures due to AccessControlException
• CustomUnifiedHighlighterTests
• HighlighterWithAnalyzersTests.testPhrasePrefix
• ChildQuerySearchIT.testHighlightersIgnoreParentChild
• HighlighterSearchIT failures
• IndexShardHotspotTests
• RelocationIT.testRelocationWhileIndexingRandom (seed -Dtests.seed=6158774967A60B81)
• NumberFieldTypeTests.testIndexSortIntRange
• IndexPrimaryRelocationIT ([Unmute] IndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing #2488)

relates #1109

[reta]

These ones should be fixed now:

CustomUnifiedHighlighterTests
HighlighterWithAnalyzersTests.testPhrasePrefix
ChildQuerySearchIT.testHighlightersIgnoreParentChild
HighlighterSearchIT failures

But we would need to work on accommodating these change: https://github.com/apache/lucene/pull/362/files

[reta]

For NumberFieldTypeTests, opened https://issues.apache.org/jira/browse/LUCENE-10466

[reta]

PluginsServiceTests are green, #2484

[nknize]

• RelocationIT.testRelocationWhileIndexingRandom (seed -Dtests.seed=6158774967A60B81)

Fixed by #2452

[reta]

@nknize I am working on NumberFieldTypeTests.testIndexSortIntRange

[nknize]

IndexShardHotspotTests are passing but this one test is taking upwards of 25 minutes+. @ashwinpankaj can we reduce this test from 100 indices + 5 shards to something much more expedient?

I don't want to unmute if this is going to kill our tests on each PR.

Also, a whole slew of the following logs:

...
[2022-03-16T12:08:09,916][INFO ][o.o.c.r.a.a.BalancedShardsAllocator] [testClusterScaleInWithSkew] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration
[2022-03-16T12:08:11,283][INFO ][o.o.c.r.a.a.BalancedShardsAllocator] [testClusterScaleInWithSkew] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration
[2022-03-16T12:08:12,343][INFO ][o.o.c.r.a.a.BalancedShardsAllocator] [testClusterScaleInWithSkew] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration
[2022-03-16T12:08:13,229][INFO ][o.o.c.r.a.a.BalancedShardsAllocator] [testClusterScaleInWithSkew] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration
[2022-03-16T12:08:13,846][INFO ][o.o.c.r.a.a.BalancedShardsAllocator] [testClusterScaleInWithSkew] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration
[2022-03-16T12:08:16,243][INFO ][o.o.c.r.a.a.BalancedShardsAllocator] [testClusterScaleInWithSkew] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration
[2022-03-16T12:08:19,200][INFO ][o.o.c.r.a.a.BalancedShardsAllocator] [testClusterScaleInWithSkew] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration
[2022-03-16T12:08:20,625][INFO ][o.o.c.r.a.a.BalancedShardsAllocator] [testClusterScaleInWithSkew] Cannot move any shard in the cluster as there is no node on which shards can be allocated. Skipping shard iteration

[nknize]

Documenting Points dimension failure from #2483 for posterity:

./gradlew ':server:internalClusterTest' --tests "org.opensearch.search.simple.SimpleSearchIT.testSimpleIndexSortEarlyTerminate" -Dtests.seed=E45EED588B09BBF2 -Dtests.security.manager=true -Dtests.jvm.argline="-XX:TieredStopAtLevel=1 -XX:ReservedCodeCacheSize=64m" -Dtests.locale=be-BY -Dtests.timezone=Pacific/Easter -Druntime.java=17

  1> java.lang.IllegalArgumentException: Field rank is indexed with 4 bytes per dimension, but org.apache.lucene.search.SortedNumericSortField$3@3f6c3860 expected 8
  1> 	at org.apache.lucene.search.comparators.NumericComparator$NumericLeafComparator.<init>(NumericComparator.java:116)
  1> 	at org.apache.lucene.search.comparators.LongComparator$LongLeafComparator.<init>(LongComparator.java:65)
  1> 	at org.apache.lucene.search.SortedNumericSortField$3$1.<init>(SortedNumericSortField.java:312)
  1> 	at org.apache.lucene.search.SortedNumericSortField$3.getLeafComparator(SortedNumericSortField.java:312)
  1> 	at org.apache.lucene.index.CheckIndex.testSort(CheckIndex.java:1109)
  1> 	at org.apache.lucene.index.CheckIndex.testSegment(CheckIndex.java:1012)
  1> 	at org.apache.lucene.index.CheckIndex.lambda$checkIndex$1(CheckIndex.java:762)
  1> 	at org.apache.lucene.index.CheckIndex.lambda$callableToSupplier$2(CheckIndex.java:860)
  1> 	at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768)
  1> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
  1> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
  1> 	at java.base/java.lang.Thread.run(Thread.java:833)