DrillSideways does not support intra-segment concurrency

[javanna]

With the introduction of intra-segment search concurrency (see #13542), DrillSideways is the only case that disables it in tests which can't support intra-segment slicing given its implementation requirements to go through all docs in a segment at once. This is asserted as well in its score method which checks that min and max are respectively 0 and NO_MORE_DOCS.

With intra-segment slicing randomly enabled in tests, we need to explicitly disable intra-segment concurrency in any test that relies on DrillSideways. That required adding an additional newSearcher method that takes in the supported concurrency: none, inter-segment or intra-segment.

This issue is to discuss a plan to support intra-segment concurrency in DrillSideways. Ideally, we'd be able to one day enable intra-segment slicing by default, once #13745 is addressed, but that would not make IndexSearcher usable in DrillSideways out of the box which is not a good user experience.

[kashkambath]

Hi @javanna,

Your explanation makes sense that intra-segment concurrency can't be leveraged by the existing DrillSidewaysScorer since it goes through all the docs in a segment. However, DrillSidewaysScorer is only used when a non-concurrent execution of drill sideways occurs via DrillSideways#searchSequentially() AFAIK.

If an Executor is passed into DrillSideways at its instantiation, then DrillSideways will execute the DrillSideways#searchConcurrently() codepath, which does not leverage the DrillSidewaysScorer. searchConcurrently instead runs in parallel N+1 queries for N facets (one base query + N holdout queries for facet counts.) I believe each of these N+1 queries are traditional search queries which could leverage inter-segment/intra-segment parallelism.