Search can fail when size is 0 and long sort optimization enabled.

[toby-jn]

Elasticsearch version (bin/elasticsearch --version):

Version: 7.7.0, Build: oss/docker/81a1e9eda8e6183f5237786246f6dced26a10eaf/2020-05-12T02:01:37.602180Z, JVM: 14

Plugins installed: []

JVM version (java -version):

openjdk version "14" 2020-03-17
OpenJDK Runtime Environment AdoptOpenJDK (build 14+36)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14+36, mixed mode, sharing)

OS version (uname -a if on a Unix-like system):

Linux elasticsearch-0 4.19.102+ #1 SMP Tue Feb 11 05:33:03 PST 2020 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:

After upgrading to 7.7.0 from 7.5.1, when doing a search with track_total_hits=true, size=0 and sort I get an error:

curl -X POST '127.0.0.1:9200/v33.tcpevent-000174/_search?pretty' -H content-type:application/json --data '{"size":0,"sort":[{"@timestamp":{"order":"desc"}},{"conn_uuid":{"order":"desc"}}],"track_total_hits":true}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 1,
        "index" : "v33.tcpevent-000174",
        "node" : "j07kSSNmSFueB_E9i911MQ",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count"
        }
      }
    ],
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count"
      }
    }
  },
  "status" : 400
}

The index settings have:

        "sort" : {
          "field" : "@timestamp",
          "order" : "desc"
        }

If I set size to 1 or change the sort parameter to match the index sort then the query works. Both these queries work:

{"size":0,"sort":[{"@timestamp":{"order":"desc"}}],"track_total_hits":true}
{"size":1,"sort":[{"@timestamp":{"order":"desc"}},{"conn_uuid":{"order":"desc"}}],"track_total_hits":true}

I've tried to create a minimal reproducible setup for this error, but using the same mappings and query I've not been able to trigger the error.

Logs:
Query run with error trace:

curl -s -X POST '127.0.0.1:9200/v33.tcpevent-000174/_search?pretty&error_trace=true' -H content-type:application/json --data '{"size":0,"sort":[{"@timestamp":{"order":"desc"}},{"conn_uuid":{"order":"desc"}}],"track_total_hits":1000}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count",
        "stack_trace" : "[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count];\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)\n\tat org.elasticsearch.action.search.SearchPhaseExecutionException.guessRootCauses(SearchPhaseExecutionException.java:167)\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:626)\n\tat org.elasticsearch.ElasticsearchException.generateFailureXContent(ElasticsearchException.java:572)\n\tat org.elasticsearch.rest.BytesRestResponse.build(BytesRestResponse.java:138)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:96)\n\tat org.elasticsearch.rest.BytesRestResponse.<init>(BytesRestResponse.java:91)\n\tat org.elasticsearch.rest.action.RestActionListener.onFailure(RestActionListener.java:58)\n\tat org.elasticsearch.rest.action.RestCancellableNodeClient$1.onFailure(RestCancellableNodeClient.java:106)\n\tat org.elasticsearch.action.support.TransportAction$1.onFailure(TransportAction.java:79)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.raisePhaseFailure(AbstractSearchAsyncAction.java:571)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:580)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245)\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:402)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139)\n\tat org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:242)\n\tat org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:234)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:137)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\nCaused by: java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count\n\tat org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:464)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:502)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:495)\n\tat org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:166)\n\tat org.elasticsearch.search.query.QueryPhase.searchWithCollectorManager(QueryPhase.java:402)\n\tat org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:297)\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)\n\tat org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1344)\n\tat org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1396)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)\n\tat org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:125)\n\tat org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1402)\n\tat org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1341)\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:359)\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)\n\tat org.elasticsearch.search.SearchService.access$200(SearchService.java:135)\n\tat org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)\n\tat org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.lang.Thread.run(Thread.java:832)\n"
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 2,
        "index" : "v33.tcpevent-000174",
        "node" : "yQLAAU2QTN6ORtWRIhqsFQ",
        "reason" : {
          "type" : "illegal_argument_exception",
          "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count",
          "stack_trace" : "java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count\n\tat org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:464)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:502)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:495)\n\tat org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:166)\n\tat org.elasticsearch.search.query.QueryPhase.searchWithCollectorManager(QueryPhase.java:402)\n\tat org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:297)\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)\n\tat org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1344)\n\tat org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1396)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)\n\tat org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:125)\n\tat org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1402)\n\tat org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1341)\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:359)\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)\n\tat org.elasticsearch.search.SearchService.access$200(SearchService.java:135)\n\tat org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)\n\tat org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.lang.Thread.run(Thread.java:832)\n"
        }
      }
    ],
    "caused_by" : {
      "type" : "illegal_argument_exception",
      "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count",
      "caused_by" : {
        "type" : "illegal_argument_exception",
        "reason" : "numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count",
        "stack_trace" : "java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count\n\tat org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:464)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:502)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:495)\n\tat org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:166)\n\tat org.elasticsearch.search.query.QueryPhase.searchWithCollectorManager(QueryPhase.java:402)\n\tat org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:297)\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)\n\tat org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1344)\n\tat org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1396)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)\n\tat org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:125)\n\tat org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1402)\n\tat org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1341)\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:359)\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)\n\tat org.elasticsearch.search.SearchService.access$200(SearchService.java:135)\n\tat org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)\n\tat org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.lang.Thread.run(Thread.java:832)\n"
      },
      "stack_trace" : "[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count];\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:307)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:580)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245)\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:402)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139)\n\tat org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:242)\n\tat org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:234)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:137)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\nCaused by: java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count\n\tat org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:464)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:502)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:495)\n\tat org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:166)\n\tat org.elasticsearch.search.query.QueryPhase.searchWithCollectorManager(QueryPhase.java:402)\n\tat org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:297)\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)\n\tat org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1344)\n\tat org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1396)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)\n\tat org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:125)\n\tat org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1402)\n\tat org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1341)\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:359)\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)\n\tat org.elasticsearch.search.SearchService.access$200(SearchService.java:135)\n\tat org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)\n\tat org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.lang.Thread.run(Thread.java:832)\n"
    },
    "stack_trace" : "Failed to execute phase [query], all shards failed; shardFailures {[yQLAAU2QTN6ORtWRIhqsFQ][v33.tcpevent-000174][2]: RemoteTransportException[[elasticsearch-2][10.32.130.157:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; }{[6RDmCBlcRM-dVtnwzcXTog][v33.tcpevent-000174][0]: RemoteTransportException[[elasticsearch-0][10.32.136.152:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; }{[VX6qSJggQAmNL_HPnz6hhw][v33.tcpevent-000174][1]: RemoteTransportException[[elasticsearch-3][10.32.134.121:9300][indices:data/read/search[phase/query]]]; nested: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; }\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:551)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:309)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:580)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:393)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.access$100(AbstractSearchAsyncAction.java:68)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction$1.onFailure(AbstractSearchAsyncAction.java:245)\n\tat org.elasticsearch.action.search.SearchExecutionStatsCollector.onFailure(SearchExecutionStatsCollector.java:73)\n\tat org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59)\n\tat org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:402)\n\tat org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1139)\n\tat org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:244)\n\tat org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225)\n\tat org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:242)\n\tat org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:234)\n\tat org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:137)\n\tat org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:103)\n\tat org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:676)\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:62)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:227)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:832)\nCaused by: [numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count]; nested: IllegalArgumentException[numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count];\n\tat org.elasticsearch.ElasticsearchException.guessRootCauses(ElasticsearchException.java:644)\n\tat org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:307)\n\t... 40 more\nCaused by: java.lang.IllegalArgumentException: numHits must be > 0; please use TotalHitCountCollector if you just need the total hit count\n\tat org.apache.lucene.search.TopFieldCollector.create(TopFieldCollector.java:464)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:502)\n\tat org.apache.lucene.search.TopFieldCollector$1.newCollector(TopFieldCollector.java:495)\n\tat org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:166)\n\tat org.elasticsearch.search.query.QueryPhase.searchWithCollectorManager(QueryPhase.java:402)\n\tat org.elasticsearch.search.query.QueryPhase.executeInternal(QueryPhase.java:297)\n\tat org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:151)\n\tat org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$21(IndicesService.java:1344)\n\tat org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$22(IndicesService.java:1396)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:176)\n\tat org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:159)\n\tat org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:433)\n\tat org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:125)\n\tat org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1402)\n\tat org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1341)\n\tat org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:359)\n\tat org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:434)\n\tat org.elasticsearch.search.SearchService.access$200(SearchService.java:135)\n\tat org.elasticsearch.search.SearchService$2.lambda$onResponse$0(SearchService.java:395)\n\tat org.elasticsearch.search.SearchService.lambda$runAsync$0(SearchService.java:411)\n\tat org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44)\n\tat org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692)\n\tat org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)\n\tat java.lang.Thread.run(Thread.java:832)\n"
  },
  "status" : 400
}

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[toby-jn]

It looks to me like the error is coming from: https://github.com/elastic/elasticsearch/blob/v7.7.0/server/src/main/java/org/elasticsearch/search/query/QueryPhase.java#L376
Though I'm not sure why it's not easier to reproduce this error.

[jtibshirani]

@toby-jn thanks for raising this bug. From digging into the stack trace, it looks like this happens when (1) the search size is 0, and (2) the 'long sort' optimization is enabled.

We added the optimization in 7.6 to speed up searches that sort on a long field (#48804). The optimization only kicks in if there are > 512 documents, and the sort doesn't match the index sort exactly (otherwise we could use the normal index sort optimization). This could help explain why you aren't able to put together a simple reproduction using just a few documents.

We will look into a fix. In the meantime, a workaround would be to remove the sort section for these searches that just count total hits, since sorting won't have an affect on the result.

[toby-jn]

Just managed to reproduce it, but looks like you beat me to why it wasn't working before, needed to add more documents for testing. Now able to reliably reproduce this with the following script on a fresh 7.7.0 install:

curl -X PUT 127.0.0.1:9200/twitter -H 'Content-Type: application/json' -d'
{
    "settings" : {
        "index" : {
            "sort.field" : "date",
            "sort.order": "desc"
        }
    },
    "mappings": {
        "properties": {
            "date": {
                "type": "date"
            },
            "title": {
                "type": "keyword"
            }
        }
    }
}
'

# index 20k documents
yes '{"index": {}}
{"date": "2020-05-18T19:21:36Z","title": "title1"}' | head -n 40000 | curl -X POST 127.0.0.1:9200/twitter/_bulk?refresh -H 'Content-Type: application/json' --data-binary @- > /dev/null

# works
curl -X POST '127.0.0.1:9200/twitter/_search?error_trace=true&pretty' -H content-type:application/json --data '{"size":0,"sort":[{"date":{"order":"desc"}}],"track_total_hits":true}'

# error
curl -X POST '127.0.0.1:9200/twitter/_search?error_trace=true&pretty' -H content-type:application/json --data '{"size":0,"sort":[{"date":{"order":"desc"}},{"title":{"order":"asc"}}],"track_total_hits":true}'