Composite Aggregation runs out of heap space when composite size is too large

[plcstevens]

Elasticsearch version (bin/elasticsearch --version):

$ elasticsearch --version
Version: 6.2.3, Build: c59ff00/2018-03-13T10:06:29.741383Z, JVM: 1.8.0_161

Plugins installed:

[]

JVM version (java -version):

$ java -version
java version "9.0.1"
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)

OS version (uname -a if on a Unix-like system):

$ uname -a
Darwin XXX 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

Description of the problem including expected versus actual behavior:

Using a rather large size value: 1_000_000_000 will lead to a Java OutOfMemory error and crash the ElasticSearch node (on a local machine)

While this number is outrageously large, if you use a multiple number of sources it is easy to lower this and thus hit the out of memory error earlier, even in indexes with very few documents and fields.

Expectation is that the aggregation would be more intelligent about handling large sizes (either by failing the request or limiting them to the matrix possibilities of sources and documents).

Steps to reproduce:

example.json

{ "index" : { "_id" : "1" } }
{"my_field": "example a"}
{ "index" : { "_id" : "2" } }
{"my_field": "example b"}

1. Bulk insert our data into ElasticSearch, amount unimportant.

curl -s -XPOST 'localhost:9200/my_index/my_type/_bulk?pretty' -H "Content-Type: application/x-ndjson" --data-binary @example.json

2. Request a composite aggregation (do not set size), response is as expected.

curl -s -XPOST 'localhost:9200/my_index/my_type/_search?pretty' -H "Content-Type: application/json" -id'
{
  "size": 0,
  "track_total_hits": false,
  "aggregations": {
    "attributes": {
      "composite": {
        "sources": [
          {
            "my_field": {
              "terms": {
                "field": "my_field.keyword"
              }
            }
          }
        ]
      }
    }
  }
}
'

3. Set an unreasonably high size for the composite bucket, (again only two documents).

curl -s -XPOST 'localhost:9200/my_index/my_type/_search?pretty' -H "Content-Type: application/json" -id'
{
  "size": 0,
  "track_total_hits": false,
  "aggregations": {
    "attributes": {
      "composite": {
        "size": 1000000000,
        "sources": [
          {
            "my_field": {
              "terms": {
                "field": "my_field.keyword"
              }
            }
          }
        ]
      }
    }
  }
}
'

4. ElasticSearch node will crash if it runs out of heap memory space and throw the log seen below.

Provide logs (if relevant):

[2018-04-04T21:19:49,478][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [elasticsearch[Q3vBGVN][search][T#5]], exiting
java.lang.OutOfMemoryError: Java heap space
	at org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesSource$GlobalOrdinalValuesSource.<init>(CompositeValuesSource.java:137) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesSource.wrapGlobalOrdinals(CompositeValuesSource.java:123) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.aggregations.bucket.composite.CompositeValuesComparator.<init>(CompositeValuesComparator.java:50) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregator.<init>(CompositeAggregator.java:69) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.aggregations.bucket.composite.CompositeAggregationFactory.createInternal(CompositeAggregationFactory.java:52) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.aggregations.AggregatorFactory.create(AggregatorFactory.java:216) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.aggregations.AggregatorFactories.createTopLevelAggregators(AggregatorFactories.java:216) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.aggregations.AggregationPhase.preProcess(AggregationPhase.java:55) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:105) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$14(IndicesService.java:1133) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.indices.IndicesService$$Lambda$1869/1694278624.accept(Unknown Source) ~[?:?]
	at org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$15(IndicesService.java:1186) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.indices.IndicesService$$Lambda$1870/957087815.get(Unknown Source) ~[?:?]
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:160) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:143) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:412) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:116) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1192) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1132) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:305) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:340) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:316) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:312) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1002) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.2.3.jar:6.2.3]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.3.jar:6.2.3]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]

[elasticmachine]

Pinging @elastic/es-search-aggs

[colings86]

@jimczi could you look into this one?