[ML] Support single bucket aggregations in datafeeds

[davidkyle]

This use case was described in the discuss forum Referencing field name from datafeed aggregation to use as a detector in an ML job
where it is desired to use a filter aggregation in the datafeed.

  "aggs": {
    "metric1":{
      "filter":{"term":{"my_filter_field":"A"}},
      "aggs":{
        "metric1_value":{"value_count":{"field":"my_filter_field"}}
      }
    }

The canonical example in the docs is

    "aggs" : {
        "t_shirts" : {
            "filter" : { "term": { "type": "t-shirt" } },
            "aggs" : {
                "avg_price" : { "avg" : { "field" : "price" } }
            }
        }
    }

The solution is to find single bucket value aggs in AggregationToJsonProcessor and handle them as if they were leaf node aggregations.

[elasticmachine]

Pinging @elastic/ml-core

[benwtrent]

The solution is to find single bucket value aggs in AggregationToJsonProcessor and handle them as if they were leaf node aggregations.

I do not think that is necessarily correct.

"aggregations" : {
        "buckets" : {
          "date_histogram" : {
            "field" : "timestamp",
            "time_zone" : "UTC",
            "interval" : "15m"
          },
          "aggregations" : {
            "timestamp" : {
              "max" : {
                "field" : "timestamp"
              }
            },
            "ITBuckets" : {
              "filter": {
                    "term": {
                      "DestCountry": "IT"
                    }
                  },
                  "aggs": {
                    "sourceFlights": {
                      "terms": {
                        "field": "OriginCountry"
                      },
                      "aggs": {
                        "avgFlightMinutes": {
                          "avg": {
                            "field": "FlightTimeMin"
                          }
                        }
                      }
                    }
                  }
            }
          }
        }
      }

And having the job do analysis on avgFlightMinutes.

If we do not restrict the number of single bucket aggs that can be on the same level, it is conceivable that two "sister" aggs both have a multi-bucket sub aggregation.

So, I think we should restrict either:

• How many single bucket aggs are allowed at a level
• Restrict what sub-aggregations are allowed for single bucket aggs

What do you think @davidkyle ^^

[davidkyle]

Currently we limit the number of multi bucket aggregations in each level to 1 because if there were more than one how would we flatten that into a single document. Yes you are correct a single bucket agg should count towards that limit unless it doesn't have sub-aggregations could we make that the restriction?

https://github.com/elastic/x-pack-elasticsearch/blob/master/plugin/ml/src/main/java/org/elasticsearch/xpack/ml/datafeed/extractor/aggregation/AggregationToJsonProcessor.java#L116