bucket_script and gap_policy

[jcornez]

Here is a simple and complete case. I tested against ES 7.0.1

PUT bucket_scipt_test

POST bucket_scipt_test/_doc/1
{
  "timestamp_utc": "2019-05-01",
  "foo": 1
}

POST bucket_scipt_test/_doc/3
{
  "timestamp_utc": "2019-05-03",
  "foo": 3
}

POST bucket_scipt_test/_search
{
  "size": 0,
  "aggs": {
    "days": {
      "date_histogram": {
        "min_doc_count": 0, 
        "field": "timestamp_utc",
        "interval": "1d"
      },
      "aggs": {
        "foo_script": {
          "bucket_script": {
            "buckets_path": "_count", 
            "script": "1"
        }},
        "copy": {
          "bucket_script": {
            "buckets_path": {"foo":"foo_script"},
            "script": "params.foo",
            "gap_policy": "insert_zeros"
        }},
        "sum": {
          "cumulative_sum": {
            "buckets_path": "foo_script"
        }}
    }}
  },
  "query": {
    "match_all": {}
  }
}

And the results I see from the search are

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 2,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "days" : {
      "buckets" : [
        {
          "key_as_string" : "2019-05-01T00:00:00.000Z",
          "key" : 1556668800000,
          "doc_count" : 1,
          "foo_script" : {
            "value" : 1.0
          },
          "copy" : {
            "value" : 1.0
          },
          "sum" : {
            "value" : 1.0
          }
        },
        {
          "key_as_string" : "2019-05-02T00:00:00.000Z",
          "key" : 1556755200000,
          "doc_count" : 0,
          "foo_script" : {
            "value" : 1.0
          },
          "copy" : {
            "value" : 0.0
          },
          "sum" : {
            "value" : 1.0
          }
        },
        {
          "key_as_string" : "2019-05-03T00:00:00.000Z",
          "key" : 1556841600000,
          "doc_count" : 1,
          "foo_script" : {
            "value" : 1.0
          },
          "copy" : {
            "value" : 1.0
          },
          "sum" : {
            "value" : 2.0
          }
        }
      ]
    }
  }
}

The problem I see is that gap_policy can only be insert_zeros or skip. But with a bucket_script, I really want the script to just run as normal, even when the doc_count in the bucket is zero.

Specifically, I expect the "copy.value" in the second bucket to be 1.0 and the "sum.value" to be 2.0. In the third bucket, I expect the "sum.value" to be 3.0.

Thank you for your consideration.

[elasticmachine]

Pinging @elastic/es-analytics-geo

[polyfractal]

Agreed! We actually have an issue open here tracking this general issue. We chatted about it in the past and decided that bucket_sort really shouldn't accept gap_policy, since it doesn't make sense at all for this agg.

There's a community PR open, but we ran into some backwards compatibility issues that need resolved (not overly difficult, just some work to make sure it doesn't break existing users). We should revisit on that PR, or see if one of the ES devs should take it over and do the BWC work.

I'm going to close this in favor of #36487, just so we have a single tracking location. But I'm 100% agreed this is something that we should get fixed. Sorry you ran into this behavior!

[jcornez]

Fair enough. Meantime, do you know if there is any workaround?

[polyfractal]

Oh wow, I completely misread the agg in this ticket; bucket_script not bucket_sort. Sorry about that! I was on the road last week and my brain was a bit frazzled. Thanks for leaving a note on the ticket about bucket_script.

Unfortunately, I don't think there's any workaround here. The bucket_script is interpreting doc_count: 0 as an "empty" bucket and applying the gap policy. Theoretically setting min_doc_count: 1 would fix it... but pipeline aggs require that the min doc count is zero so that's not an option.

This actually relates to #27377 more than #36487, now that I'm reading closer. There are various issues with the pipeline aggs which all revolve around empty bucket handling.

Essentially, we need to refactor the framework to either wrap values in Optionals (so that each new agg can see if the value is "missing" vs "null"), or pass both value and doc_count to pipeline aggs so that they can make better decisions. And in the case of a script, the user would be able to decide what to do.

It's on our radar, and I'm hoping to get to it soon, but I'm not sure there is a workaround at this time :(

[jcornez]

Thank you. Just to note that I do want the bucket with doc_count: 0. I'm working with values from siblings that are computed by moving_fn and so even though this bucket might be empty, a moving_fn can indeed have a value here.