Feature Request: Date Histogram agg bounds

[spalger]

Currently the date_histogram accepts an extended_bounds argument, but it would be great if it offered a bounds argument.

bounds and extended_bounds would be mutually exclusive, but operate similarly with two main differences:

1. bounds would effectively act like a filter and ignore any documents that did not fit into the specified time window
2. the bounds argument would continue to be an object with min and max properties, but could optionally have a target_buckets property. Specifying the target_buckets would automatically calculate an interval interval that cleanly dividing the specified bounds into about as many buckets as requested.

---

Here is an example request that would ideally return 52 weekly buckets. Note that a "clean" interval (1w) is chosen over an interval that would have provided exactly 50 buckets.

{
  "x-axis": {
    "date_histogram": {
      "field": "@timestamp",
      "bounds": {
        "min": 1388534400000, // 2014-01-01T00:00:00.000Z
        "max": 1420070399999, // 2014-12-31T23:59:59.999Z
        "target_buckets": 50
      }
    }
  }
}

[jpountz]

The advantage of today's extended_bounds is that it is something that we only check after the aggregation has been built to see if there are some blanks that we need to fill in with empty buckets.

On the other hand, a bounds option like you describe would require to check the values on every document in order to filter out those that are outside of the bounds. So it would essentially consist of putting the date_histogram inside of a filter aggregation. This does not sound right to me given that the initial goal of aggs was to decouple facets (eg. the term-stats facet can now be achieved with a terms and a stats agg) while this use-case would require to go the other way and put a filter agg inside of a date_histogram agg.

I'm curious about the target_buckets idea however. It's an idea that comes back regularly. In your particular case, would you be happy with any interval or would it ideally need to be rounded (ie. day, week or month). Implementation-wise, it is something that we might be able to do by deferring the execution of the histogram aggregation and taking advantage of the first pass in order to figure out the minimum and maximum values of the field in order to compute an interval that would generate the desired number of buckets. A limitation is that this would be computed per shard, but this would probably be good enough in most cases?

[spalger]

Having nicely rounded buckets is pretty important.

Ideally the time window used to calculate these nicely rounded buckets would be something we could specify (either through a combination of a filter and extended_bounds or by specifying it right in the aggregation). I worry how it would work if we just used phase 1 to calculate the extent of the data and based the buckets on that.

[jpountz]

Maybe it could work in such a way that you can provide several intervals in increasing duration, eg. [ "hour", "day", "month", "year"] and it would pick the first one which has less than target buckets?

Now that I'm reading my previous message, I realize that I forgot to think about reconciling shards, ie what happens if two shards pick a different interval. It could be quite tricky... Maybe an easier way to implement this feature would be for it to just be a reducer (ie. buckets would be computed with an hourly or daily interval and we would later reduce it to a daily interval if there are too many buckets).

[spalger]

Passing the intervals might work, but we would want them to be chosen based on how close they are to the target, not just if they are more or less than it. In my example 1w technically produces 2 extra buckets, but is much closer to the likely next interval of 1d.

Regarding calculating the interval per shard, that is what I imagined we would need to pass the expected time range. Would reducers be able to scale the buckets regardless of the trees underneath?

[bleskes]

@spenceralger if I understand correctly, the goal here is to take an arbitrary time period specified by a user using a time picker and translate it to a round number of buckets to be displayed where you have some variance in the number buckets as long as it is somewhere in there area of what's visually appealing/ can be drawn by a browser. I would also guess (but correct me if I'm wrong) that you would always like the shown period to be the extension of the request one (so everything the user asked for is displayed). Would also require the buckets to always snap into a fixed grid? I.e., months are always complete like jan, feb, march etc. but not 5 jan to 5 feb, 5 feb to 5 march etc. Do you also want the to always "snap into" the given buckets. If we choose the fixed grid option, this is important in order to make sure that there are no artificially small buckets because the date a range doesn't cover them completely. Think displaying data by months when the user chose 28 of january till 15 of november. If we only count the data explicitly in this range, january and november will have artificially lower bars when compared to the months february and march which are fully in range.

Depending on the choices made we might opt for two building blocks here. The first is a filter called, say, auto_expand_date_range (name to be compressed) which take a minimal range and expands it to the smallest unit which gives at most the request buckets (or the closest to it) but then also expands it to a whole unit. This can work together with an automatic interval selection for the data histogram using the similar logic and parameters (but doesn't need to worry about the actually data, just the interval selection). The filter then gives in an parent agg or in the query, depending on what you want to achieve.

[spalger]

@bleskes You bring up some great points. I think that the expected behavior would be for "monthly" buckets to produce buckets for January, February, etc. We love this behavior, but as you mention this causes partial buckets to be created at the edge of the time range. The partial buckets behavior is what spurred this train of thought and lead to this request.

At this point though I think my request is invalid. This API addition really isn't going to solve the partial buckets problem any better than we can and are.

[spalger]

That said, we are still interested in the ability to ask for a specific number of buckets from any histogram. We can't currently provide this feature without knowing the extents of the data. This should be a separate request though, right?

[bleskes]

@spenceralger yeah, maybe a separate issue is better. The tricky part is what @jpountz already mentioned - no single shard has a complete picture of the data. Say the data on one shard fits in 30 day buckets or 4 week buckets (probably meaning we want the data by days). On another shard it may be that we have 40 day buckets and 6 week buckets which may tilt the decision to a bucket by week. These will be impossible to merge in the reduce phase.

[jpountz]

We just discussed this issue, here is a summary of the conversation:

• Computing the interval based on a target number of buckets is very hard because of the sharding of the data
• Having the bounds reducing the scope of the date_histogram aggregation would be more user-friendly than what we have today which forces users to add a filter on top of the date_histogram aggregation.
• The bounds would apply to the generated buckets, not to the values. So if the interval is month and the min bucket is 2012-04-23T18:25:43.511Z, then all documents after 2012-04-01T00:00:00.000Z would be considered. This needs to be documented carefully.
• Then having the date_histogram applying the filter would bring value since it would do it in a way which would be aware of the value of interval (which is otherwise hard to do from client side).
• If you want to filter based on values and not buckets, then the recommandation to use a filter on top of the date_histogram aggregation still applies.
• We could potentially have a more optimized impl of the histogram aggregation when bounds are specified by pre-allocating the buckets, which we cannot do in the general case.

[clintongormley]

Moving discussion of the target_buckets part of this issue to #9572

[colings86]

The target_buckets part of this issue has been moved to a different issue and thebounds` part of this issue can be implemented using a parent filter aggregation and we can't do any better than that in the histogram aggregation itself