Add `auto` interval to histogram AggConfig

[timroes]

Since the histogram aggConfig does already do a preflight request to determine min and max to later make sure we're not running over the histogram:maxBars setting, we can use that min/max information to easily allow an auto interval option.

The following steps need to be made:

• Allow auto as an allowed interval option (this shoul not require any changes by itself)
• Add an option autoIntervalTarget to the histogram AggConfig and the corresponding expression function, that takes a number or undefined. This parameter should default if not specified to the histogram:barTarget advanced setting.
• Adjust the logic of how the actual interval is calculated (see below)
• Add lots of tests

I think the way those parameters (interval, intervalBase, autoIntervalTarget) are playing together should be the following:

• If interval has a value !== auto everything should behave as beforehand:
  • If using that value would create too many buckets (higher than histogram:maxBars), increase the interval.
  • Always make sure if intervalBase is set, that the used interval is a multiple of intervalBase (this is required for rollups).
  • autoIntervalTarget has no effect if it's set as long as interval !== 'auto'
• If interval is set to auto do the following:
  • Use the max/min value (if they couldn't be loaded this request should fail altogether).
  • Check which interval we need to reach autoIntervalTarget buckets AND have a multiple of intervalBase (if set) AND be a more or less roundish value (so we're not ending up with intervals of 146.12). cc @wylieconlon @lukeelmers interested to hear your opinion on what reasonable roundings could be here. We have similar code for the maxInterval check, not sure if that's the logic we want to keep.
  • Make the actual request with that interval.

One question for me would be: What happens if the autoIntervalTarget is larger than the histogram:maxBars defined in advanced settings? I see two solutions:

• Either set autoIntervalTarget to histogram:maxBars if it exceeds it, or
• ignore histogram:maxBars in this case, since the "consumer" (e.g. the editor like Lens) explicitly set a higher value, but then again it would kind of render the histogram:maxBars useless, if it's not really limiting max bars.

Would also be curious about your thoughts on that?

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[lukeelmers]

@wylieconlon @lukeelmers interested to hear your opinion on what reasonable roundings could be here. We have similar code for the maxInterval check, not sure if that's the logic we want to keep.

Yeah my sense was that we could replicate the existing rounding logic to incorporate autoIntervalTarget

Either set autoIntervalTarget to histogram:maxBars if it exceeds it, or

IMHO this is the least confusing way to handle the case where autoIntervalTarget is larger. I think we risk introducing more confusion if this becomes some magic property that ignores the global advanced settings. I can imagine the questions coming in around "why is histogram:maxBars respected on some of my histogram visualizations but not others?"

But if this is a concern, another option would be combining both of your ideas -- give the user an option to explicitly override histogram:maxBars when they choose an autoIntervalTarget. This is arguably still a bit confusing/complex from a UX perspective, but at least the override is explicit instead of implicit.

Curious what @wylieconlon thinks on both of these questions as he will likely have some more concrete use cases for how they would affect the UX.

[wylieconlon]

We have historically mixed the user-experience part of "max buckets" with the cluster performance question, and these are really best handled separately. The number of bars a user wants to show in their visualization will depend on their exact analysis use case, and might even be the cluster maximum. But the hard maximum is not based on Kibana settings, it's determined by the cluster-level max buckets setting, which was raised to 65,000 recently. If the histogram is the only bucket agg, then we can request up to 65,000 buckets. If the histogram is combined with "Top 100 terms", then we can only request 6,500 buckets. If the histogram is combined with "Top 5 terms" and "Date histogram" with an expected 100 buckets, then we can only request 130 buckets for the histogram agg.

The user experience I'd expect is that the user would indicate the maximum number of bars they want to show. If they choose a number that is higher than we can actually use, we request that instead. So for example, if I indicate I want 1000 bars but that would crash the query, I'd see the maximum that we are comfortable querying (which might be histogram:maxBars).

Secondly, I think the way we should select auto intervals is based on powers of 10 with some adjustments. My favorite implementation is the Vega-Lite binning options, which defaults to powers of 10 that might be divided by 5 or 2 to create the final interval.

CPU Example

I'm choosing an example with fractional data because it's still powers of 10 even if they are negative powers. So for example, if I'm looking at CPU data and I want a histogram with up to 100 bars, the first thing we do is look at the difference between the maximum and minimum CPU value in the data. Let's say it's 0.5 minimum and 2.3 maximum. If we divide the difference between min and max into 100 buckets, we'd get 0.018, which is not a power of 10. 0.01 is, but that would create 180 buckets. The next higher power of 10 is 0.1, which would create 18 buckets. But if we divide this by 5 into increments of 0.02, we'd get 90 buckets.

Using the same logic, if I choose a maximum bucket count of 10, then the interval would be set to 0.2

Algorithm for determining auto-interval

1. Define maxBars as Math.min(<user input>, <max buckets setting>)
2. Find the min and max values in the data
3. Subtract the min from max to get diff
4. Set exactInterval to diff / maxBars
5. Based on exactInterval, find the power of 10 that's lower and higher
6. Find the number of expected buckets that lowerPower would create: diff / lowerPower
7. Find the number of expected buckets that higherPower would create: diff / higherPower
8. There are three possible final intervals, pick the one that's closest to maxBars:
   • The lower power of 10
   • The higher power of 10, divided by 5
   • The higher power of 10, divided by 2

[wylieconlon]

Edit: It looks like the scaling logic in the histogram is already able to create a type of auto histogram when the intervals are greater than 1, for example with bytes. The algorithm I described above does not make any assumptions about the intervals, so it is more generally applicable. For example, this is what happens when looking at CPU data on Kibana master:

Vs with the algorithm I described above:

[th0ger]

Welcoming this improvement.
While you're at it, consider if the Minimum interval can support values <1. This would be in order to support metric values all in the range, say, [0,1] or even smaller scales.