[Meta] Aggregations Roadmap

[lukeelmers]

Summary

Kibana's shared aggregations infrastructure lives inside of the data plugin's search service (plugins.data.search.aggs), and is owned by @elastic/kibana-app-arch. This infrastructure exists to provide a way of modeling aggregations for use in a UI, and then converting them to Elasticsearch DSL. An example of a consumer of this service is the default visualizations editor, which provides the UI for configuring most of Kibana's core visualizations.

The aggs service also works in conjunction with SearchSource, allowing consumers to push their modeled agg (AggConfig) to SearchSource, and have it automatically converted to DSL for them when a query is executed.

The main components of the aggs service are:

1. Classes to model each of the different ES aggregations (AggTypes) supported in Kibana, and the different params they accept as configuration.
2. A types registry which exposes each of the available agg types to applications.
3. A static AggConfig class, which is a model representing an individual aggregation, including its type, & params used to configure that type.
4. A createAggConfigs method, which allows you to create a collection of AggConfigs that can ultimately be configured and sent to Elasticsearch.

This meta issue exists to track ongoing planned work for this service. For an overview of currently supported aggs in Kibana, please refer to the Kibana ES aggs support meta issue.

Remaining migration tasks

• Remove legacy static exports from the public contract
• Remove __LEGACY runtime contracts
• Expression functions for accessing & configuring agg types [data.search.aggs]: Expression functions for bucket agg types  #63762 [data.search.aggs]: Expression functions for metric agg types #63760

Refactoring

• Remove createFilter from agg types and instead have each agg return values for the filter
  • For ranges this should be straightforward; have a method to return from and to for a given agg
  • Terms agg which also uses createFilter is more complex and would require further discussion
• Expression functions for object-based agg params
• Unify configStates interface with the AggConfig.toJSON() output
• Clean up AggConfig

Features & Enhancements

General

• Aggs available on the server
  • Clean up TimeBuckets
  • Remove service getters from agg types
  • Remove field formats dependency from AggConfig/AggConfigs
• Deduplicate aggs in DSL in pipeline aggregations

esaggs

• make esaggs available on the server
• Move 'other' bucket and inspector integrations to SearchSource
• Refactor & export buildTabularInspectorData from data plugin contract
• Ability to attach filters to individual metrics
  • could be done either in AggConfigs or esaggs
• Add time offset argument to esaggs
• implicitly add Count metric when only a bucket agg is provided
• Provide root-level metrics for metricsAtAllLevels

Agg-specific features

Bucket

• Adjacency Matrix Aggregation
• Auto-interval Date Histogram Aggregation
  • Not planned as we already support this via a custom Kibana implementation. See [Meta] Kibana support for ES aggregations #58628
• Children Aggregation
• Composite aggregation
• Date histogram aggregation
• Date Range Aggregation
• Diversified Sampler Aggregation
• Filter Aggregation
• Filters Aggregation
• Geo Distance Aggregation
• GeoHash grid Aggregation
• GeoTile Grid Aggregation
• Global Aggregation
  • Currently there is no generic use-case to support this in Kibana. See [Meta] Kibana support for ES aggregations #58628
• Histogram Aggregation
• IP Range Aggregation
• Missing Aggregation
• Nested Aggregation
• Parent Aggregation
• Range Aggregation
• Rare Terms Aggregation*
• Reverse nested Aggregation
• Sampler Aggregation
• Significant Terms Aggregation
• Significant Text Aggregation*
• Terms Aggregation
  • Use rare terms agg internally for sorting Terms by Count

Metrics

• Avg Aggregation
• Weighted Avg Aggregation*
• Boxplot Aggregation*
• Cardinality Aggregation
• Stats Aggregation
• Extended Stats Aggregation*
• Geo Bounds Aggregation
• Geo Centroid Aggregation
• Max Aggregation
• Min Aggregation
• Median Absolute Deviation Aggregation*
• Percentiles Aggregation
  • Add support for HDR Histogram & Compression params
• Percentile Ranks Aggregation
• Scripted Metric Aggregation
  • Currently there is no plan to introduce this in Kibana. See Support Scripted Metrics Aggregation #2646 (comment)
• String Stats Aggregation*
• Sum Aggregation
• Top Hits Aggregation
• Top Metrics Aggregation
• Value Count Aggregation*

Matrix

• Matrix Stats*

Pipeline

• Bucket Script Aggregation*
• Bucket Selector Aggregation
• Bucket Sort Aggregation
• Avg Bucket Aggregation
• Max Bucket Aggregation
• Min Bucket Aggregation
• Sum Bucket Aggregation
• Cumulative Cardinality Aggregation
• Cumulative Sum Aggregation
  • Allow an option for getting the initial_value for cumulative sum agg
• Derivative Aggregation
• Normalize
• Percentiles Bucket Aggregation*
• Moving Average Aggregation
• Moving Function Aggregation
• Moving Percentiles Aggregation
• Serial Differencing Aggregation
• Stats Bucket Aggregation*
• Extended Stats Bucket Aggregation*

* potential low-hanging fruit

Related

• Kibana ES aggs support meta issue