Integrate rollup search with _search endpoint

[csoulios]

Currently searching in rollups is facilitated by a specific endpoint named _rollup_search. For seamless integration of rollups we should remove this endpoint and implement its functionality in the _search endpoint.

When running aggregations using the existing _search endpoint we must implement the following:

• Merge rollup buckets with regular buckets.
• Ensure that partial buckets can merge across shards (e.g. when rollover is not on a clean interval).
• Modify existing aggregators to pull doc_count from rollup indices, rather than just incrementing. This is needed so a "rollup metric" can provide doc counts for avg and bucketing aggregations, by allowing the aggregator framework to treat a single document as if it were multiple docs.

Producing rollups is discussed in #48003

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Rollup)

[timroes]

💯 This would be very awesome and ease a lot of code in Kibana, if we would have the integration in the regular _search endpoint.

[lizozom]

@csoulios is there a target version for this change?

Anyway, please ping @elastic/kibana-app-arch when this is being implemented, so we can go ahead an implement elastic/kibana#64873. 🙏

[tangpf877]

Is there any news about rollup? I am wandering when _rollup_search endpoint will be removed, and what the ILM side will be like in the future? Does 'Create an ILM rollup action' means rollup will proceed automatically and we needn't start an rollup job? Thanks for reply

[csoulios]

Hi @benjam1n77, rollups refactoring is a task we are actively working on. You can track progress and related discussions in this ticket: #42720

When it comes to ILM and rollups, the goal is to stop having rollups as continues jobs. Instead rollup will be an action in the ILM configuration. See the relevant ticket for more: #48003