Rollup : add "count document" aggregation metric

[adityaj1107]

Issue by pmarjou22
Thursday Jan 14, 2021 at 09:31 GMT
Originally opened as opendistro-for-elasticsearch/index-management#387

---

I'm currently using Rollup on 1.12.0, and I have to create a dummy field on all my documents to be able to count the number of documents from source index.

It would be nice to be able to add this aggregation directly from RollUp configuration without this workaround

Current Workaound with event.count set to 1 on all documents

Source Index
POST sourceindex/_doc/
{
"event.name" : "event1",
"event.category" : "category1",
"event.count" : 1,
"date": "2015-01-01T12:10:30Z"
}

POST sourceindex/_doc
{
"event.name" : "event2",
"event.category" : "category1",
"event.count" : 1,
"date": "2015-01-01T12:10:30Z"
}

POST sourceindex/_doc
{
"event.name" : "event3",
"event.category" : "category2",
"event.count" : 1,
"date": "2015-01-01T12:10:30Z"
}

Target Index workaround : Sum on event.count

[dbbaughe]

We had an open issue from Elasticsearch that was tracking an enhancement that would have solved this.
Essentially the bucket collector is calculating the documents as +1 each when counting up the doc_counts, and there was an enhancement to allow a field to be used such as _doc_count or something that would be used instead if it existed.
I believe it was targeted for 8.0 though and obviously we've done a hard fork at 7.10 so we need to look into adding the similar enhancement into upstream core. That way the rollup response will include the doc_count as expected to the user instead of the rolled up document counts.

[downsrob]

For collecting the number of documents from the source index, the index management code base was prepped for a change in core which would make use of the _doc_counts metadata field to allow for seamlessly aggregating summary documents as we do in rollups. This core change did not come, but the change really belongs in core. This issue is open in the core repo here:
opensearch-project/OpenSearch#3712
We should pick this up as the solution to aggregating document counts from summary docs.