“fix#829-PreAgg-field_doc_count”

[cwillum]

Signed-off-by: cwillum cwmmoore@amazon.com

Fixes #829

Description

Add documentation to Bucket Aggregation describing the use of the _doc_count field for computing documents that store pre-aggregated data.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[cwillum]

@petardz Thanks for your input on documentation for this issue. Could you review this content for technical accuracy? Thanks.

[kolchfa-aws]

Can we rephrase this to simplify?
While the doc_count field represents the number of individual documents aggregated in a bucket, the field by itself does not account for documents that store pre-aggregated data, such as histogram.

[kolchfa-aws]

Also,
When a document includes the _doc_count field, all bucket aggregations increase the bucket doc_count by the value of _doc_count.

[Naarcha-AWS]

@petardz Thanks for your input on documentation for this issue. Could you review this content for technical accuracy? Will this backported to v1.3, v.2.0, and v2.1? Thanks.

@cwillum: Based on the conversation from this PR(opensearch-project/OpenSearch#3985) it seems like this code wasn't included in OpenSearch 1.0. Furthermore, the PR only contains a "backport 2.x" label, meaning that the change will be backported to the latest minor version of OpenSearch. We shouldn't need to backport on this one. Once the PR is approved and gone through editorial review, add the "5- Done and waiting to merge" label to this PR.

[petardz]

@cwillum I'm sorry, link with example I provided earlier is not compatible with opensearch. Here is a one synthetic example:

1. Create index:

PUT my_index
{
  "settings": {
    "number_of_replicas": 0
  }, 
  "mappings" : {
    "properties" : {
      "str" : {
        "type" : "keyword"
      },
      "number" : {
        "type" : "integer"
      }
    }
  }
}

2. Add few documents:

POST my_index/_doc
{
  "_doc_count": 10,
  "str": "abc",
  "number" : 500
}

POST my_index/_doc
{
  "_doc_count": 5,
  "str": "xyz",
  "number" : 100
}

POST my_index/_doc
{
  "_doc_count": 7,
  "str": "foo",
  "number" : 100
}

4. Perform aggregations

POST my_index/_search
{ 
  "size" : 0,
  "aggs" : { 
    "num_terms" : { 
      "terms" : { 
        "field" : "number" 
      }
    }
  }
}

Response:


{
  "took" : 2,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "num_terms" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : 100,
          "doc_count" : 12
        },
        {
          "key" : 500,
          "doc_count" : 10
        }
      ]
    }
  }
}

Notice how _doc_count was used when calculating doc_count of buckets

[kolchfa-aws]

@cwillum: please use the following example:

PUT /my_index/_doc/1
{
  "response_code": 404,
  "date":"2022-08-05",
  "_doc_count": 20
}

PUT /my_index/_doc/2
{
  "response_code": 404,
  "date":"2022-08-06",
  "_doc_count": 10
}

PUT /my_index/_doc/3
{
  "response_code": 200,
  "date":"2022-08-06",
  "_doc_count": 300
}

GET /my_index/_search
{
  "size": 0,
  "aggs": {
    "response_codes": {
      "terms": {
        "field" : "response_code"
      }
    }
  }
}

Response:

{
  "took" : 20,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 3,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "response_codes" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : 200,
          "doc_count" : 300
        },
        {
          "key" : 404,
          "doc_count" : 30
        }
      ]
    }
  }
}

[kolchfa-aws]

LGTM

[natebower]

@cwillum Just one comment for you. Let me know if you have any questions. Thanks!

[natebower]

Change "To get a better sense for" to "For information on" (or something similar). Change "Index Management" to "Index Management (IM)".