Cardinality aggregation missing from response inconsistently

[noamkush]

Elasticsearch version: 7.9.1 (reproducible on demo.elastic.co)

Description of the problem including expected versus actual behavior:
The response to the exact same search request sometimes contains the cardinality aggregation and sometimes it doesn't - the issue is transient.

Steps to reproduce:
The following request (on demo.elastic.co):

POST _search?request_cache=false
{
  "aggs": {
    "date_range": {
      "aggs": {
        "unique_users": {
          "cardinality": {
            "field": "agent.name"
          }
        }
      },
      "date_range": {
        "field": "@timestamp",
        "ranges": [
          {
            "from": "2020-09-01T00:00:00.000000+00:00"
          }
        ]
      }
    }
  },
  "query": {
    "bool": {
      "must": [
        {
          "terms": {
            "agent.id": []
          }
        }
      ]
    }
  }
}

Results in:

{
  "took" : 962,
  "timed_out" : false,
  "_shards" : {
    "total" : 480,
    "successful" : 480,
    "skipped" : 477,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "date_range" : {
      "buckets" : [
        {
          "key" : "2020-09-01T00:00:00.000Z-*",
          "from" : 1.5989184E12,
          "from_as_string" : "2020-09-01T00:00:00.000Z",
          "doc_count" : 0,
          "unique_users" : {
            "value" : 0
          }
        }
      ]
    }
  }
}

And other times in

{
  "took" : 444,
  "timed_out" : false,
  "_shards" : {
    "total" : 480,
    "successful" : 480,
    "skipped" : 478,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "date_range" : {
      "buckets" : [
        {
          "key" : "2020-09-01T00:00:00.000Z-*",
          "from" : 1.5989184E12,
          "from_as_string" : "2020-09-01T00:00:00.000Z",
          "doc_count" : 0
        }
      ]
    }
  }
}

Notice the unique_users metric missing from the second response. It doesn't take more than a few tries to get both responses.

[elasticmachine]

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

[nik9000]

This sounds like something I may have broken when I was reworking the timing of aggs. And it may be fixed by #64016 which is an extension of a bug that I fixed that looked similar.

[nik9000]

I've created a local reproduction and confirmed that it still hits master. This happens when a few things line up:

1. Some shards map the field that you are running the range on and some do not.
2. You are querying enough shard to kick in the can_match phase (128).
3. Your query matches no docs and we recognize this during the can_match phase.
4. The shards that don't map the field for the range sort before those that do map the field for the range.

All of that causes us to run the "Range.Unmapped" special aggregator which mistakenly doesn't include its subAggs. I'll open a fix for it.

[nik9000]

It's a bit simpler than that, I just discovered: all you need is for the date_range to target an index that doesn't map the target field. That'll get you the same results. can_match comes up on demo.elastic.co because it has lots of shards and it makes the results more confusing because, well, some shards do map that field so you'd expect not to use the unmapped implementation. But can_match kicks in here to help and we confuse it with a bad aggregator impl.