Reject invalid `reverse_nested` aggs

[nik9000]

Elasticsearch Version

all

Installed Plugins

No response

Java Version

bundled

OS Version

all

Problem Description

We currently accept reverse_nested aggregations even when the configuration is "unreversed." Check this out:

This sequence of aggs is invalid:

* nested (path = a)
  * nested (path = a.b)
    * reverse_nested (path = a)
      * reverse_nested (path = a.b)

The second reverse_nested presumes that it's in a unnesting from a child of a.b but the first reverse_nested has made the documents a parent of a.b. This fails assertion errors while collecting. If assertions are enabled. And has done so at least back to 7.17. If assertion are not enabled it fails with a "read past EOF" error.

Steps to Reproduce

password="password"
http=http

curl -k -s -uelastic:$password -XDELETE ${http}://localhost:9200/test
curl -k -s -uelastic:$password -XPUT -HContent-Type:application/json ${http}://localhost:9200/test -d'{
  "mappings": {
    "dynamic": "false",
    "properties": {
      "n1": {
        "type": "nested",
        "properties": {
          "n2": {
            "type": "nested"
          }
        }
      }
    }
  }        
}'


curl -k -s -uelastic:$password -XPOST -HContent-Type:application/json ${http}://localhost:9200/test/_doc?pretty -d'{
  "n1": {
    "n2": {
      "rating": 1
    }
  }
}'
curl -k -s -uelastic:$password -XGET ${http}://localhost:9200/_refresh

curl -k -s -uelastic:$password -XPOST -HContent-Type:application/json ${http}://localhost:9200/test/_search?pretty -d'
{
  "size": 0,
  "aggregations": {
    "n1": {
      "nested": { "path": "n1" },
      "aggregations": {
        "n2": {
          "nested": { "path": "n1.n2" },
          "aggregations": {
            "r1": {
              "reverse_nested": { "path": "n1" },
              "aggregations": {
                "r2": {
                  "reverse_nested": { "path": "n1.n2" }
                }
              }
            }
          }
        }
      }
    }
  }
}'

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[dnhatn]

I think the innermost aggregation (r2) in the reproduction line should be reverse_nested instead of nested. I understand the purpose of reverse_nested and how it works: its intention is to allow collecting ancestor documents in the hierarchy. In the example, n1.n2 is not in the ancestry hierarchy of n1, so this query should be rejected. It appears we missed this validation in the original implementation in #5507. The documentation in #5507 states that the path must be within the hierarchy:

The `reverse_nested` aggregation has a single option:
* `path` - Which defines to what nested object field should be joined back. The default is empty, which means that it
joins back to the root / main document level. The path cannot contain a reference to a nested object field that falls
outside the nested structure a `nested` and `reverse_nested` is in.

[dnhatn]

The first query should be invalid because reverse_nested with the path n1.n2 does not belong the hierarchy of n1:

curl -k -s -uelastic:$password -XPOST -HContent-Type:application/json ${http}://localhost:9200/test/_search?pretty -d'
{
  "size": 0,
  "aggregations": {
    "n1": {
      "nested": { "path": "n1" },
      "aggregations": {
        "n2": {
          "nested": { "path": "n1.n2" },
          "aggregations": {
            "r1": {
              "reverse_nested": { "path": "n1" },
              "aggregations": {
                "r2": {
                  "reverse_nested": { "path": "n1.n2" }
                }
              }
            }
          }
        }
      }
    }
  }
}'

But the second query should be valid because reverse_nested with the path n1 belongs the hierarchy of n1.n2:

curl -k -s -uelastic:$password -XPOST -HContent-Type:application/json ${http}://localhost:9200/test/_search?pretty -d'
{
  "size": 0,
  "aggregations": {
    "n1": {
      "nested": { "path": "n1" },
      "aggregations": {
        "n2": {
          "nested": { "path": "n1.n2" },
          "aggregations": {
            "r1": {
              "reverse_nested": { "path": "n1.n2" },
              "aggregations": {
                "r2": {
                  "reverse_nested": { "path": "n1" }
                }
              }
            }
          }
        }
      }
    }
  }
}'

[nik9000]

I think the innermost aggregation (r2) in the reproduction line should be reverse_nested instead of nested.

Right! Sorry. I had copy and pasted the wrong bits.

[nik9000]

But the second query should be valid because reverse_nested with the path n1 belongs the hierarchy of n1.n2:

Indeed! That one should be fine.