Strange behaviour of `fields` API with mixed object notation

[cbuescher]

It appears we can currently index and retrieve documents with mixed object notation like this:

PUT my-index-000001/_doc/1
{
  "obj.products" : { "manufacturer" : "Supersoft" } ,
  "obj" : {
    "products" : 
       { "manufacturer" : "HyperSmart" }
  }
}

When searching on field obj.products.manufacturer we get a hit for both terms "Supersoft" and "HyperSmart".
However, the fields API output only returns second value:

POST /my-index-000001/_search
{
  "query": {
            "match": {
              "obj.products.manufacturer": "Supersoft"
            }
    },
  "_source" : false,
  "fields" : ["*"]
}

returns

       "fields" : {
          "obj.products.manufacturer.keyword" : [
            "HyperSmart"
          ],
          "obj.products.manufacturer" : [
            "HyperSmart"
          ]
        }

The same output is produced when searching for "HyperSmart". Tested on 7.10.0.

[elasticmachine]

Pinging @elastic/es-search (Team:Search)

[javanna]

@cbuescher have you checked the mappings for the created index? I would hope that there is one obj.products in there, in the object form.

[cbuescher]

@javanna the mapping looks fine to me:

{
  "my-index-000001" : {
    "mappings" : {
      "properties" : {
        "obj" : {
          "properties" : {
            "products" : {
              "properties" : {
                "manufacturer" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

[javanna]

Cool, sounds like the fields API should return both values then?

[romseygeek]

Possibly related to #65539

[romseygeek]

Definitely related! This is actually something that @jtibshirani called out when reviewing #65539 - I had fixed this exact case for XContentMapValues.extractRawValues() but not for XContentMapValues.extractValue(), and SourceValueFetcher uses the latter. I wonder if we can just use the former instead, which would also allow us to remove the complicated array nesting logic in SourceValueFetcher, or if that will break something else?

[jtibshirani]

Thanks @cbuescher for filing this, I forgot to open a follow-up about it after reviewing #65539.

I wonder if we can just use the former instead, which would also allow us to remove the complicated array nesting logic in SourceValueFetcher, or if that will break something else?

We didn't use XContentMapValues.extractRawValues() because it won't return object values. For example, if you have a geopoint structured as an object with lat and lon fields, then that method won't be able to extract it.