[Lens] Fields which contain periods in _source don't show up in field list

[morgango]

Kibana version:

7.6.1

Elasticsearch version:

7.6.1

Server OS version:

Uknnown

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):

Elastic Cloud

Describe the bug:

If I create a field name with a dot in them (a '.' character), it will not show up by default in Lens. It will show up if you remove the dot.

I am loading data through Eland, so I can't turn the data into a JSON object in order to make it conform. However, I would like to conform to ECS, and that means dots in the names.

Steps to reproduce:

1. Create an index with field of any type with a period in it. For example, test.field
2. Create an index pattern.
3. Open the index pattern in lens

POST covid*/_search
{ "docvalue_fields": ["geo.country_name"] }

returns

    "hits" : [
      {
        "_index" : "covid_time_series",
        "_type" : "_doc",
        "_id" : "31500",
        "_score" : 1.0,
        "_source" : {
          "case_type" : "Confirmed",
          "cases" : 0,
          "difference" : 0,
          "geo.country_name" : "US",
          "geo.region_name" : "Oregon",
          "geo.admin_name" : "Grant",
          "geo.fips_code" : "41023",
          "@timestamp" : "2020-03-03T00:00:00",
          "geo.coordinates" : [
            -119.0068631,
            44.49024544
          ],
          "url.original" : "https://query.data.world/s/fhfyt2uyvekfpnalnm4fazz4bafhav"
        },
        "fields" : {
          "geo.country_name" : [
            "US"
          ]
        }
      },

Expected behavior:

There are data in these fields, they should show up by default.

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:

Thank you! Lens is very cool and I look forward to using it in my webinars.

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)

[wylieconlon]

Thanks for writing this up! I'm pretty sure this is a bug in our existence code where we are using path.split('.') to look up values from _source.

https://github.com/elastic/kibana/blob/master/x-pack/plugins/lens/server/routes/existing_fields.ts

There are two fixes that I am considering for this issue, which I'm writing here for when we start working on the fix:

1. Work around this issue by changing the exists function in existing_fields.ts to handle dots in the field name
2. Wait for Elasticsearch to change the way that docvalue_fields can be requested, which they are currently planning. A high level way of retrieving values for certain fields elasticsearch#49028

Most likely the code change will happen in our Lens implementation of this because we can do this faster.