[Reporting] Not able to generate CSV report on object fields; Discover is able

[tsullivan]

Kibana version: First reported with 6.2.2, reproducible in 6.4.2

Describe the bug:

• Similar and possibly more general than CSV reports fail with no error if a saved search has only a single array property #23611

Steps to reproduce:

• Create an object field, either using repro from the bug above, or mapping a field as disabled
• CSV report will fail with error field [fieldName] isn't a leaf field

Representing the field value works in Discover, presumably because Discover has a request body that uses the source option. CSV generation fails, presumably because the request body uses stored_fields in the request body.

Asking for stored fields is currently potentially problematic because not all fields have stored fields. To quote @colings86

Asking for stored fields on an object field itself will never work because you can't enable a stored field at that level hence why we throw an error.

[tsullivan]

It looks like Discover uses a completely different search (_msearch) to render the table and doesn't make any filters on which fields to retrieve:

POST /_msearch
{"index":"test_index","ignore_unavailable":true,"timeout":30000,"preference":1541460732434}
{"version":true,"size":500,"sort":[{"_score":{"order":"desc"}}],"_source":{"excludes":[]},"stored_fields":["*"],"script_fields":{},"docvalue_fields":[],"query":{"bool":{"must":[{"match_all":{}}],"filter":[],"should":[],"must_not":[]}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"*":{}},"fragment_size":2147483647}}

That means that the columns are shown/hidden possibly in some part of the browser memory, getting intialized from the visState data for saved searches. That would not work for CSV export.

CSV export gets the its search request body passed from the Discover app (the getSharingData() method), which [gets it from the courier SearchSource](: https://github.com/elastic/kibana/blob/d7da13b/src/core_plugins/kibana/public/discover/controllers/discover.js#L366)

Here's an example of the search request body generated for a test index, when the Discover table only has the goofy field picked:

POST /test_index/_search
{
  "stored_fields": [
    "goofy"
  ],
  "query": {
    "bool": {
      "filter": [],
      "must_not": [],
      "should": [],
      "must": [
        {
          "match_all": {}
        }
      ]
    }
  },
  "script_fields": {},
  "_source": {
    "excludes": [],
    "includes": [
      "goofy"
    ]
  },
  "docvalue_fields": [],
  "sort": [
    {
      "_score": {
        "order": "desc"
      }
    }
  ],
  "version": true
}

The bad part is the stored_fields. Just removing it fixes the problem, since goofy is already in the _source.includes part.

Really ugly workaround: remove the stored_fields part after copying the POST URL. Example, where the part to remove is highlighted:

[tsullivan]

@gammon @kobelb @elastic/kibana-discovery I've been going back where stored_fields is used and it looks like it goes way, way back in git history. Any thoughts about the impact of removing it from

• https://github.com/elastic/kibana/blob/master/src/ui/public/courier/search_source/search_source.js#L585
• https://github.com/elastic/kibana/blob/master/src/ui/public/courier/search_source/search_source.js#L609

It seems like it's enough to just use _source.includes in this context, and that's already getting added to the request.

[kobelb]

I unfortunately don't have any memory of this aspect of CSV reporting, so I'm afraid I won't be much help here.

[Bargs]

I don't know what all the ramifications of removing it would be, but we do present stored fields to the user so removing it would likely be a breaking change.

[tsullivan]

@Bargs is there a way to test the code you linked against data with an object field, like the kind described in #23611 ?

[elasticmachine]

Pinging @elastic/kibana-app

[Bargs]

@tsullivan maybe if one of the sub-fields in the object is marked as stored:true in the mappings.

[tsullivan]

@joelgriffith this should go into the test plan for #34571