[ML] Transforms/DFA: Wizard performance degrades with indices with a lot of fields

[walterra]

With indices with a lot of fields (e.g. filebeat with 4000+ fields), performance of the transform wizard can significantly drop, resulting in page freezes of several seconds on each rerender. Some investigation is needed to identify root causes of the problem. One suspicion is that the dropdowns of the pivot configuration cause the freezes as the dropdown option might get recalculated on each render.

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[walterra]

Further investigation shows that this is not caused by the dropdown but the data grid.

Tested with a metricbeat index with 3500+ fields but just a small amount of documents. Loading the wizard takes 10+ seconds, with freezes of multiple seconds on every rerender that touches data grid.

Removing the data grid component for the source index gets the page back to a working state.

[walterra]

Proposed solution/workaround:

Similar to Discover, fetch a sample of e.g. 500 docs, find out which fields are populated/actually used in documents, and filter the list of available fields passed on to the data grid based on that. A toggle should allow the user to switch off that filtering. In Discover, this reduces the fields returned from 3500+ to ~125 based on the metricbeat index above.

This can be also used to filter the available options in the groupby/aggs-dropdowns.