General functionality test coverage improvements

[ppf2]

I apologize in advance if this is an overly generalized test request :) We all ❤️ Kibana, but we also like to see more comprehensive test coverage between releases.

Would like to see testing cover all available features:

• Come up with a whole set of comprehensive dashboards, saved searches, visualizations (eg. every visualization type, and every option that is available for each visualization type), index patterns and settings, etc.. that will be tested every single release.
• Also test all Kibana dashboard examples we provide out of the box for other products like beats, watcher, etc ..
• All available “actions” (eg. export, import, etc..) the user can perform from the UI.

Will help prevent upgrade/regression issues like the following moving forward.

• [Fixed in 4.6.1] Unable to order aggs by _term after upgrade to 4.6.0 #8141
• Kibana Crashing on a regular basis after 4.4 upgrade #6174
• Kibana 4.3 "Export Everything" exception #5524

[Bargs]

I don't think this is really an actionable ticket, it's basically "test all the things" :)

We're adding tests all the time (mostly thanks to @LeeDr) but it's impossible to test every possible permutation of features/options/settings/data. I'm going to close this since there's really no way to mark it "resolved" without a proper end goal in sight. Some of these are interesting points that could be tickets in their own right though. For instance I'd like to see a discuss ticket about how to test Beats dashboards.

[LeeDr]

@ppf2 I did add a new test after #8141 was discovered. So we're covered on that one.

We do already have automated UI tests which create at least one of each type of visualization, but each only has one bucket aggregation and they don't exercise every option. And we can't really see if the map is rendering on a Tile Map. The frameworks is there for these tests so we could certainly add more. But there's a cost in the time for them to run, and UI tests can be a pain to maintain.

We do have a couple of automated tests for Kibana dashboards that check the name, size, and position of the panels, but not the data on them.

The full-stack integration test does verify that beats install, run, and at least one visualization from each one has data in it. But further testing of all the stuff that gets loaded should be tested as part of the beats CI process.

There are not currently any UI tests (I'm not sure on unit tests) for Import/Export. We might be able to add some. But the file open and file save dialogs of the OS can't be automated very easily.