Meta-issue: Improving testing for code examples

[lcawl]

The code snippets in the documentation in the elasticsearch repo are tested as described in https://github.com/elastic/elasticsearch/tree/master/docs#snippet-testing

This meta-issue tracks out-standing issues with that functionality in the hopes that it can be fixed and rolled out to other repos as well:

1. Some APIs return “xxx doesn’t exist in the rest spec” errors when you try to use them in documentation tests: Missing rest specs for testing features' code snippets elasticsearch#48663

2. Pages that contain non-OSS-only APIs must indicate which license to use for testing ([testenv="platinum”]), which is a level of detail we’ve generally tried to omit from the documentation. Ideally, we would run all tests against the same environment and license level and leave edge cases for other types of tests: Synchronize license used for testing all documentation pages elasticsearch#51566

3. Tests that take time to generate results (e.g. ML analytics, forecasts) fail too soon or hit race conditions depending on the speed of the machine. Ideally, we could run tests against a system that has had time to set up and process data so that we get meaningful and consistent results: Testing documentation examples that require time to process their results elasticsearch#51572

4. Cleanup failures. There seem to be structures created during testing (system indices, etc) that sometimes don’t get cleaned up and then cause problems with subsequent tests: Simplifying cleanup after code examples  elasticsearch#51576

5. We need large data sets to generate interesting results. Ideally the testing system is able to pull in large data sets (e.g. Kibana sample data or other public demo data) that customers also have access to so they can run the same APIs. Otherwise, the API examples just prove the API runs but don’t return anything that customers can delve into deeply (or in some cases, don’t return anything at all): Integrate large curated data sets into the documentation tests (e.g. Kibana sample data) elasticsearch#51580

6. AFAIK, only the elasticsearch repo has code snippet testing. Ideally if we get a reliable test system it can be applied consistently to any/all repos.

7. Tests that take a lot of setup make for very slow grade checks (e.g. for ML anomaly detection, you need to add data, create data feed, create job, start data feed, open job, wait for processing of data before you can get results or stats or anything interesting). Ideally, this could be tested asynchronously and not block other PRs (e.g. once per day): Improve pull request feedback times on docs only changes elasticsearch#51582