Improve pull request feedback times on docs only changes

[lcawl]

Currently the examples in the Elasticsearch documentation are tested using the functionality described here: https://github.com/elastic/elasticsearch/tree/master/docs#snippet-testing

Tests that take a lot of setup make for very slow gradle checks. For example:

./gradlew :docs:check
BUILD SUCCESSFUL in 18m 8s

That is already slow but if we re-enable code testing for machine learning examples, it will grow even slower. For example, for machine learning anomaly detection, you need to add data, create a data feed, create a job, start the data feed, open the job, then wait for the processing of data before you can get results or stats or anything interesting).

Ideally, long-running tests can be tested asynchronously and not block other PRs (e.g. once per day).
Per @colings86, in the Elasticsearch unit and integration tests there is the concept of "slow" tests that are only run on CI (not on a PR build or local build) and "nightly" test which are run only once a day, though neither of these options are currently available for documentation tests.

[elasticmachine]

Pinging @elastic/es-docs (:Docs)

[elasticmachine]

Pinging @elastic/es-core-infra (:Core/Infra/Build)

[colings86]

Note that as mentioned offline, we need to be careful about adding slow and nightly tests, they should be rare. The downside with these tests is that because they don't run on all builds, they increase the likelihood of committed bugs to the main branches. For this reason we should make sure that if we add the ability for docs tests to be marked as slow or nightly, we are very careful that we only add it to a small number of tests where we can't avoid it

[mark-vieira]

There are things we can do to improve this. Comparatively to other types of source changes, the turnaround time on docs-only changes is very short so I'd argue whether we really need to go down the route of potentially short-circuiting testing here. Quite simply, it's very easy for us to catch errors here early, so we should really try and do so. Introducing test failures into a development branch is very costly and disruptive to development, so if we can catch them in PR checks, we should.

That said, there are some things we could do here. First of which is to not run tests for docs changes that don't touch any snippets. In theory this should be happening already but I'm not completely certain why and I will look into this. I think that should help tremendously for many, if not most, docs changes.

Also I've updated the title of this issue to better reflect what we want to accomplish here.

[mark-vieira]

Another thing we've been looking at is a potential "merge bot". That is, somehow configuring of flagging pull requests such that they are automatically merged once any checks pass. This might help alleviate the feeling that simple docs changes have a relatively high level of ceremony relative to the scope of the change.