Support setting `action.auto_create_index=false` per request

[spalger]

Describe the feature: On index/create or any request that would normally cause non-existing indexes to be created, a parameter would allow disabling automatic index creation for that request. The inverse should not be true (you shouldn't be able to opt-out of the action.auto_create_index config).

Examples:
Standard scenario:

# config/elasticsearch.yml
action.auto_create_index: true

curl -XDELETE 'http://localhost:9200/.kibana'
# <- ok

curl -XPOST 'http://localhost:9200/.kibana/visualization/id?auto_create_index=disable' -d '{}'
# <- fails the same way it would if action.auto_create_index was set to false

Inverse scenario:

# config/elasticsearch.yml
action.auto_create_index: false

curl -XDELETE 'http://localhost:9200/.kibana'
# <- ok

curl -XPOST 'http://localhost:9200/.kibana/visualization/id?auto_create_index=enable' -d '{}'
# <- should also fail, the only valid value for auto_create_index is "disable"

Why: Kibana currently polls elasticsearch every so often to determine if it is in an expected state. In those checks it ensures that the .kibana index exists and creates it if not. This has several issues, one of which is the window of time between when the .kibana index is deleted and the next Kibana health check runs where Kibana's behavior is unpredictable.

We would like to move to a new model for this check:

• make requests that try to use the .kibana index
• ensure that requests fail when the index does not exist
• react to the errors by:
  • creating the .kibana index
  • waiting for it to go yellow
  • trying the original request again

/cc @jbudz @tylersmalley

[clintongormley]

@spalger This would be better handled by creating an index template at startup which matches .kibana. Then your indexing request would create a valid .kibana index instead of failing.

[spalger]

That's not a bad idea, but we don't control the list of index templates and would still need to make sure that the template exists. (If someone deletes the index template Kibana would either need to detect it's missing or be restarted so it could be recreated)

[jasontedor]

This would be better handled by creating an index template at startup which matches .kibana. Then your indexing request would create a valid .kibana index instead of failing.

This is the strategy used by Beats already, for example.

[clintongormley]

Chatting to @spalger the concern with index templates is that somebody could delete the template and the .kibana index on a running server. I'm not sure we can really protect against that. If somebody manages to do that (either by deleting their data dir or via the REST api) then they must expect a weird outcome, and they can do it again to get their kibana index setup correctly.

[jasontedor]

We discussed this during Fix-it-Friday. We do not think this the right solution to the problem. We do think that the template strategy is the route to take. This discussion did provoke us to consider whether or not action.auto_create_index defaulting to true is the right default. I will open a separate issue for this.