Description
Proposal: When sprintf on the index name would fail because one or more fields do not exist, Logstash should DLQ the event instead of sending it to Elasticsearch.
Problem:
Logstash's event sprintf has a behavior that when a field does not exist, the formatting string is left unaltered.
For example, if a user has this:
index => "%{[my][field]}-%{+YYYY}"
and supposing that [my][field] does not exist in the event, an index will be created in Elasticsearch literally as %{[my][field]}-2017.
This is confusing and also tricky to recover from. It's tricky partly because of the {} and [] format and field-reference characters which curl uses (by default) with special meaning: globbing. So for a user to delete this index, they will experience an error:
% curl 'https://es.foo.com:9200/%{[my][field]]}-2017'
curl: (3) [globbing] nested brace in column 27
Or worse, if no nested fields are being used:
% curl -v 'http://localhost:9200/%{myfield}-2017.12.28'
> GET /%myfield-2017.12.28 HTTP/1.1
As above, curl interprets the {} to have special meaning, and this will certainly confuse users.
This is the beginning of a yak shave.