Tweak the handling of the retry-able & DLQ errors

[webmat]

In working on -- and looking at -- simple enough issues (#771 and #697), I've noticed a few problems on how the plugin handles the DLQ.

Here are a few things I think we could tweak:

• Add a new attribute to configure additional statuses to DLQ (currently only on 400 and 404)
  • This would solve situations like Make 403 response configurable for going into DLQ #697, Elasticsearch Output Drop Events Sent to Closed Index #134 (See also customer issues linked to Make 403 response configurable for going into DLQ #697)
• Deprecate & rename confusingly named failure_type_logging_whitelist. This is a list of error statuses we don't want to log. Perhaps an attribute named silence_...
• Encapsulate the DLQing in a small helper class
  • Avoid the nil setting and checking business
  • Messy logic in detecting status of DLQ would be encapsulated there, making plugins lack a proper API to understand if DLQ is enabled elastic/logstash#8064 less of an issue
  • Sending to DLQ should also log (unless it's a status configured to be silenced), it shouldn't be DLQ vs log like it currently is.
  • This encapsulation would help handle potential big changes to how the DLQ work, e.g. Internal pipelines as DLQ elastic/logstash#9502
• Consider sending events whose index has an interpolation failure to the DLQ, as suggested in DLQ when sprintf references a missing field #718
• Consider introducing a new metric for events sent to the DLQ, as suggested in [DLQ Improvements] Introduce a new metric to track how many events go to the DLQ #778.

[chandlermelton]

I have noticed that my Logstash pipelines have become less resilient to this problem since my upgrade from 5.1.2 to 5.6.9. I am looking into implementing DLQ for this. (My issue is that a server will come out of a day or weeks old maintenance and filebeat will attempt to ship any unsent logs near the time of the original crash into an index which has since been closed.)

Perhaps an option could also be adding a tag or flag on an index which could instruct Logstash's response. ( index_when_closed_repsonse: "DLQ/ drop / new_target_index" )

[webmat]

@chandlermelton Another thing you can do, when your machines can send old logs, is to have Logstash automatically drop any messages older than your index closing/pruning policy.

[chandlermelton]

I am definitely interested in this. I considered it, but wasn't sure how I could go about this without hardcoding index names in a drop filter. Is there a way to say:
if dest_index = today-7days, then {drop} or something similar?

An example would be logtype-2018.06.04-7days meaning 2018.05.28 and older would be dropped.

[webmat]

One way to do it is with the age filter. It's specifically the example given in the description of the plugin. Note that you may have to add more checks if some indices have longer retentions than others.

If age is not flexible enough, you can always use the ruby filter to make more complex checks. Your ruby filter can then add a tag to flag the event for deletion, or it can call event.cancel directly in the Ruby code (see drop filter implementation).

[andsel]

Closing this issue because the task are now tracked in the more global meta issue: elastic/logstash#10493