Filebeat Cleanup

[ruflin]

This issue is intended as meta issue to track several filebeat cleanups

Harvester separation

Log and stdin harvester should be separated and made independent potentially with separate readers.

• Remove Continuable() interface as readers are separated (will no happen as not needed at the moment)

Harvester

• Unify error message and make them more consistent. For example err_inactive.
• Use common.Backoff in file harvester.
• Improve handling of encoding Cleanup Filebeat code #1975 (comment)
• Merge shared variables between harvester and state (path/Source, offset) Prospector and Harvester Cleanups #2020

Prospectors

• Prospectors must be able to send state updates to the registrar directly (no need to initialised harvester) Enable Prospector to Update state without Harvester #1978
• startHarvester returns an error which is not checked by all implementations Log errors when harvester could not be started #2240

Testing

• Clean up inconsistent variable naming in filebeat.yml.j2. All should be with _ and not cameCase. Cleanup Filebeat code #1975

General

• Introduce expvar to track open files Prospector and Harvester Cleanups #2020
• Add additional expvar to track states

Docs

• Write "How Filebeat works" documentation guide. Explain crawler, prospector, harvestor How does Filebeat work #1963

[urso]

Use common.Backoff in file harvester.