Periodically check when a node last checked in and submitted results

[mwielgoszewski]

Run nodes through our alert manager so that we can define within the Rule Manager alerts against the following info:

• action: triggered
• query name: doorman/tasks/node_offline_checks
• columns:
  • since_last_result
  • since_last_result_days
  • since_last_result_seconds
  • since_last_checkin
  • since_last_checkin_days
  • since_last_checkin_seconds
  • since_last_checkin_to_last_result
  • since_last_checkin_to_last_result_days
  • since_last_checkin_to_last_result_seconds

For example, a decent start might be to alert when a node last checked in or submitted a result more than 7 days ago, and alert if the time between the last result and the last check is more than 2 days.

This catches those instances where a node completely goes offline, or rocksdb gets corrupted. In the latter case, we observed osquery continuing to fetch a configuration, but never actually posting any new results.

[mwielgoszewski]

Note, from e5d6bdc:

This commit undoes a lot of the work done in #112, and allows us to
reuse our existing rule manager and alerting infrastructure to also
alert when a new node is enrolled, rather than having to call a
specific method to get this functionality.

[mwielgoszewski]

What a rule looks like to alert on these conditions: