monitor: disk space usage and site uptime

[nolessafool]

was lucky to accidentally notice an out of space error while watching the netlify deploy logs this time. Also would like to know if the site goes down.

We need reliable monitoring that at least emails some people.

[onewheelskyward]

I'm pretty good at log mon. :D Fire me up.

[nolessafool]

In case the thing we aren't yet monitoring (disk space full) happens again, playbook for fixing here in da slack:

https://shift-bod.slack.com/archives/CCFLDTCF7/p1597360599007200

[onewheelskyward]

Ok, the disk space alert is in place, we should get alerts on that one.

Next step: alert when the instance stops responding on port 80.

[onewheelskyward]

I really meant port 443.

[onewheelskyward]

I also added cpu utilization >90% for 15 minutes and a binary check of the AWS System Failure flag.

[nolessafool]

@onewheelskyward mentioned maybe using gatling.io to do uptime testing (connect to an API-backed page like https://www.shift2bikes.org/calendar via HTTPS, tests all the things in one request).