Outage needs investigation

[cecilia-donnelly]

On [2016-07-07], the production site went down while it was on commit 9ba06df. We need to know what caused the outage, so that we can prevent it from happening again.

I think it might have been something to do with our connection to Mailgun (which we use to send emails), but we'll do more testing using the demo server to find out for sure.

[kfogel]

We also need to set up an auto-alert service like uptimerobot, so we know right away about an outage like this. Just gonna dump some good links here -- @cecilia-donnelly, let's pick one and, as we say in IT, operationalize that sucker:

• https://uptimerobot.com/ (see also http://blog.uptimerobot.com/whats-next-for-uptime-robot/)
• https://www.pingdom.com/
• https://www.pagerduty.com/
• http://www.servermom.org/useful-free-uptime-monitoring-services-server/1175/

[cecilia-donnelly]

Note: Our Mailgun logs don't show any problems. The Apache logs show that apache wasn't able to reach node. @kfogel suggests looking into npm_prefork or a similarly named Apache variable.

[kfogel]

(We have a monitor at uptimerobot.com now, by the way, though that doesn't close this ticket obviously.)

[kfogel]

Oops, and then I hit the wrong button and accidentally closed it anyway, sigh. Re-opening.

[kfogel]

Actually, the err.log file does have Mailgun errors. I'm not sure when they're from (see issue #224 about date stamps in logs) but they're there.