Defend against spam for public account registration

[straight-shoota]

Lately we noticed a lot of spam on the public registration form. This is especially problematic because e-mail addresses of spammers are blocked by SES which in turn might reject all mails sent from CDx (Slack thread).

A first line of defence is disabling public account registration (implemented in #1695). This should be fine for our current production use cases because new users are typically invited in.

But we should look for proper solutions to the spam problem to make public account registration a viable option.

1. We could use reCaptcha to hinder spam bots from registering (previously implemented in Add reCAPTCHA support guisso#64 for example). The downside of that is annoying UX and a dependency on an external service.
2. A less obtrusive alternative is https://github.com/markets/invisible_captcha which seems to be doing a great job telling users from bots without obvious UX implications.
3. The current spam bots seem to be registering with e-mail addresses that are easily identifiable via spam blocklists. The SES bounce notifications even name the block sources (e.g. cbl.abuseat.org, sbl.spamhaus.org). So we could perform some checks against these lists on hour end and prevent mail delivery entirely.

Regardless of which spam-detection mechanism we use, there is always a possibility that adversaries might slip through. So an important mitigation mechanism would be to make mail delivery more fault tolerant. We shouldn't lose e-mail capabilities for the entire organization just because of some spam registrations.
A simple mitigation would be to use a separate SES account for registration mails. If this gets suspended, we won't be able to register new users, but any other transactional mails from inside the app(s) won't be affected. Those are unlikely to bounce.
We could also monitor e-mail bounces and temporarily suspend registrations automatically when those pile up.

[diegoliberman]

Excellent summary @straight-shoota, thank you so much. I didn't know option 2 existed, it sounds great. Regarding option 3 and final observation, make sense to me, but I'd like to read @ysbaddaden and @matiasgarciaisaia opinions.

[ysbaddaden]

I like invisible_captcha. Honeypot and time-sensitive form submission are nice. It should be quick to implement, though it only supports Ruby 2.5 and Rails 5.2 😭

I also like using the spam blocklists. A tad more complex to implement (must download/cache the lists and regularly update them). A list of the 8 most important lists can be found here: https://www.rackaid.com/blog/email-blacklists/

[matiasgarciaisaia]

I'm not sure if it's possible to have different SES accounts in AWS - I'll take a look at that.

invisible_captcha sounds nice. If it's effective, I'm OK with using it instead of Google's reCAPTCHA.

Same for spamlists 👍

[omelao]

Do we use AWS WAF Bot Control? It's a good feature to reduce this kind of thing. The only problem is that we have to allow some important bots, like crawlers or some tool we use. But as it is a closed system and not a website, it is possible that we don't even need.

[matiasgarciaisaia]

Nope, I've never seen that service. I don't think we care about web crawlers, either.

[omelao]

Nope, I've never seen that service. I don't think we care about web crawlers, either.

I used it on a domain selling site, where all domains pointed to the same application, and there was an overload of bots entering through all domains.

It's good. It uses machine learning to detect bots using a AWS global http requests database. It's a nice feature and it's not so complicated to set. I believe we can use captcha too, but this service is also very good. Ping me if you need.

[sardar-usman]

@diegoliberman It's working fine. Tested together with @omelao.

[sardar-usman]

@omelao we have verified it locally but it has to verify on Staging.

[sardar-usman]

@omelao It's working fine. Cc @diegoliberman