Add Spike Protection

[chadwhitacre]

https://docs.sentry.io/product/accounts/quotas/spike-protection/

See #1993 and #1838 for parallel cases.

[aldy505]

I don't see why this is helpful on self-hosted, as every event never got dropped unless I set them to (via the "Delete and discard future events" button)

[antonpodolsky]

@aldy505 At scale, spikes in error reports are choking our self-hosted Sentry backend and require continuous tweaking of resource limits.

[aldy505]

@antonpodolsky May I know your machine configuration and how much traffic is going through your instance?

[antonpodolsky]

@aldy505 Traffic is several millions hourly reports. Regarding the actual setup, it's a bit elaborate, so not sure I can share it here in a concise way.

[Aulig]

I don't see why this is helpful on self-hosted, as every event never got dropped unless I set them to (via the "Delete and discard future events" button)

I normally get <200 events per hour. Recently some kind of bug is suddenly sending 4000 events in a very short time, which is taking down my entire sentry instance (8GB ram, 4 CPU cores). My server load jumps to 280 or whatever and sometimes I can't even SSH in anymore. A simple server reboot doesn't solve it either, since it seems like Redis crashes completely, so it doesn't get restarted on server reboot. I need to manually shut down the docker containers and start them again.

When the Sentry instance goes down, it massively slows down my website (since the sentry error handling script needs to be loaded early in the HTML), since browsers attempt to wait for the sentry script to load (which it doesn't, so it waits until the timeout).

Having this happen every 12-48 hours is no fun ;)