Massive issue influx from single client

[bobvandevijver]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Self-hosted/on-premise

Which package are you using?

@sentry/vue

SDK Version

7.19.0

Framework Version

7.19.0

Link to Sentry event

No response

Steps to Reproduce

With our latest production update yesterday, we went from Sentry 7.17.4 to 7.19.0. Shortly afterwards, our installation was flooded by events from a single IP-address, causing all kinds of JS errors that are not related to the webpage at all, but are clear result from an automated scan. Examples are:

• ReferenceError: createPopper is not defined
• ReferenceError: Swiper is not defined
• ReferenceError: PrefixFree is not defined

The common divider here is that all those event are caused by an anonymous script that is probably injected by the user itself.

This single user (with a Chinese IP-address) singlehandedly triggered about 500 events (of which 200x an ingress limited event). A second (again Chinese) IP-address just triggered 2.4k of events (of which 1.4k ingress limited events).

This has not happened before, so maybe something broke with the detection of these kinds of events? I am not seeing anything related in the release notes though...

Expected Result

I believe these events should be filtered before being posted to Sentry.

Actual Result

Complete chaos 😨

[Lms24]

Hi @bobvandevijver thanks for writing in! This seems strange to me, as we didn't change anything SDK-wise w.r.t filtering out known, problematic errors. Is this still happening or was it just at this one time?

We do try filtering out certain errors on the server-side, in Relay. For example crawler errors or browser extension errors. You can enable or disable automatic filtering, as well as apply custom rules under your project's settings tab in Sentry: "Project Settings" -> "Inbound Filters". It might be worth trying to filter out specific error messages that you observed were only triggered by this one user.

Do you by any chance have access to the user-agent sent by this user? Perhaps it's a new crawler or something we could add to our filters.

[bobvandevijver]

We've seen it three times, but I've blocked the source IPs on the application firewall to at least prevent them from further scanning.

For example crawler errors or browser extension errors.

Yeah, both are enabled (by default I believe?)

It might be worth trying to filter out specific error messages that you observed were only triggered by this one user.

They are basically all unique errors, all reference errors for a lot of different items making that quite impossible. The one I mentioned in the opening post (2.4k events, 1.4k ingress events) triggered 1k different error events!

Here is a screenshot of just the first page of the last time it happened:

Here's a screenshot of one of the events:

Most curious for me is the error when trying to resolve that anonymous URL actually. When trying to reproduce this by triggering something manually with setTimeout(() => UnknownSomething.call(), 1000) Sentry didn't bother trying to resolve the anonymous line at all...

Do you by any chance have access to the user-agent sent by this user? Perhaps it's a new crawler or something we could add to our filters.

Absolutely, here is is:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/599.37 (KHTML, like Gecko) Chrome/55.0.2590 Safari/537.36

It seems to be using an ancient Chrome version...

[thallada]

We are also seeing the massive influx of errors. I keep trying to block them by IP but they keep using new IPs so it's a never ending cat and mouse. Can they get added to the crawler blocking? (we have both crawlers and errors from extensions blocked).

User agents I've seen (this is definitely not all of them, just a random sample from some of the spam errors):

• Mozilla/5.0 (Windows NT 7_2_2; Win64; x64) AppleWebKit/577.46 (KHTML, like Gecko) Chrome/105.0.1775 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 9_1) AppleWebKit/602.42 (KHTML, like Gecko) Chrome/54.0.2748 Safari/537.36
• Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/579.45 (KHTML, like Gecko) Chrome/58.0.605 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 9_0_1) AppleWebKit/577.54 (KHTML, like Gecko) Chrome/69.0.2215 Safari/537.36

Since these user agents also seems to be frequently changing, it might not be a good way to block them (even if I could do that, doesn't seem possible, see getsentry/sentry#12753).

The only thing consistent between all of them is that the IP addresses are geolocated to somewhere in China. Also they all have stack traces that start with at ? (<anonymous>:1:28).

This started happening for us around November 4th.

Edit: I added a new Issue Grouping Fingerprint Rule that I hope will target these spam errors specifically:

error.type:ReferenceError stack.abs_path:"<anonymous>" -> spam-reference-error

It only applies to future errors, so I won't know if it will work until they spam us again. If it does I should be able to permanently ignore the "spam-reference-error" issue group.

Edit 2: It worked. I can now ignore all of these spam errors grouped together.

[Lms24]

Hi @bobvandevijver and @thallada ,
can you confirm that this only happens with newer SDK versions? Does it still happen if you downgrade the SDK to 7.17.4 or older? We're curious to see if this is just a bad timing or if there's in fact some problem with newer SDK versions.
Overall, to us these user agents look pretty normal in the sense that you can't really identify the user as a crawler by them. Which makes filtering them on our end pretty hard.

[bobvandevijver]

@Lms24 Actually, the instance I placed the screenshots of was also the last time it happened to us (for now). I updated a couple projects to the same release at the same time (as it was our deployment evening), but it only happened at that particular project. So, I'm starting to think it was indeed just bad timing, also due to the comment of @thallada (thanks for the grouping rule by the way, I just configured it for our project as well!)

I will keep monitoring this, but I am still curious about the cause of Sentry trying to resolve that 'anonymous' url... I have not seen that before, also not with my own anonymous functions triggering similar stack traced.

[thallada]

can you confirm that this only happens with newer SDK versions?

The two apps I'm getting these errors from are both on a pretty old version of @sentry/nextjs actually: 6.19.7

I'll see if upgrading changes anything.

[Lms24]

I'm going to close this issue for the time being. Please feel free to reply here in case these problems continue to appear.