High percentage of invalid assessments using App Check with reCAPTCHA Enterprise

[harrisonlo]

Operating System

Various

Browser Version

Various

Firebase SDK Version

10.7.0

Firebase SDK Product:

App Check

Describe your project's tooling

React app on desktop web and mobile web

Describe the problem

App Check

I'm observing a high percentage of invalid requests for App Check on the Firestore Database and the Realtime Database:


Additionally, the graph for Storage is showing 100% verified, but it only has a few hundred requests.

reCAPTCHA Enterprise

Below are screenshots from the GCP reCAPTCHA Enterprise dashboard, also for the past 30 days:

Issue and thoughts

Given that requests are almost all marked as low risk on the GCP dashboard, I'm guessing there might be some bug on the App Check client that's not sending valid requests to reCAPTCHA. I could be wrong, but given the high false positive rates (as reported by users), I simply cannot enforce it across the databases. I'm grateful to the Firebase team for supporting this and hope I can enforce it one day. Let me know if there's any more information I can provide.

Steps and code to reproduce issue

This is hard to reproduce. I opened an issue a while back #7116 that was related. From my own usage in the past, app check starts to fail after having the app opened for a long period of time, potentially related to #6708. However, I've also had a (returning) user seeing it on app open, meaning assessment passed before but failed later on the same device.

In the app, I would initializeApp() then initializeAppCheck immediately. This happens when users open the app.

const app = initializeApp(options, name)

initializeAppCheck(app, {
  provider: new ReCaptchaEnterpriseProvider(key),
  isTokenAutoRefreshEnabled: true
})

[google-oss-bot]

I couldn't figure out how to label this issue, so I've labeled it for a human to triage. Hang tight.

[hsubox76]

There are two ways we can dig further into this:

(1) On your end, you can create a test app that does not use Firebase and only performs a reCAPTCHA validation using reCAPTCHA tooling. Try this on the same devices and situations where you are getting a lot of failures and see if you get a similar failure rate. If so, that indicates that reCAPTCHA isn't playing nice with your devices/usage. This may be easy to do if you're familiar with using reCAPTCHA by itself, and may be a bit of a pain otherwise.

(2) On our end, if you can get the network request logs in your browser from one of the failed attempts, that would be great. We're looking for calls to https://www.google.com/recaptcha/enterprise.js and https://content-firebaseappcheck.googleapis.com/v1. If you can share the request and response data from each of those requests, that would help determine where the problem is. The first endpoint is getting the ReCAPTCHA token and the second is sending that token to the AppCheck endpoint to validate it, and exchange it for an AppCheck token. Also, if you have any errors or warnings logged in your browser's developer console (on the client that is running, not the backend console).

[harrisonlo]

Thanks for providing a direction for debugging this. While I don't have access to the devices of my users, I just checked the network requests on my device (Chrome browser) and noticed some random failed firestore requests.

Not sure if this is related. There are no response data in the failed requests. The recaptcha and appcheck requests were fine in this particular session. Note that there are 2 firebase instances running, which I'm guessing explain the two different gsessionIds.

Here's the successful request of the same gsessionId (starting with q7Kz...) that was 2 minutes before the failed one:

Another request of the same type succeeded 1 second after the failed one, now with a different gsessionId. Note that I'm not doing anything on the page, it's just idle there with several query listeners open.

I'm seeing the same failed firestore requests 10 minutes later. Pasting the console errors and warnings here:

I don't know if these are relevant at all. Let me know if they're not, I'll remove them (or move them to another issue) so we can focus on the app check issue. I'll keep checking the recaptcha and app check requests in the future.

[hsubox76]

I'll have to consult a firestore team member about if this is indicative of anything but I don't think it's related to the App Check issue, since Firestore (1) hits a different endpoint with a different protocol and (2) is downstream of App Check, so if anything, App Check errors would cause Firestore errors, but not be affected by them. It could indicate spotty network connections, which could affect both, though. Let me see if I can find someone from Firestore to confirm.

[hsubox76]

So I talked to a Firestore team member and it's unlikely the Firestore errors are related to the App Check errors. I would suggest making another issue for the Firestore errors if you find they are impacting you a lot. From a glance, it looks like it could just be a spotty network connection that missed one request, but if it seems like a bigger or more serious problem than that, feel free to open a Firestore issue for it.

We'll keep this issue open to try and debug the App Check issues.

[sergio-dd]

Hello @harrisonlo how are you?
I also use recaptcha enterprise, I'm having a lot of access errors, real users who receive 403 forbidden when trying to access my site, funny thing is, the error occurs on Wi-Fi, but if I connect to 4G internet on my cell phone, it works normally, do you have any recommendations? Thanks.

[anthony-autrey]

@harrisonlo I'm having this exact issue with my web app, and I've very concerned by the idea of turning on app check in production. Did you ever find a solution? Thanks in advance!

[harrisonlo]

@sergio-dd @anthony-autrey Unfortunately we decided to just not turn on App Check. I hope this gets fixed in the future tho.