API rate limit exceeded when editing pages

[jeremyzilar]

We are still having issues with our NetlifyCMS instance where our team of editors keep running into issues updating content in our site (they are not technical people). When looking into it, I see a series of API rate limit errors in their consoles:

Pausing requests for 3600 second due to fetch failures: API rate limit exceeded for user ID 395641.

In digging further, these errors seem to happen when they try to load content via a relation widget in Netlify CMS. (see the three dots)

There are times when this works well, but for many, they experience so many issues uploading content that it can often take many attempts through the week to upload their content. It has been incredibly hard to debug what is happening. So far, my best guess is that it has something to do with authenticating with GitHub and the auth going stale.

Our config & setup

• Our packages are up to date https://github.com/brooklynrail/brooklynrail-platform/blob/master/package.json
• Here is our NetlifyCMS Config https://github.com/brooklynrail/brooklynrail-platform/blob/master/themes/brooklynrail/static/workflow/config.yml
• Each of our staff members have GitHub accounts and have been registered in Netlify Identity
• GitHub is registered as an External Provider in Identity, though seems to only work when we log in with our Identity user/pass
• In our GitHub Org,
  • "Netlify Auth" has been added and configured as a third-party app in GitHub
  • Netlify has been added as a GitHub App
• We are using GitLFS for all of our images

A theory

It could be that they log in at one point in the day they are logging in to our Netlify CMS, making a few edits, then leaving that browser window open. Then later in the day, they continue editing and at this point the CMS no longer authenticated with GitHub.

These are not people who will likely log in to GitHub during the day, nor are they checking out the code.

Any advice you can provide would be amazing 🎉

[wing5wong]

still experiencing related issue #4097

[ryangittings]

@jeremyzilar did you manage to resolve this? Any advice/workarounds/fixed?

Cheers!

[willkuerlich]

It is very likely that you run into these https://docs.github.com/en/rest/rate-limit in scenarios of heavy usage of the github API

[wing5wong]

It is very likely that you run into these https://docs.github.com/en/rest/rate-limit in scenarios of heavy usage of the github API

The usage is supposed to be cached, and when this issue occurs it does not appear to be.

Of course if there is no local caching this would be expected

[willkuerlich]

YOu can easily verify that issue at least by either checking the http headers or via "_token" (I believe this is the name) endpoint of github API

[wing5wong]

YOu can easily verify that issue at least by either checking the http headers or via "_token" (I believe this is the name) endpoint of github API

Verifying the rate limit being hit is not the issue though. The issue is exactly that - the rate limit being hit unexpectedly

[samo4]

The crux of the problem is that CMS is firing the API requests more often than necessary (on relation widget see #5920, but also elsewhere). GIT backends on the other hand have strict rate limits. Github for example currently has 5k/h but that can be increased to 15k/h if you go with one of their paid plans.

I'm aware of various efforts to optimize this by reducing the number of requests, but ultimately there will always be a hard limit.

[jeremyzilar]

Thank you @samo4 -- I will look at our use of the Relation Widget and maybe I can remove that. This is still happening, and our editors only have a few windows in the day when they can make edits, otherwise they are locked out. :/

[mohab-sameh]

@martinjagodic @demshy is there any progress on this issue? Quite troublesome for our workflow.

[demshy]

Back then, we fixed it on our fork and it seems to have helped. Since then, the fixes were already merged into Decap: #6410
#6902

Have you already tried using Decap 3.0? If you're still hitting rate limits, please inspect the network and maybe try to identify which sections are doing a ton of requests (possibly duplicate). Sharing your config would also help (to see if path regex fix applies to your case).

[mohab-sameh]

@demshy Oh that's great. I haven't tried Decap 3.0 yet. Looks like there shouldn't be breaking changes going from 2.10.79. I'll test it and report back. I'm deploying the CMS locally for now till I make sure the new version doesn't introduce breaking changes. Thanks a lot for pointing me to this.

[wing5wong]

Just updated and still receiving this.
Going to allow a few reload to see if it only occurs on the initial application load since upgrading to decap 3.0

[mohab-sameh]

@demshy Oh that's great. I haven't tried Decap 3.0 yet. Looks like there shouldn't be breaking changes going from 2.10.79. I'll test it and report back. I'm deploying the CMS locally for now till I make sure the new version doesn't introduce breaking changes. Thanks a lot for pointing me to this.

Forgot to report back. After upgrading, this issue happens much less than before. However, #7218 is now the new blocker.

[wing5wong]

Still experiencing rate limits.
Only 3 changes were made to pages before being limited.

I am not certain if this is related to relations on the pages, or the user utilising the Search field to find pages