403: You have exceeded a secondary rate limit

[danieldelcore]

Hey folks, I seem to be running into some rate limits when running Create Release PR or Publish to npm

It looks my Org/Account hasn't hit a rate limit so I'm wondering if it has done so on the changesets org.

My action config looks like so: https://github.com/CodeshiftCommunity/CodeshiftCommunity/blob/main/.github/workflows/release.yml

Currently using changesets/cli 2.6.2

Failing build: https://github.com/CodeshiftCommunity/CodeshiftCommunity/runs/6961550448?check_suite_focus=true

[akphi]

Hm, I'm not sure if this is the rate limit of your org or the rate limit of the changesets action bot, @Andarist do you happen to know?

[frencojobs]

I don't really think it's possible for me to go over my personal rate limit, so I assume it's due to changesets action bot. I'm also experiencing this error, for the second time now and I only have like 4 repos and barely ran the action in like days.

Maybe related to this? semantic-release/semantic-release#2204

[Andarist]

Hm, I'm not sure if this is the rate limit of your org or the rate limit of the changesets action bot, @Andarist do you happen to know?

To the best of my understanding it's related to this specific token. Looking into the GitHub docs we can see that the primary rate limit for this kind of a token is 1000 requests / hour: https://docs.github.com/en/rest/overview/resources-in-the-rest-api#requests-from-github-actions

However, the report here is about the "secondary rate limit" and we can read here that:

Secondary rate limits are not intended to interfere with legitimate use of the API. Your normal rate limits should be the only limit you target.

So my line of thinking is that one should always first receive an error about the primary rate limit being exceeded and only if they continue to make requests then the secondary rate limit can kick in. This doesn't quite match the provided information about this issue.

I'm not sure what to do here because I lack the required information. Perhaps somebody could create a GitHub support ticket about this to get a better understanding of what could have caused that?

[glasser]

I'm seeing this too. Might make sense to include retries, respecting the Retry-After header?

[Andarist]

That's probably a good idea - it would be great to add some logs for this error first and log the value returned in this retry-after header. I wonder what's the best way to handle this because if this value is too big then it might not be a feasible solution for the action to wait for too long, remember that an idle, but running, action could still cost somebody money.

[varl]

We are hitting a secondary rate limit for this action as well, and it looks related to search API usage.

We have a theory that it relates to this part of the code:
https://github.com/changesets/action/blob/main/src/run.ts#L295-L297

The query it builds on L294 looks like the same one that triggers the request hit by secondary rate limits:

repo:${repo}+state:open+head:${versionBranch}+base:${branch}

https://api.github.com/search/issues?q=
repo:[REDACTED]/[REDACTED]+state:open+head:changeset-release/master+base:master

The full log is attached before for reference[1].

Absolutely looks like the same issue mentioned above[2].

Haven't looked into how to avoid using the search API to figure out if there is an existing PR to update or if a new one needs to be created, and doesn't look like semantic-release has worked out a solution either.

---

[1] Full error log:

[HttpError]: You have exceeded a secondary rate limit. Please wait a few minutes before you try again.
    at /home/runner/work/_actions/changesets/action/v1/dist/index.js:718:926
    at processTicksAndRejections (internal/process/task_queues.js:97:5) {
  status: 403,
  headers: {
    'access-control-allow-origin': '*',
    'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
    connection: 'close',
    'content-encoding': 'gzip',
    'content-security-policy': "default-src 'none'",
    'content-type': 'application/json; charset=utf-8',
    date: 'Thu, 29 Sep 2022 12:55:19 GMT',
    'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
    server: 'GitHub.com',
    'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
    'transfer-encoding': 'chunked',
    vary: 'Accept-Encoding, Accept, X-Requested-With',
    'x-content-type-options': 'nosniff',
    'x-frame-options': 'deny',
    'x-github-media-type': 'github.v3; format=json',
    'x-github-request-id': '0780:1463:1903E5B:3347AE5:633595B7',
    'x-ratelimit-limit': '30',
    'x-ratelimit-remaining': '30',
    'x-ratelimit-reset': '1664456179',
    'x-ratelimit-resource': 'search',
    'x-ratelimit-used': '1',
    'x-xss-protection': '0'
  },
  request: {
    method: 'GET',
    url: 'https://api.github.com/search/issues?q=repo%3A[REDACTED]%2F[REDACTED]+state%3Aopen+head%3Achangeset-release%2Fmaster+base%3Amaster',
    headers: {
      accept: 'application/vnd.github.v3+json',
      'user-agent': 'octokit-core.js/3.1.0 <environment undetectable>',
      authorization: 'token [REDACTED]'
    },
    request: { agent: [Agent], hook: [Function: bound bound r] }
  },
  documentation_url: 'https://docs.github.com/en/free-pro-team@latest/rest/overview/resources-in-the-rest-api#secondary-rate-limits'
}

[2] #192 (comment)

[adesso-os]

Given the action uses Octokit already, could this be solved by just using https://github.com/octokit/plugin-retry.js/?

[thenbe]

Was anyone able to figure out the underlying cause?

It seems that most* people facing this issue encounter the following headers:

{
    'x-ratelimit-limit': '30',
    'x-ratelimit-remaining': '29',
    'x-ratelimit-resource': 'search',
    'x-ratelimit-used': '1',
}

These headers imply that we are far from reaching the api rate limit. Although, the error message does say that it is a secondary rate limit we're exceeding. I'm not sure what might be causing that either, since this issue happens randomly, sometimes even when invoking the api for the first time in days.

Reading through the docs and linked comments leads me to believe that this is a bug in the Github search api.

*sample 403 responses when encountering this issue:

1. HttpError: You have exceeded a secondary rate limit. semantic-release/semantic-release#2204 (comment)
2. HttpError: You have exceeded a secondary rate limit. semantic-release/semantic-release#2204 (comment)
3. HttpError: You have exceeded a secondary rate limit. semantic-release/semantic-release#2204 (comment)
4. HttpError: You have exceeded a secondary rate limit. semantic-release/semantic-release#2204 (comment)
5. 403: You have exceeded a secondary rate limit #192 (comment)

[maximveksler]

I'm seeing many instances of this error as well.

@DanRigby maybe can shed some light on this?

[Andarist]

Yeah, I've also seen it more often lately - and also at very suspicious times (when there was almost no other activity in the whole organization).

I'd very much like to do something about it - but quite frankly, I'm not sure how to handle this in Changesets because it mostly seems to be out of our control.

[thenbe]

@Andarist What if we simply retry the search query multiple times whenever we encounter a 403? This will make us more resilient against the flakiness of what seems like a bug (on github's side). I'm not even sure we have to wait the full retry-after duration. In the 403 responses I've encountered, the retry-after header wasn't even included.

Admittedly, this isn't a great solution. We don't really know if it will solve the underlying issue, one which does not lie in this repo at that. Maybe I'm just desperate because the publishing workflow is my most vulnerable ci pipeline.

[varl]

Octokit provides a "throttle and retry" plugin[1] that:

Implements all recommended best practices to prevent hitting secondary rate limits.

It would be interesting if it is possible to wrap the Octokit client[2] with the plugin using the method outlined in the @actions/github toolkit[3].

[1] https://github.com/octokit/plugin-throttling.js
[2] https://github.com/changesets/action/blob/main/src/run.ts#L272
[3] https://github.com/actions/toolkit/tree/main/packages/github#extending-the-octokit-instance

edit: I saw a comment discussing plugin-retry.js, but that one explicitly does not retry 403 errors, which is why the throttling plugin looks promising.

[trivikr]

We reached the secondary limit as well three times in aws-sdk-js-codemod repo.
A manual retry of the action fixes it though.

Example run: https://github.com/awslabs/aws-sdk-js-codemod/actions/runs/3858163665/jobs/6576401193

[akphi]

@Andarist do you want to try out using https://github.com/octokit/plugin-throttling.js/ and https://github.com/octokit/plugin-retry.js/ ? I can help with that

[robingronlund]

Any updates regarding this issue? I'm seeing this as well.