fix: CloudFront CDN rate limit #74

alexanderroidl · 2023-07-24T12:36:09Z

Problem

While TMDB themselves still do not apply a rate limit, their CDN provider CloudFront does.

When sending many requests in a short timespan, a HTTP error is likely to be thrown (due to a 429 - Too Many Requests HTTP response). See complete error object below.

Here is an excerpt from the error, indicating it's origin from CloudFront:

{
  (...),
  response: {
    status: 429,
    statusText: 'Too Many Requests',
    headers: {
      'content-type': 'application/json; charset=utf-8',
      'transfer-encoding': 'chunked',
      connection: 'keep-alive',
      date: 'Sun, 23 Jul 2023 14:14:22 GMT',
      server: 'openresty',
     'x-cache': 'Error from cloudfront',                                        <-----------------
      via: '1.1 4cef84d6d9d593ea2c191c9370b4ebba.cloudfront.net (CloudFront)',  <-----------------
      'x-amz-cf-pop': 'TXL50-P4',
      'alt-svc': 'h3=":443"; ma=86400',
      'x-amz-cf-id': 'f2hXJu173SWIWxZZsPsdR7-ivJkA30RqnxCn0uQyWmewCo0g1ZXuiQ==',
      vary: 'Origin'
    },
  },
 (...)
}

This was confirmed by TMDB employees a while ago:

One of our CDN providers enforces some base level rate limiting to help prevent DDOS attacks. I believe it's a maximum of 50 requests per second and 20 connections per IP.

Changes made

fix: Implemented a request rate limit of 50 requests/second, as stated by the TMDB employee in their forums.
chore: Improved RegExp match typing.
chore: Replaced deprecated String.prototype.substr with String.prototype.slice.
chore: Removed unusued method/import.

Complete AxiosError object:

{
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus],
    headers: {
      Accept: 'application/json, text/plain, */*',
      'User-Agent': 'axios/0.26.1'
    },
    method: 'get',
    url: 'https://api.themoviedb.org/3/movie/1063591/external_ids',
    params: { api_key: 'REDACTED_API_KEY' },
    data: undefined
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Function (anonymous)],
      timeout: [Function (anonymous)],
      finish: [Function: requestOnFinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: false,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: true,
    maxRequestsOnConnectionReached: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: false,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    strictContentLength: false,
    _contentLength: 0,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    _closed: false,
    socket: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: false,
      _SNICallback: null,
      servername: 'api.themoviedb.org',
      alpnProtocol: false,
      authorized: true,
      authorizationError: null,
      encrypted: true,
      _events: [Object: null prototype],
      _eventsCount: 9,
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'api.themoviedb.org',
      _closeAfterHandlingError: false,
      _readableState: [ReadableState],
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: undefined,
      _server: null,
      ssl: [TLSWrap],
      _requestCert: true,
      _rejectUnauthorized: true,
      timeout: 5000,
      parser: null,
      _httpMessage: [Circular *1],
      autoSelectFamilyAttemptedAddresses: [Array],
      [Symbol(res)]: [TLSWrap],
      [Symbol(verified)]: true,
      [Symbol(pendingSession)]: null,
      [Symbol(async_id_symbol)]: 4405,
      [Symbol(kHandle)]: [TLSWrap],
      [Symbol(lastWriteQueueSize)]: 206,
      [Symbol(timeout)]: Timeout {
        _idleTimeout: 5000,
        _idlePrev: [TimersList],
        _idleNext: [Timeout],
        _idleStart: 5632,
        _onTimeout: [Function: bound ],
        _timerArgs: undefined,
        _repeat: null,
        _destroyed: false,
        [Symbol(refed)]: false,
        [Symbol(kHasPrimitive)]: false,
        [Symbol(asyncId)]: 4407,
        [Symbol(triggerId)]: 1
      },
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kSetNoDelay)]: false,
      [Symbol(kSetKeepAlive)]: true,
      [Symbol(kSetKeepAliveInitialDelay)]: 60,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(connect-options)]: [Object]
    },
    _header: 'GET /3/movie/1063591/external_ids?api_key=REDACTED_API_KEY HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'User-Agent: axios/0.26.1\r\n' +
      'Host: api.themoviedb.org\r\n' +
      'Connection: keep-alive\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: nop],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 443,
      protocol: 'https:',
      options: [Object: null prototype],
      requests: [Object: null prototype] {},
      sockets: [Object: null prototype],
      freeSockets: [Object: null prototype],
      keepAliveMsecs: 1000,
      keepAlive: true,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 915,
      maxCachedSessions: 100,
      _sessionCache: [Object],
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'GET',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    joinDuplicateHeaders: undefined,
    path: '/3/movie/1063591/external_ids?api_key=REDACTED_API_KEY',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      joinDuplicateHeaders: undefined,
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 429,
      statusMessage: 'Too Many Requests',
      client: [TLSSocket],
      _consuming: true,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'https://api.themoviedb.org/3/movie/1063591/external_ids?api_key=REDACTED_API_KEY',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 22,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0
    },
    aborted: false,
    timeoutCb: [Function: emitRequestTimeout],
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'api.themoviedb.org',
    protocol: 'https:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 0,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'https://api.themoviedb.org/3/movie/1063591/external_ids?api_key=REDACTED_API_KEY',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kBytesWritten)]: 0,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      'user-agent': [Array],
      host: [Array]
    },
    [Symbol(errored)]: null,
    [Symbol(kHighWaterMark)]: 16384,
    [Symbol(kRejectNonStandardBodyWrites)]: false,
    [Symbol(kUniqueHeaders)]: null
  },
  response: {
    status: 429,
    statusText: 'Too Many Requests',
    headers: {
      'content-type': 'application/json; charset=utf-8',
      'transfer-encoding': 'chunked',
      connection: 'keep-alive',
      date: 'Sun, 23 Jul 2023 14:14:22 GMT',
      server: 'openresty',
      'x-cache': 'Error from cloudfront',
      via: '1.1 4cef84d6d9d593ea2c191c9370b4ebba.cloudfront.net (CloudFront)',
      'x-amz-cf-pop': 'TXL50-P4',
      'alt-svc': 'h3=":443"; ma=86400',
      'x-amz-cf-id': 'f2hXJu173SWIWxZZsPsdR7-ivJkA30RqnxCn0uQyWmewCo0g1ZXuiQ==',
      vary: 'Origin'
    },
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'get',
      url: 'https://api.themoviedb.org/3/movie/1063591/external_ids',
      params: [Object],
      data: undefined
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: true,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: false,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      strictContentLength: false,
      _contentLength: 0,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: [TLSSocket],
      _header: 'GET /3/movie/1063591/external_ids?api_key=REDACTED_API_KEY HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Host: api.themoviedb.org\r\n' +
        'Connection: keep-alive\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'GET',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      joinDuplicateHeaders: undefined,
      path: '/3/movie/1063591/external_ids?api_key=REDACTED_API_KEY',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: [Function: emitRequestTimeout],
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'api.themoviedb.org',
      protocol: 'https:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype],
      [Symbol(errored)]: null,
      [Symbol(kHighWaterMark)]: 16384,
      [Symbol(kRejectNonStandardBodyWrites)]: false,
      [Symbol(kUniqueHeaders)]: null
    },
    data: {
      status_code: 25,
      status_message: 'Too many requests.',
      success: false
    }
  },
  isAxiosError: true,
  toJSON: [Function: toJSON]
}

alexanderroidl · 2023-07-24T12:37:00Z

src/moviedb.ts

@@ -130,7 +118,7 @@ export class MovieDb {

    // Get the params that are needed for the endpoint
    // to remove from the data/params of the request
-    const omittedProps = (endpoint.match(/:[a-z]*/gi) || []).map((prop) => prop.substr(1))
+    const omittedProps = [...(endpoint.match(/:[a-z]*/gi) ?? [])].map((prop) => prop.slice(1))


Destructed RegExpMatchArray into an array to improve typing.

alexanderroidl · 2023-07-24T12:38:06Z

src/moviedb.ts

@@ -130,7 +118,7 @@ export class MovieDb {

    // Get the params that are needed for the endpoint
    // to remove from the data/params of the request
-    const omittedProps = (endpoint.match(/:[a-z]*/gi) || []).map((prop) => prop.substr(1))
+    const omittedProps = [...(endpoint.match(/:[a-z]*/gi) ?? [])].map((prop) => prop.slice(1))


String.prototype.slice was marked as deprecated.

alexanderroidl · 2023-07-24T12:38:14Z

src/moviedb.ts

@@ -71,7 +70,7 @@ export class MovieDb {

    if (matches.length === 1) {
      return matches.reduce((obj, match) => {
-        obj[match.substr(1)] = params
+        obj[match.slice(1)] = params


String.prototype.slice was marked as deprecated.

alexanderroidl · 2023-07-24T12:39:34Z

src/moviedb.ts

@@ -80,17 +79,6 @@ export class MovieDb {
    return {}
  }

-  /**


Remove unused method.

alexanderroidl · 2023-07-24T12:39:47Z

src/moviedb.ts

-import {
-  HttpMethod,
-  AuthenticationToken,
-  RequestOptions,


Remove unused import.

grantholle · 2023-08-26T05:09:34Z

The irony is throttling was removed recently, but this is a better solution anyway

SubJunk · 2023-09-16T02:36:40Z

This seems like a good fix for simple environments. Unfortunately it still won't be enough for production environments with more than one instance, for example those of us who use https://www.npmjs.com/package/pm2 if we are running 4 load-balanced instances, that's 200 requests per second with this change.
If the rate is really 50 per second there will need to be a rate limiter outside of Node.

alexanderroidl · 2023-09-18T06:51:08Z

This seems like a good fix for simple environments. Unfortunately it still won't be enough for production environments with more than one instance, for example those of us who use https://www.npmjs.com/package/pm2 if we are running 4 load-balanced instances, that's 200 requests per second with this change. If the rate is really 50 per second there will need to be a rate limiter outside of Node.

This is a great idea. I did indeed not think about this problem before. How do you think could a limiter outside of Node look like, exactly?

SubJunk · 2023-09-29T04:00:02Z

@alexanderroidl I guess an event queue would allow Node to have several instances but still a single pipeline for a certain event like this, but that's a lot of infrastructure complexity

Maybe a simple further enhancement to this library could involve an optional delimiter field for total instances, with the logic requestsPerSecondLimit = Math.floor(requestsPerSecondLimit / totalNodeInstances). Then if PM2 is configured for 4 instances, you could pass ..., 50, 4) into the constructor

fix: Rate limits for CDN

a4ee3d6

alexanderroidl commented Jul 24, 2023

View reviewed changes

alexanderroidl changed the title ~~fix: Rate limits for CDN~~ fix: Rate limits concerning CDN providers Jul 24, 2023

alexanderroidl commented Jul 24, 2023

View reviewed changes

src/moviedb.ts

@@ -80,17 +79,6 @@ export class MovieDb {

return {}

}

/**

Copy link

Contributor Author

alexanderroidl Jul 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove unused method.

alexanderroidl commented Jul 24, 2023

View reviewed changes

src/moviedb.ts

import {

HttpMethod,

AuthenticationToken,

RequestOptions,

Copy link

Contributor Author

alexanderroidl Jul 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove unused import.

chore: Added collaborator

e6ccedb

alexanderroidl changed the title ~~fix: Rate limits concerning CDN providers~~ fix: CloudFront CDN rate limit Jul 24, 2023

Merge branch 'main' into fix/cdn-rate-limits

5fa08c0

grantholle merged commit f57a503 into grantholle:main Aug 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: CloudFront CDN rate limit #74

fix: CloudFront CDN rate limit #74

alexanderroidl commented Jul 24, 2023 •

edited

Loading

alexanderroidl Jul 24, 2023

alexanderroidl Jul 24, 2023

alexanderroidl Jul 24, 2023

alexanderroidl Jul 24, 2023

alexanderroidl Jul 24, 2023

grantholle commented Aug 26, 2023

SubJunk commented Sep 16, 2023

alexanderroidl commented Sep 18, 2023

SubJunk commented Sep 29, 2023

fix: CloudFront CDN rate limit #74

fix: CloudFront CDN rate limit #74

Conversation

alexanderroidl commented Jul 24, 2023 • edited Loading

Problem

Changes made

alexanderroidl Jul 24, 2023

Choose a reason for hiding this comment

alexanderroidl Jul 24, 2023

Choose a reason for hiding this comment

alexanderroidl Jul 24, 2023

Choose a reason for hiding this comment

alexanderroidl Jul 24, 2023

Choose a reason for hiding this comment

alexanderroidl Jul 24, 2023

Choose a reason for hiding this comment

grantholle commented Aug 26, 2023

SubJunk commented Sep 16, 2023

alexanderroidl commented Sep 18, 2023

SubJunk commented Sep 29, 2023

alexanderroidl commented Jul 24, 2023 •

edited

Loading