Accelerated need to limit large url requests to “list records”, with alternative POST version

[FredZhao-at]

Hi! Fred here again, from the Airtable API team. Another update is that as part of recent infrastructure improvements to airtable.com, we needed to accelerate enforcement of rejecting any requests whose request URLs exceed 16k characters. The changes described below are in effect today.

• Q: What endpoints does this affect?
  • The 16k character limit applies to all current and future API request URLs
  • Based on our internal audit of current usage, this primary impacts the “list records” endpoint, which is a GET request to /v0/{baseId}/{tableIdOrName} (https://airtable.com/developers/web/api/list-records)
• Q: What is the impact of this change?
  • The vast majority of our API requests have remained well under the 16k character limit, and would not be affected
  • However, requests exceeding this limit will now be rejected, so we expect some calls to now require an update to handle the large cases. For example, requests whose query parameter for filterbyFormula is long enough to make the overall URL length to exceed that character limit
  • Therefore, we recommend either switching entirely to the alternative detailed below, or switching to it when you can detect that a request URL will exceed the 16k character limit
• Q: What is the alternative?
  • APIs can instead make a POST request to /v0/{baseId}/{tableIdOrName}/listRecords, and encode the query params as the body of that POST request. We currently still require two parameters to remain as query params, timeZone and userLocale; you can see the full details and examples here: https://support.airtable.com/docs/enforcement-of-url-length-limit-for-web-api-requests
• Q: Why the urgency for this change?
  • I apologize for the inconvenience due to the accelerated timeline here. While we originally planned to start this enforcement in January 2023, we needed to deploy this change recently last week, as part of required infrastructure improvements to maintain the stability of our public API

[Meekohi]

Thanks for the heads up Fred, I guess we'll see if anyone is impacted by this. I'm not developing on Airtable actively anymore, but if there's a need I might be able to carve out some time to make a PR.

[andruo11]

I ran into (I think) this issue today, while trying to find associated records from a parent row, maybe when there's a lot of child rows, kept getting 414 communication error which Mozilla says = URL too long.

[theblang]

@FredZhao-at Thanks so much for communicating this! I've put in a PR to switch the list-records call over to POST.

Tangentially related question, and I'll preface by saying it's really a moot question now that we're switching to POST. But I wanted to ask out of curiosity, because I thought the behavior was strange. Do you know of any special behavior regarding indexed vs non-indexed array params, ?foo[0]=0&foo[1]=1 vs ?foo[]=0&foo[]=1? I initially found this Github issue because our list-records call started failing with 422 Invalid request: parameter validation failed. Check your request data. after I increased the size of the fields param. I initially suspected maybe the length of the GET request was the problem, but it's well under 16k. Eventually I realized that this library is encoding array params as indexed with a note, "Airtable expects that arrays in query strings be encoded with indices." But the Airtable docs show non-indexed, so in my PR I switched the encoder back to the default Faraday one (which is non-indexed). Then the list-records call started working again! I think the number of fields that would trigger the issue was about 22 (it would work fine with less than that). It's also worth noting that this library is encoding spaces as %20 vs + (which the default Faraday one uses).