fix(source-linkedin-ads): increase concurrency by 3x and add an initial HTTPAPIBudget

[brianjlai]

LinkedIn's API is very annoying in that there are basically no documented rate limits, I can do some experimentation, but for now, I feel like we are leaving performance on the table.

Some historical notes:

• After the initial migration, we set concurrency to 19 (the total number of streams). This proved too much and we experienced a lot of rate limiting issues
• We reset this back down to 2.
• Linkedin Ads have 2 levels of daily rate limits, application and member. Application is the total amount of requests to our OAuth app. Member is the amount of requests for an individual user/token.
• We can look up our application limits, but member limits are unpublished.
• Our analytics streams are the most frequently inquired about so this PR focused on them. The per-day application limit for the /adAnalytics endpoint is 15,000,000 req/day.
• As an initial start I've tripled our concurrency and set an API budget of 5 req/sec (10 second sliding window).

After the release I'll monitor the changes and if we see an increase in 429s affecting syncs we will revert and tune this further

[github-actions]

👋 Greetings, Airbyte Team Member!

Here are some helpful tips and reminders for your convenience.

Helpful Resources

• Breaking Changes Guide - Breaking changes, migration guides, and upgrade deadlines
• Developing Connectors Locally
• Managing Connector Secrets
• On-Demand Live Tests
• On-Demand Regression Tests
• #connector-ci-issues
• #connector-publish-updates
• #connector-build-statuses

PR Slash Commands

Airbyte Maintainers (that's you!) can execute the following slash commands on your PR:

• /format-fix - Fixes most formatting issues.
• /bump-version - Bumps connector versions.
  • You can specify a custom changelog by passing changelog. Example: /bump-version changelog="My cool update"
  • Leaving the changelog arg blank will auto-populate the changelog from the PR title.
• /run-cat-tests - Runs legacy CAT tests (Connector Acceptance Tests)
• /build-connector-images - Builds and publishes a pre-release docker image for the modified connector(s).
• JVM connectors:
  • /update-connector-cdk-version connector=<CONNECTOR_NAME> - Updates the specified connector to the latest CDK version.
    Example: /update-connector-cdk-version connector=destination-bigquery
  • /bump-bulk-cdk-version type=patch changelog='foo' - Bump the Bulk CDK's version. type can be major/minor/patch.
• Python connectors:
  • /poe connector source-example lock - Run the Poe lock task on the source-example connector, committing the results back to the branch.
  • /poe source example lock - Alias for /poe connector source-example lock.
  • /poe source example use-cdk-branch my/branch - Pin the source-example CDK reference to the branch name specified.
  • /poe source example use-cdk-latest - Update the source-example CDK dependency to the latest available version.

📝 Edit this welcome message.

[github-actions]

source-linkedin-ads Connector Test Results

49 tests   40 ✅  1m 51s ⏱️
 2 suites   8 💤
 2 files     1 ❌

For more details on these failures, see this check.

Results for commit 95c9450.

♻️ This comment has been updated with latest results.

[github-actions]

Deploy preview for airbyte-docs ready!

✅ Preview
https://airbyte-docs-cvm53egib-airbyte-growth.vercel.app

Built with commit 95c9450.
This pull request is being automatically deployed with vercel-action

[maxi297]

I'm fine trying this budget but it feels like if we don't see a lot of 429 today on /adAnalytics, we could allow for more requests

[maxi297]

I would assume with a budget like that that we're seeing 429 in the logs in prod right now, right? My reasoning is that we might have two threads running analytics streams which would mean that the HTTP request + the processing of records would need to that 4 seconds for each HTTP request to avoid being rate limited.

[brianjlai]

we do see some 429 errors predominantly on the /adAnalytics endpoint in the developer portal metrics, so it is a definite possibility that we'll hit more of these. i'll increase it a little more, but erring on the side of caution since we already hit sync affecting limits the last time we increased