Rate limit errors

[ezeev]

Facebook's rate limits are quite low for apps in "Development" mode and getting approved to come out of development mode is not trivial. This makes it difficult to use Meltano with a new app.

Would it be possible to implement a rate limit backoff? It is possible to see how close you are to the rate limit using a header that FB provides in responses from the API:

def check_limit(account_number, access_token):

    check=requests.get('https://graph.facebook.com/v5.0/'+account_number+'/insights?access_token='+access_token)
    usage=float(find_between(check.headers['x-business-use-case-usage'],'"total_time":',','))
    print('\tRate limit for account %s threshold: %d%%' % (account_number, usage))
    return usage

[ezeev]

Also it looks like the Meltano SDK supports implementing a rate limit backoff? Custom Backoff

I'm brand new to Meltano so my assumptions may be wrong about how or where to implement the rate limit back off.

[pnadolny13]

cc @edgarrmondragon @kgpayne I think you'd know best how to implement this

[edgarrmondragon]

The tap does try to handle rate limits, but harsher Development tier limits may not be correctly handled:

tap-facebook/tap_facebook/client.py

Lines 118 to 170 in f8f3295

	def validate_response(self, response: requests.Response) -> None:
	"""Validate HTTP response.
	Raises:
	FatalAPIError: If the request is not retriable.
	RetriableAPIError: If the request is retriable.
	"""
	full_path = urlparse(response.url).path
	if response.status_code in self.tolerated_http_errors:
	msg = (
	f"{response.status_code} Tolerated Status Code "
	f"(Reason: {response.reason}) for path: {full_path}"
	)
	self.logger.info(msg)
	return
	if (
	HTTPStatus.BAD_REQUEST
	<= response.status_code
	< HTTPStatus.INTERNAL_SERVER_ERROR
	):
	msg = (
	f"{response.status_code} Client Error: "
	f"{response.content!s} (Reason: {response.reason}) for path: {full_path}"
	)
	# Retry on reaching rate limit
	if (
	response.status_code == HTTPStatus.BAD_REQUEST
	and "too many calls" in str(response.content).lower()
	) or (
	response.status_code == HTTPStatus.BAD_REQUEST
	and "request limit reached" in str(response.content).lower()
	):
	raise RetriableAPIError(msg, response)
	raise FatalAPIError(msg)
	if response.status_code >= HTTPStatus.INTERNAL_SERVER_ERROR:
	msg = (
	f"{response.status_code} Server Error: "
	f"{response.content!s} (Reason: {response.reason}) for path: {full_path}"
	)
	raise RetriableAPIError(msg, response)
	def backoff_max_tries(self) -> int:
	"""The number of attempts before giving up when retrying requests.
	Setting to None will retry indefinitely.
	Returns:
	int: limit
	"""
	return 20

If the API provides a header lets the user now when the rate limit will be refreshed, the tap could leverage that in a custom implementation of backoff_runtime.